feat: repo known files tracking [CM-1329]

[mbani01]

This pull request adds support for tracking well-known files (such as README, SECURITY.md, CONTRIBUTING.md, etc.) in GitHub repositories, both in the enrichment pipeline and in the database. The changes introduce a new database table to inventory these files, update the enrichment worker to extract and classify them, and ensure the data is efficiently upserted and soft-deleted as repositories change.

Database changes:

• Added a new table repo_well_known_files to store inventory of well-known files per repository, including file type, directory, path, and blob OID, with support for soft deletion and change tracking.

Enrichment pipeline updates:

• Modified the GraphQL query in fetchLightRepo.ts to fetch the root, .github, and docs directory trees, and removed the previous REST-based check for SECURITY.md. [1] [2]
• Added logic to classify well-known files and determine if a security file is present using the new directory trees, storing results in the enrichment output. [1] [2] [3] [4] [5]
• Updated types to include well-known files in LightRepoResult and added a new RepoWellKnownFilesUpdate type for batch updates. [1] [2]

Batch processing and upsert:

• Extended the enrichment write buffer to collect, flush, and clear well-known file updates in batches, ensuring deduplication and transactional integrity. [1] [2] [3] [4] [5] [6]
• Added a new function bulkUpsertRepoWellKnownFiles to upsert and soft-delete well-known file records efficiently in the database.

Well-known file classification logic:

• Introduced wellKnownFiles.ts with logic to classify files by type and directory, and to derive the presence of a security file from the directory trees.

These changes enable the system to efficiently track, update, and query the presence and state of well-known files across all monitored repositories.

[cursor]

PR Summary

Medium Risk
Touches the enrichment write path and changes how security-file presence is detected (GraphQL trees vs REST), which could affect downstream metrics if tree access differs from Contents API; new table upserts run in the existing flush transaction.

Overview
Adds repo_well_known_files so the GitHub repos enricher can persist an inventory of community/governance files (README, SECURITY, CONTRIBUTING, etc.) per repo, with blob OIDs, soft deletes, and change_detected_at when content or presence changes.

fetchLightRepo now loads root, .github, and docs trees via GraphQL and drops the extra REST Contents calls for SECURITY.md. security_file_enabled is derived from those trees (root + .github only) so behavior stays aligned with the old probe; wellKnownFiles is classified in new wellKnownFiles.ts.

The enrichment write buffer batches bulkUpsertRepoWellKnownFiles in the same transaction as other repo writes, upserting paths and soft-deleting rows that disappear on the latest scan.

^{Reviewed by Cursor Bugbot for commit 5a06648. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Adds GitHub repository well-known-file inventory and change tracking.

Changes:

• Fetches and classifies files from root, .github, and docs.
• Adds transactional upsert, soft-delete, and reappearance tracking.
• Introduces the repo_well_known_files table and indexes.

Metadata: Rename the PR to feat: track repository well-known files (CM-1329) to match repository conventions.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
wellKnownFiles.ts	Classifies well-known files.
updateWellKnownFiles.ts	Persists inventory changes.
types.ts	Adds inventory types.
runEnrichmentLoop.ts	Buffers and flushes inventory updates.
fetchLightRepo.ts	Fetches repository directory trees.
V1784905809__repo_well_known_files.sql	Creates the inventory schema and indexes.
updateEnrichedRepos.ts	Context reviewed for enrichment persistence integration.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

services/apps/packages_worker/src/enricher/updateWellKnownFiles.ts:8

• This new database query is implemented inside the worker application, but the repository architecture places database query functions in services/libs/data-access-layer (CLAUDE.md:32-34) and requires new code to follow that pattern. Move this helper into the DAL and import it from there so persistence remains reusable and centralized.

export async function bulkUpsertRepoWellKnownFiles(
  qx: QueryExecutor,
  updates: RepoWellKnownFilesUpdate[],
): Promise<void> {

services/apps/packages_worker/src/enricher/wellKnownFiles.ts:54

• The new classifier has no unit coverage, although comparable pure normalization logic in this worker is covered under src/*/__tests__ (for example maven/__tests__/normalize.test.ts and utils/__tests__/canonicalizeRepoUrl.test.ts). Add cases for all directory/path mappings, non-blob filtering, filename normalization, and the legacy security-file semantics; otherwise a small normalization change can silently corrupt the inventory.

export function classifyWellKnownFiles(trees: RepoTrees): WellKnownFileEntry[] {

services/apps/packages_worker/src/enricher/updateWellKnownFiles.ts:38

• The insert/update/soft-delete/reappearance state machine is untested. Similar reconciliation code has integration coverage (for example osv/__tests__/reconcileOsvRanges.integration.test.ts:121-183). Add integration cases for initial insert, unchanged and changed blobs, disappearance, reappearance, and multiple updates for one repo so timestamp and tombstone regressions are caught.

    soft_deleted AS (
      UPDATE repo_well_known_files w
      SET deleted_at = i.checked_at,
          change_detected_at = i.checked_at
      FROM input i

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}