Draft: feat: backfill subject_id when subject arrives (referrer completeness)

What

Implements the referrer-completeness follow-up from #2376 (items 2 and 3): record the OCI subject digest on every push and backfill subject_id when the subject manifest arrives, so a referrer pushed before its subject is correctly linked once the subject exists.

Stacking / dependencies

This branch is stacked on !2927 (merged) and the subject_digest column MR (!2928 (merged)) and contains their commits. Review/merge order: !2927 (merged) -> !2928 (merged) -> this. Once those land, this rebases down to just the plumbing + write-path + trigger + index + tests.

How

  1. Write path (registry/handlers/manifests.go): set subject_digest from the manifest's subject on every push, whether or not the subject is present.
  2. Backfill trigger (migration): an AFTER INSERT ON manifests trigger, backfill_manifest_subject_id, sets subject_id = NEW.id for referrers in the same repository whose subject_digest = NEW.digest and subject_id IS NULL. Atomic with the insert, so there is no window where the new subject is dangling before the link exists; it covers all manifest creation paths.
  3. Index (migrations): a partial index WHERE subject_id IS NULL on (top_level_namespace_id, repository_id, subject_digest) backs the trigger lookup. It is built with the batched per-partition CREATE INDEX CONCURRENTLY rollout (an integration testing variant, four batches of 16 partitions, then the parent CREATE INDEX that attaches them), mirroring the subject_id index, to avoid a long write-blocking lock on manifests.
  4. Datastore plumbing: thread subject_digest through models.Manifest, the manifest INSERTs, the row scanners, and the manifest SELECT column lists. Read paths (referrers listing, GC IsDangling) are unchanged; they keep using subject_id, now kept correct.

Why no data backfill (BBM) is needed

Existing referrers already have subject_id resolved (before !2927 (merged) a referrer could only be pushed if its subject existed). NULL-subject_id orphans only appear once REGISTRY_FF_TOLERATE_MISSING_SUBJECT is enabled; if it is enabled only after this ships, the trigger links everything going forward and nothing historical needs repair. Keep the flag off until this is deployed.

Testing

  • New integration test TestGitlabAPI_ManifestPut_SubjectBackfilledWhenSubjectArrives: push a referrer before its subject (tolerated, null subject_id), push the subject, then assert the referrer appears in the referrers=true listing.
  • go build ./... passes; integration build compiles (both integration and !integration).
  • Migrations applied cleanly on Postgres 17 (200 pre + 20 post); structure.sql regenerated with pg_dump 17 + pgFormatter 5.0.

Closes part of #2376. Depends on !2927 (merged) and !2928 (merged).

Edited by Stan Hu

Merge request reports

Loading
Loading