Draft: feat: backfill subject_id when subject arrives (referrer completeness)
What
Implements the referrer-completeness follow-up from #2376 (items 2 and 3): record the OCI
subject digest on every push and backfill subject_id when the subject manifest arrives, so
a referrer pushed before its subject is correctly linked once the subject exists.
Stacking / dependencies
This branch is stacked on !2927 (merged) and the subject_digest column MR (!2928 (merged)) and contains
their commits. Review/merge order: !2927 (merged) -> !2928 (merged) -> this. Once those land, this rebases down
to just the plumbing + write-path + trigger + index + tests.
How
- Write path (
registry/handlers/manifests.go): setsubject_digestfrom the manifest'ssubjecton every push, whether or not the subject is present. - Backfill trigger (migration): an
AFTER INSERT ON manifeststrigger,backfill_manifest_subject_id, setssubject_id = NEW.idfor referrers in the same repository whosesubject_digest = NEW.digestandsubject_id IS NULL. Atomic with the insert, so there is no window where the new subject is dangling before the link exists; it covers all manifest creation paths. - Index (migrations): a partial index
WHERE subject_id IS NULLon(top_level_namespace_id, repository_id, subject_digest)backs the trigger lookup. It is built with the batched per-partitionCREATE INDEX CONCURRENTLYrollout (an integration testing variant, four batches of 16 partitions, then the parentCREATE INDEXthat attaches them), mirroring thesubject_idindex, to avoid a long write-blocking lock onmanifests. - Datastore plumbing: thread
subject_digestthroughmodels.Manifest, the manifest INSERTs, the row scanners, and the manifest SELECT column lists. Read paths (referrers listing, GCIsDangling) are unchanged; they keep usingsubject_id, now kept correct.
Why no data backfill (BBM) is needed
Existing referrers already have subject_id resolved (before !2927 (merged) a referrer could only be
pushed if its subject existed). NULL-subject_id orphans only appear once
REGISTRY_FF_TOLERATE_MISSING_SUBJECT is enabled; if it is enabled only after this ships, the
trigger links everything going forward and nothing historical needs repair. Keep the flag off
until this is deployed.
Testing
- New integration test
TestGitlabAPI_ManifestPut_SubjectBackfilledWhenSubjectArrives: push a referrer before its subject (tolerated, nullsubject_id), push the subject, then assert the referrer appears in thereferrers=truelisting. go build ./...passes; integration build compiles (bothintegrationand!integration).- Migrations applied cleanly on Postgres 17 (200 pre + 20 post);
structure.sqlregenerated with pg_dump 17 + pgFormatter 5.0.
Closes part of #2376. Depends on !2927 (merged) and !2928 (merged).