fix(oci): fence the image upsert on the upload session open (#1122 part 2 stack: 3/3)

What this delivers

This MR is part 2c of issue 1122, and the last MR of a stack of three:

  1. The repository-fenced image upsert in the datastore: !3378 (merged).
  2. The initiate write deadline: !3379 (merged), which targets the branch of !3378 (merged).
  3. This MR: the fence on the OCI write paths. It targets the branch of !3379 (merged).

This MR closes the repository tombstone window on the three write paths that open an upload session: the blob upload initiate, the single-request POST ?digest=, and the fallback arm of the cross-repository blob mount. On a repository that was tombstoned after the resolve, each path now answers the repository-tier 404 NAME_UNKNOWN and writes no container_images row and no upload session.

Cause

CreateSession wrote the image row with UpsertContainerImage as one statement on the pool handle, outside a transaction. Nothing read the repository row again between the resolve and that write. Also, writeSessionOpenError had no arm for datastore.ErrRepositoryConcurrentlyDeleted. As a result, a tombstone that committed after the resolve let the initiate and the mount fallback answer 202, and let the single-request POST answer 201. Each of these requests left a container_images row for the purge to remove.

The fix

  • The wire adapter containerImageUpserter sends every oci.ImageUpserter call to ContainerImageStore.UpsertContainerImageInLiveRepository, the method that !3378 (merged) adds. As a result, the image upserts in the finalize (CompleteUpload) and in the mount (MountBlob) are fenced too. oci.ImageUpserter and datastore.UpsertContainerImage keep their signatures.
  • writeSessionOpenError answers the sentinel with the repository-tier 404 NAME_UNKNOWN, before the caller-gone check. The 404 is the same masked answer that a name that was never held gets.
  • CreateSession, CompleteUpload and MountBlob run the image upsert on a context of its own, bounded at lifecycle.TombstoneTimeout. Past that bound, the answer is 500, or 499 when the client is gone.
  • mountWriteTimeout now adds the bound of the upsert: lifecycle.TombstoneTimeout + counterEmitTimeout + blobLinkTimeout + 5s. The fallback arm of the mount opens a session under the same budget, so it spends the 10s upsert bound plus the admission-gate wait of storage.upload_session_wait_timeout. At the 15s ceiling of that field, the fallback arm fits with 5s left, because counterEmitTimeout + blobLinkTimeout is 15s. TestMountWriteTimeoutCoversTheImageUpsertBound asserts that fit, so a higher ceiling fails the test.
  • The initiate write deadline from !3379 (merged) now adds the upsert bound too: lifecycle.TombstoneTimeout plus storage.upload_session_wait_timeout plus a 5s tail (operator ruling q29, option B). At the 10s example value of server.timeouts.write, an upsert that used its full bound wrote its 500 after the deadline.
  • The mount fault log line now reads oci mount: destination image write or blob link failed, and its 499 sibling changes the same way. The old text, link source blob failed, named the wrong step when the destination image upsert failed.

Documentation changed with the code

  • S20-A: the tombstone-window text no longer says "until the initiate is re-verified too". The Error Cases row now states the uploadLockWaitTimeout bound of the Maven upload, in place of the claim that the Maven upload has no deadline. The row on a container blob link that parks on the repositories row lock now names every arm that parks, the image-upsert wait and its bound, and the write deadline of each arm. It also names which arms hold an upload admission unit (operator ruling q28, option C), and it names the Maven lock as FOR SHARE OF repositories.
  • S12: the mount status table, the mount sequence and the SQL step 3 comment match the new answer. The 499 row names the source blob lookup, the destination image write and the blob link. Upload Read Deadlines gives the four-term mount budget, adds the image-write bound to the initiate budget, and names the fallback arm and its fit at the 15s wait ceiling. The mount fallback paragraph counts four session-open answers, and it names the 503 as the one answer that the status table does not list.
  • api/openapi/v1.yaml (deleteRepository) and api/bruno/management-api/repositories/delete-repository.bru now list the blob upload initiate, and both arms of the mount, as paths that re-verify the repository.
  • docs/dev/configuration-reference.md: the server.timeouts.write row gives the four-term mount budget and adds the 10-second image-write bound to the initiate budget. It says that the five read-timeout keys replace the write deadline with their own value. The row also names the fallback arm of the mount: it spends the 10-second image-write bound plus storage.upload_session_wait_timeout, which leaves 5 seconds at the 15-second ceiling. The storage.upload_session_wait_timeout row adds the image-write bound to the initiate deadline.

Spec coverage

Rows come from the spec, S20-A (## Acceptance Criteria, ## Error Cases, ## Security Considerations), plus one table for the card's part 2 acceptance items.

Spec: docs/specs/S20-a-lifecycle-closed-beta.md Unit: issue 1122 part 2 (blob upload initiate, B2 image fence). "Other owner" rows are outside this unit: merged S20-A steps, the part 1 MR (f6577278), or parts 3 to 5 of this issue.

Acceptance criteria

# Criterion Tests
AC-1 Retention-window discovery of repository tombstones Other owner: S20-A purger steps (merged).
AC-2 A live row is never a purge root Other owner: S20-A purger steps (merged).
AC-3 Four scans use the tombstone index and prune to one partition Other owner: S20-A database review (merged).
AC-4 Version-level scans use the tombstone indexes Other owner: S20-A database review (merged).
AC-5 Purge re-run is a no-op; an interrupted purge resumes Other owner: S20-A purger steps (merged).
AC-6 npm walk reaps live and tombstoned rows in one pass Other owner: S20-A purger steps (merged).
AC-7 Every purge transaction is bounded Other owner: S20-A purger steps (merged).
AC-8 Purge job in a frozen namespace refuses Other owner: S20-A purger steps (merged).
AC-9 Tombstoned npm version is reaped Other owner: S20-A npm reap step (merged).
AC-10 Unpublish then republish leaves no orphan Other owner: S20-A npm reap step (merged).
AC-11 Tombstoned npm package is reaped with tags and metadata Other owner: S20-A npm reap step (merged).
AC-12 DeleteNpmMetadataFiles removes attachments Other owner: S20-A npm reap step (merged).
AC-13 Tombstoned Maven version and package are reaped Other owner: S20-A Maven reap step (merged).
AC-14 Container repository reaped through the per-artifact deleters Other owner: S20-A container reap step (merged).
AC-15 Image index reaped parent-first Other owner: S20-A container reap step (merged).
AC-16 Tombstoned image in a live repository is reaped Other owner: S20-A container reap step (merged).
AC-17 Container walk reaps live and tombstoned images Other owner: S20-A container reap step (merged).
AC-18 Remote-cache subtree is reaped Other owner: S20-A remote reap step (merged).
AC-19 container_remote_manifests and container_remote_blobs are reaped Other owner: S20-A remote reap step (merged).
AC-20 Reaped blobs keep zero attachments Other owner: S20-A purger steps (merged).
AC-21 DELETE without destructive answers 400 Other owner: S20-A entry-point step (merged).
AC-22 destructive=false on an empty repository answers 204 Other owner: S20-A entry-point step (merged).
AC-23 destructive=false on a non-empty repository answers 409 Other owner: S20-A entry-point step (merged).
AC-24 The 409 message names the remedy Other owner: S20-A entry-point step (merged).
AC-25 destructive=true on a non-empty repository answers 202 and enqueues Other owner: S20-A entry-point step (merged).
AC-26 destructive=true on an empty repository answers 204 Other owner: S20-A entry-point step (merged).
AC-27 After the 202, every client route naming the repository answers 404 Serial, passes today: TestTombstonedRepositoryIntegration_OCIRoutes404 (row POST blobs/uploads). Tombstone committed after resolve, new: TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert (initiate, single_post, mount_fallback). The from= sub-clause, passes today: TestBlobMountLivenessRace_TombstonedSourceKeepsIts202. npm routes: parts 3 and 4.
AC-28 Create at a tombstoned name: 409, then 201 Other owner: S20-A entry-point step (merged).
AC-29 The finalizer cannot succeed early Other owner: S20-A finalizer step (merged).
AC-30 The finalizer removes the row, the child row and the links Other owner: S20-A finalizer step (merged).
AC-31 Sweep re-queues within one interval Other owner: S20-A sweep step (merged).
AC-32 Sweep reads oldest-first Other owner: S20-A sweep step (merged).
AC-33 Orphaned npm_packages tombstone is enumerated Other owner: S20-A sweep step (merged).
AC-34 No re-queue while a live job exists Other owner: S20-A sweep step (merged).
AC-35 Re-queue after a terminal job Other owner: S20-A sweep step (merged).
AC-36 A permanent purge failure stays counted and re-queued Other owner: S20-A sweep step (merged).
AC-37 The sweep registers its worker Other owner: S20-A sweep step (merged).
AC-38 A frozen namespace is not enqueued by the sweep Other owner: S20-A sweep step (merged).
AC-39 FindByNameInNamespace hides a tombstoned repository (docker, oci) Other owner: S20-A visibility step (merged).
AC-40 RepositoryStore.FindByID hides a tombstoned repository Other owner: S20-A visibility step (merged).
AC-41 RepositoryStore.Update refuses a tombstoned repository Other owner: S20-A visibility step (merged).
AC-42 Delete, decrements, MarkRepositoryLastUpdated and Missing still act Other owner: S20-A visibility step (merged).
AC-43 parentRepositoryIsActive fails on a mid-transaction tombstone Other owner: S20-A visibility step (merged).
AC-44 The four container_images gates hide a tombstoned image Other owner: S20-A image-tier step (merged).
AC-45 /v2/ image-tier 404 on a tombstoned image Other owner: S20-A image-tier step (merged).
AC-46 Mount answers 202 when every source image is tombstoned Other owner: S20-A image-tier step (merged). This unit changes the fallback only on a tombstoned destination repository; the live-destination 202 stays pinned by TestInitiateLivenessRace_RolledBackTombstoneOpensTheSession/mount_fallback.
AC-47 A tombstoned image named in from= still mounts with 201 Other owner: S20-A image-tier step (merged).
AC-48 Mount destination: image tier 201; repository tier answered by the Error Cases row Repository tier, new: TestMountFallbackDestinationTombstonedReturns404 (source_missing, source_denied), TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert/mount_fallback. Image tier 201: other owner, unchanged. Dropping "until the initiate is re-verified too" from the text: implementer (docs).
AC-49 UpsertContainerImage does not resurrect a tombstoned image Other owner: S20-A image-tier step (merged), TestContainerImageStore_Upsert_WithASoftDeletedSibling. The new method delegates to the same upsert.
AC-50 ContainerImageStore.Delete still acts on a tombstoned row Other owner: S20-A image-tier step (merged).
AC-51 Destructive DELETE emits no delta at the tombstone Other owner: S20-A accounting step (merged).
AC-52 Size and component decrements fire at the purger Other owner: S20-A accounting step (merged).
AC-53 Every emit site commits before it emits Other owner: S20-A accounting step (merged).
AC-54 A purge chunk emits from its committed aggregate Other owner: S20-A accounting step (merged).
AC-55 No raw foreign-key cascade Other owner: S20-A accounting step (merged).
AC-56 The three index migrations are in structure.sql Other owner: S20-A schema step (merged).
AC-57 No column added or dropped, no CHECK on soft_deleted_at Other owner: S20-A schema step (merged).
AC-58 unique_container_images_ns_id_cr_id_name is partial Other owner: S20-A schema step (merged).
AC-59 Config accepts and rejects retention_window and sweep_interval Other owner: S20-A config step (merged).
AC-60 Maven metadata tombstone discovered with its package live Other owner: S20-A Maven metadata step (merged).
AC-61 Maven metadata retention-window filter Other owner: S20-A Maven metadata step (merged).
AC-62 Maven metadata partial index and its migration Other owner: S20-A Maven metadata step (merged).
AC-63 Republished Maven metadata survives a purge Other owner: S20-A Maven metadata step (merged).
AC-64 Concurrent Maven metadata purges count once Other owner: S20-A Maven metadata step (merged).
AC-65 Tombstone gauge includes Maven metadata Other owner: S20-A Maven metadata step (merged).

Error cases

# Condition Tests
E-1 DELETE omits destructive Other owner: S20-A entry-point step (merged).
E-2 destructive is not true or false Other owner: S20-A entry-point step (merged).
E-3 destructive=false on a non-empty repository Other owner: S20-A entry-point step (merged).
E-4 The hard-delete phase parks on the row lock Other owner: S20-A entry-point step (merged).
E-5 A container write parks on the repositories row lock: each park bounded Image-upsert park, new: TestContainerImageStore_UpsertContainerImageInLiveRepository_HonorsCallerDeadline, TestCreateSession_BoundsTheImageUpsertAtTombstoneTimeout, TestCompleteUpload_BoundsTheImageUpsertAtTombstoneTimeout, TestMountBlob_BoundsTheImageUpsertAtTombstoneTimeout, TestMountWriteTimeoutCoversTheImageUpsertBound. Past the bound 500, passes today: TestUploadPOST_UpsertErrorReturns500; 499 with the client gone, passes today: TestUploadShed_CallerGoneTakes499. LinkBlob park: other owner, part 1 (merged).
E-6 An artifact published mid-DELETE Other owner: S20-A entry-point step (merged).
E-7 A write resolved before the tombstone commits after it Initiate, single-request POST and mount fallback, new: TestContainerImageStore_UpsertContainerImageInLiveRepository_RepositoryGate, _TombstonedMidTransaction, _RolledBackTombstoneUpserts, _RunsAtReadCommitted, TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert, TestInitiateLivenessRace_RolledBackTombstoneOpensTheSession, TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown, TestSessionOpen_TombstoneSentinelAnswers404AfterClientHangup, TestMountFallbackDestinationTombstonedReturns404, TestContainerImageUpserter_ProjectsRowAndWrapsError. Finalize and mount sentinel arms, passes today: TestCompleteUpload_SurfacesRepositoryConcurrentlyDeleted, TestMountBlob_SurfacesRepositoryConcurrentlyDeleted, TestUploadFinalize_RepositoryTombstonedMidLinkReturns404NameUnknown. Live 202 and 201, passes today: TestUploadPOST_InitiateReturns202WithLocation, TestUploadPUT_FinalizeReturns201, TestUploadPOST_SingleShotReturns201, TestMountSucceedsReturns201. Maven-deadline sentence: implementer (docs). npm dist-tag PUT, npm deprecate, upsertContainerTag: parts 3, 4 and 5.
E-8 A package tombstone mid Maven publish Other owner: S20-A Maven step (merged).
E-9 DELETE on a missing or tombstoned repository Other owner: S20-A entry-point step (merged).
E-10 Create at a name held by a tombstoned repository Other owner: S20-A entry-point step (merged).
E-11 Any client route naming a tombstoned repository answers 404 in its own vocabulary Serial, passes today: TestTombstonedRepositoryIntegration_OCIRoutes404. Session-open race, new: TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert (NAME_UNKNOWN code, message and detail against the unknown-name baseline), TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown, TestMountFallbackDestinationTombstonedReturns404.
E-12 A mount whose from= names a tombstoned repository answers 202 Passes today: TestBlobMountLivenessRace_TombstonedSourceKeepsIts202. Unchanged by this unit.
E-13 Purge job: 23503 on a parent delete Other owner: S20-A purger steps (merged).
E-14 Purge job: manifest refused as referenced Other owner: S20-A purger steps (merged).
E-15 Purge job: the finalizer aborts Other owner: S20-A finalizer step (merged).
E-16 Purge job: repository already hard-deleted Other owner: S20-A purger steps (merged).
E-17 Purge job: retention window changed Other owner: S20-A purger steps (merged).
E-18 Sweep: enumeration query fails Other owner: S20-A sweep step (merged).
E-19 Sweep: interval elapses mid-enumeration Other owner: S20-A sweep step (merged).
E-20 Emit API unavailable Other owner: S20-A accounting step (merged).
E-21 Parent gate: parent tombstoned before the child INSERT Other owner: S20-A visibility step (merged).
E-22 A tombstoned coordinate is reoccupied before the reap Other owner: S20-A npm reap step (merged).

Security considerations

# Concern Tests
S-1 Authorization is unchanged Other owner: S09. This unit adds no authorization path. Not tested in this unit.
S-2 destructive is a confirmation, not a control Other owner: S20-A entry-point step (merged).
S-3 Irreversibility is documented Other owner: S20-A docs (merged).
S-4 Input validation Other owner: S20-A entry-point step (merged). Zero-id guard on the new method: TestContainerImageStore_UpsertContainerImageInLiveRepository_RepositoryGate (zero id).
S-5 No injection surface is added The new method reuses the existing parameterized statements and adds none. Not tested in this unit.
S-6 Error messages do not become an existence oracle New: TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert (byte-identical to the unknown-name answer, modulo the requested name), TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown, TestMountFallbackDestinationTombstonedReturns404.
S-7 Purge failures are visible Other owner: S20-A sweep step (merged).

Card part 2 acceptance

# Item Tests
A-1 Initiate on a repository tombstoned after resolve: 404 NAME_UNKNOWN, no image row, no session TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert/initiate, TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown
A-2 Mount fallback with a tombstoned destination: 404 TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert/mount_fallback, TestMountFallbackDestinationTombstonedReturns404
A-3 Single-request POST on a tombstoned repository: 404 at session open TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert/single_post, TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown
A-4 The new method waits; commit gives the sentinel and no row; rollback inserts TestContainerImageStore_UpsertContainerImageInLiveRepository_TombstonedMidTransaction, _RolledBackTombstoneUpserts, TestInitiateLivenessRace_RolledBackTombstoneOpensTheSession
A-5 The new method refuses a pool handle Open question Q1: the new method takes no handle. The guard it calls is pinned by TestContainerRepositoryStore_ReVerifyRepositoryAlive_ArgumentGuards (row connection pool), passes today.
A-6 The new method opens its transaction at READ COMMITTED TestContainerImageStore_UpsertContainerImageInLiveRepository_RunsAtReadCommitted (repeatable-read session default)
A-7 The wait is bounded by TombstoneTimeout; past it 500, or 499 when the client is gone TestContainerImageStore_UpsertContainerImageInLiveRepository_HonorsCallerDeadline, TestCreateSession_BoundsTheImageUpsertAtTombstoneTimeout; 500 and 499 pass today: TestUploadPOST_UpsertErrorReturns500, TestUploadShed_CallerGoneTakes499
A-8 Finalize and mount image upserts bounded on their own context; sentinel reaches the 404 arms; mountWriteTimeout includes the bound TestCompleteUpload_BoundsTheImageUpsertAtTombstoneTimeout, TestMountBlob_BoundsTheImageUpsertAtTombstoneTimeout, TestMountWriteTimeoutCoversTheImageUpsertBound; sentinel arms pass today: TestCompleteUpload_SurfacesRepositoryConcurrentlyDeleted, TestMountBlob_SurfacesRepositoryConcurrentlyDeleted, TestUploadFinalize_RepositoryTombstonedMidLinkReturns404NameUnknown
A-9 Sentinel answers 404 regardless of the hangup check TestSessionOpen_TombstoneSentinelAnswers404AfterClientHangup
A-10 Live repository: 202 Range 0-0 on initiate, 201 on finalize and mount Pass today: TestUploadPOST_InitiateReturns202WithLocation, TestUploadPUT_FinalizeReturns201, TestUploadPOST_SingleShotReturns201, TestMountSucceedsReturns201; through the fence: TestInitiateLivenessRace_RolledBackTombstoneOpensTheSession, TestContainerImageStore_UpsertContainerImageInLiveRepository_RepositoryGate
A-11 oci.ImageUpserter and UpsertContainerImage keep their signatures Compile-time: the unchanged fakes in upload_test.go, emit_gate_internal_test.go, upload_finalize_integration_test.go build against them.
A-12 "until the initiate is re-verified too" removed; oci.md and S12 match Docs only. Implementer.
A-13 S20-A :1055 states the Maven uploadLockWaitTimeout bound Docs only. Implementer.

The tables cover the whole unit, that is, all three MRs of the stack. These tests in the tables land in !3378 (merged): TestContainerImageStore_UpsertContainerImageInLiveRepository_RepositoryGate, _TombstonedMidTransaction, _RolledBackTombstoneUpserts, _RunsAtReadCommitted and _HonorsCallerDeadline. All other tests in the tables land in this MR.

Tests changed after the coverage table

The table above was written in the test-first commit of the unit, before the split. Later commits changed these tests, and the third column names the MR that carries each one:

Test Change MR
TestUploadPOST_UpsertWaitPastBoundReturns500Or499 Added. A context error out of the image upsert through the initiate answers 500 INTERNAL under a live caller, and 499 with no body when the caller hung up during the upsert (A-7, E-5). This MR
TestUpload_InitiateArmsWriteDeadline Added. The initiate arms its own write deadline, sized from the configured session wait, and arms no read deadline. This MR adds the assertion that the deadline outlasts lifecycle.TombstoneTimeout plus the wait. !3379 (merged), this MR
TestUpload_InitiateSetWriteDeadlineFailureReturns500 Added. A connection that refuses the write deadline answers 500 before the image upsert runs. !3379 (merged)
TestInitiateWriteTimeoutCoversTheUpsertAndTheSessionWait Renamed from TestInitiateWriteTimeoutCoversTheSessionWait (!3379 (merged)). The initiate budget is more than lifecycle.TombstoneTimeout plus the wait, for a configured, a zero, a negative and an unset wait. This MR
TestUpload_BodylessPathsSetNoDeadline Changed. The POST initiate row is removed, because the initiate now arms a write deadline. !3379 (merged)
TestMountWriteTimeoutCoversItsOwnBoundedWork Removed. It asserted two bounds, and TestMountWriteTimeoutCoversTheImageUpsertBound asserts all three. This MR
TestMountWriteTimeoutCoversTheImageUpsertBound Changed. It also asserts that mountWriteTimeout is at least lifecycle.TombstoneTimeout plus 15s plus 5s, which is the fallback arm at the 15s ceiling of storage.upload_session_wait_timeout (E-5, A-8). This MR
TestBuildOCIStoreHandlers_ThreadsUploadSessionWait Added. The composition root builds the OCI handlers with a 12s session wait, and the upload handler reports 12s through the UploadSessionWait accessor. !3379 (merged)
TestContainerImageStore_UpsertInLiveRepository_NilContext Added. UpsertContainerImageInLiveRepository returns errContainerImageNilContext for a nil context, before it reads the database. !3378 (merged)
TestContainerImageUpserter_ProjectsRowAndWrapsError Changed. The pool-call counter, the fake's pool-handle UpsertContainerImage and the NotPanics wrappers are removed, because the narrowed store interface makes the call that they guarded impossible. The fenced-call and argument assertions stay. This MR
TestMountLinkErrorAfterClientHangupReturns499, TestMountLinkErrorWithLiveClientStays500 Changed. They assert the new destination image write or blob link failed log text. This MR

e2e scenario catalog

The e2e.oci.setup.delete-repository row in docs/testing/e2e/oci.md now says that a mount whose destination is tombstoned and that takes the fallback arm answers 404 NAME_UNKNOWN at the repository tier, with no Location header. The row gives the cause as the mechanism, not as an assertion: the fallback writes the image row through a transaction that re-verifies the repository first, so the refused fallback writes no image row and opens no upload session. This MR adds no row, because the existing row covers the delete-repository journey that changed.

User documentation

No page under docs/user/ describes the race between a repository delete and a write that is in flight, or the answers that the three session-open paths give in that race. Thus, docs/user/ does not change.

Size

This MR is 1,064 changed lines (876 added, 188 deleted) in 15 files, measured against the head of !3379 (merged) (1cd0a37bc).

File group Files Added Deleted Total
Production Go (internal/, cmd/) 3 66 96 162
Go tests 6 738 42 780
Specs, API contracts and docs 6 72 50 122

The production change is 162 lines. The tests are 73% of the diff, and two test files hold 531 of those lines: internal/format/oci/initiate_liveness_race_test.go (290, new) and internal/format/oci/upload_test.go (241). This MR does not split further:

  • The adapter switch fences every caller at the same time, so the 404 arm, the upsert bounds and the larger mountWriteTimeout must land with it.
  • The new tests assert the new 404 answers, so they cannot merge before the switch that gives those answers.
  • The upsert-bound comments and the ImageUpserter doc describe a row-lock wait that exists only after the switch.

Split validation

Before the split, this MR had head 6b8b295e8. The stack !3378 (merged) → !3379 (merged) → this MR has head 47bd9b5d7. git diff 6b8b295e8 47bd9b5d7 is empty: both commits have the tree 890ff79a9c358c810686476e254ee6d6a3abd711. Thus, the three MRs merged in order give the same code as the earlier head of this MR.

The intermediate states also build:

Head Content go build, go vet (also with -tags integration) Pre-commit hooks
e20353dc0 !3378 (merged) Pass Pass
1cd0a37bc !3378 (merged) + !3379 (merged) Pass Pass
47bd9b5d7 Full stack Pass Pass, on the second run. The first run failed in internal/virtual, a package that the stack does not touch: TestResolve_Integration_ConcurrentResolutions_CoalesceToOneUpstreamGET fails under go test -count=3 on this tree and passes on a single run.

Verification

The OCI conformance suite did not run locally. db:setup failed on a port conflict on host port 5432 before the suite started. The CI conformance:oci:* jobs cover the suite on this MR.

Database review

!3378 (merged) holds the database review of UpsertContainerImageInLiveRepository: its three statements, their locks and the earlier plans. This MR adds the callers on the request path. Before this MR, the upsert ran as one statement on the pool handle, outside a transaction. Now each image upsert of the initiate, the single-request POST, the finalize and the mount runs in its own transaction, after the share lock on the parent repositories row. The transaction adds a BEGIN, the re-verification and a COMMIT, so each upsert makes three more round trips. The share lock waits behind a tombstone UPDATE of the same row, and the caller bounds that wait at lifecycle.TombstoneTimeout.

The cost of the share lock. Each image upsert now takes a FOR SHARE lock on the parent repositories row, in a transaction of its own. LinkBlob already took the same lock before this MR. Thus, a pushed layer takes this lock 3 times: in the initiate upsert, in the finalize upsert and in the link. A mount takes it 2 times: in the destination image upsert and in the link.

The k6 push_small scenario ran in the test:perf:caproni job on the branch and on main, with 5 virtual users for 20 s against one repository.

Run Commit Pipeline Job Blob upload data phase, p95 manifest_put, p95
Branch fd209a75e 2956117669 17241442123 96.37 ms 84.68 ms
main 4b95807b9 2955951868 17241499561 111.97 ms 99.67 ms

The blob upload data phase is the metric http_req_waiting{op:blob_upload,phase:data}. Both blob upload values are less than the 150 ms threshold, and both manifest_put checks passed. Each run used one runner instance, so the difference between the two runs is noise. At 5 virtual users, no regression is measurable. The runs sampled no database lock statistics. The MR author accepts this cost for the closed beta.

The k6 runs above used the head fd209a75e from before the split. The split did not change the code that they measured.

Merge order

Merge this MR after !3378 (merged) and !3379 (merged). The stack targets enforce this order: GitLab retargets this MR when the MR below it merges. This MR widens no predicate ahead of the code that serves it: three paths now refuse where they answered success before.

The merged-tree check at head fd209a75e against the !3317 (merged) head 960a04162 found text conflicts in two paths. After the split, they fall on these MRs:

  • internal/format/oci/upload_test.go: this MR. Both branches add new tests after the last function of that file.
  • docs/dev/configuration-reference.md: !3379 (merged) and this MR. One hunk covers the storage.max_concurrent_upload_sessions row, which !3317 (merged) rewrites, and the storage.upload_session_wait_timeout row beside it, which both MRs of this stack change.

The MR that merges second needs a rebase.

Still open in issue 1122: the npm dist-tag PUT (part 3), the npm deprecate (part 4) and the management-API upsertContainerTag (part 5).

Related to #1122 (closed)

🤖 Generated with Claude Code

Edited by Pawel Rozlach

Merge request reports

Loading
Loading