fix(oci): fence the image upsert on the upload session open (#1122 part 2 stack: 3/3)
What this delivers
This MR is part 2c of issue 1122, and the last MR of a stack of three:
- The repository-fenced image upsert in the datastore: !3378 (merged).
- The initiate write deadline: !3379 (merged), which targets the branch of !3378 (merged).
- This MR: the fence on the OCI write paths. It targets the branch of !3379 (merged).
This MR closes the repository tombstone window on the three write paths that open an upload session: the blob upload initiate, the single-request POST ?digest=, and the fallback arm of the cross-repository blob mount.
On a repository that was tombstoned after the resolve, each path now answers the repository-tier 404 NAME_UNKNOWN and writes no container_images row and no upload session.
Cause
CreateSession wrote the image row with UpsertContainerImage as one statement on the pool handle, outside a transaction.
Nothing read the repository row again between the resolve and that write.
Also, writeSessionOpenError had no arm for datastore.ErrRepositoryConcurrentlyDeleted.
As a result, a tombstone that committed after the resolve let the initiate and the mount fallback answer 202, and let the single-request POST answer 201.
Each of these requests left a container_images row for the purge to remove.
The fix
- The wire adapter
containerImageUpsertersends everyoci.ImageUpsertercall toContainerImageStore.UpsertContainerImageInLiveRepository, the method that !3378 (merged) adds. As a result, the image upserts in the finalize (CompleteUpload) and in the mount (MountBlob) are fenced too.oci.ImageUpserteranddatastore.UpsertContainerImagekeep their signatures. writeSessionOpenErroranswers the sentinel with the repository-tier404 NAME_UNKNOWN, before the caller-gone check. The404is the same masked answer that a name that was never held gets.CreateSession,CompleteUploadandMountBlobrun the image upsert on a context of its own, bounded atlifecycle.TombstoneTimeout. Past that bound, the answer is500, or499when the client is gone.mountWriteTimeoutnow adds the bound of the upsert:lifecycle.TombstoneTimeout + counterEmitTimeout + blobLinkTimeout + 5s. The fallback arm of the mount opens a session under the same budget, so it spends the 10s upsert bound plus the admission-gate wait ofstorage.upload_session_wait_timeout. At the 15s ceiling of that field, the fallback arm fits with 5s left, becausecounterEmitTimeout + blobLinkTimeoutis 15s.TestMountWriteTimeoutCoversTheImageUpsertBoundasserts that fit, so a higher ceiling fails the test.- The initiate write deadline from !3379 (merged) now adds the upsert bound too:
lifecycle.TombstoneTimeoutplusstorage.upload_session_wait_timeoutplus a 5s tail (operator ruling q29, option B). At the 10s example value ofserver.timeouts.write, an upsert that used its full bound wrote its500after the deadline. - The mount fault log line now reads
oci mount: destination image write or blob link failed, and its499sibling changes the same way. The old text,link source blob failed, named the wrong step when the destination image upsert failed.
Documentation changed with the code
- S20-A: the tombstone-window text no longer says "until the initiate is re-verified too".
The Error Cases row now states the
uploadLockWaitTimeoutbound of the Maven upload, in place of the claim that the Maven upload has no deadline. The row on a container blob link that parks on therepositoriesrow lock now names every arm that parks, the image-upsert wait and its bound, and the write deadline of each arm. It also names which arms hold an upload admission unit (operator ruling q28, option C), and it names the Maven lock asFOR SHARE OF repositories. - S12: the mount status table, the mount sequence and the SQL step 3 comment match the new answer.
The
499row names the source blob lookup, the destination image write and the blob link. Upload Read Deadlines gives the four-term mount budget, adds the image-write bound to the initiate budget, and names the fallback arm and its fit at the 15s wait ceiling. The mount fallback paragraph counts four session-open answers, and it names the503as the one answer that the status table does not list. api/openapi/v1.yaml(deleteRepository) andapi/bruno/management-api/repositories/delete-repository.brunow list the blob upload initiate, and both arms of the mount, as paths that re-verify the repository.docs/dev/configuration-reference.md: theserver.timeouts.writerow gives the four-term mount budget and adds the 10-second image-write bound to the initiate budget. It says that the five read-timeout keys replace the write deadline with their own value. The row also names the fallback arm of the mount: it spends the 10-second image-write bound plusstorage.upload_session_wait_timeout, which leaves 5 seconds at the 15-second ceiling. Thestorage.upload_session_wait_timeoutrow adds the image-write bound to the initiate deadline.
Spec coverage
Rows come from the spec, S20-A (## Acceptance Criteria, ## Error Cases, ## Security Considerations), plus one table for the card's part 2 acceptance items.
Spec: docs/specs/S20-a-lifecycle-closed-beta.md Unit: issue 1122 part 2 (blob upload initiate, B2 image fence). "Other owner" rows are outside this unit: merged S20-A steps, the part 1 MR (f6577278), or parts 3 to 5 of this issue.
Acceptance criteria
| # | Criterion | Tests |
|---|---|---|
| AC-1 | Retention-window discovery of repository tombstones | Other owner: S20-A purger steps (merged). |
| AC-2 | A live row is never a purge root | Other owner: S20-A purger steps (merged). |
| AC-3 | Four scans use the tombstone index and prune to one partition | Other owner: S20-A database review (merged). |
| AC-4 | Version-level scans use the tombstone indexes | Other owner: S20-A database review (merged). |
| AC-5 | Purge re-run is a no-op; an interrupted purge resumes | Other owner: S20-A purger steps (merged). |
| AC-6 | npm walk reaps live and tombstoned rows in one pass | Other owner: S20-A purger steps (merged). |
| AC-7 | Every purge transaction is bounded | Other owner: S20-A purger steps (merged). |
| AC-8 | Purge job in a frozen namespace refuses | Other owner: S20-A purger steps (merged). |
| AC-9 | Tombstoned npm version is reaped | Other owner: S20-A npm reap step (merged). |
| AC-10 | Unpublish then republish leaves no orphan | Other owner: S20-A npm reap step (merged). |
| AC-11 | Tombstoned npm package is reaped with tags and metadata | Other owner: S20-A npm reap step (merged). |
| AC-12 | DeleteNpmMetadataFiles removes attachments | Other owner: S20-A npm reap step (merged). |
| AC-13 | Tombstoned Maven version and package are reaped | Other owner: S20-A Maven reap step (merged). |
| AC-14 | Container repository reaped through the per-artifact deleters | Other owner: S20-A container reap step (merged). |
| AC-15 | Image index reaped parent-first | Other owner: S20-A container reap step (merged). |
| AC-16 | Tombstoned image in a live repository is reaped | Other owner: S20-A container reap step (merged). |
| AC-17 | Container walk reaps live and tombstoned images | Other owner: S20-A container reap step (merged). |
| AC-18 | Remote-cache subtree is reaped | Other owner: S20-A remote reap step (merged). |
| AC-19 | container_remote_manifests and container_remote_blobs are reaped | Other owner: S20-A remote reap step (merged). |
| AC-20 | Reaped blobs keep zero attachments | Other owner: S20-A purger steps (merged). |
| AC-21 | DELETE without destructive answers 400 |
Other owner: S20-A entry-point step (merged). |
| AC-22 | destructive=false on an empty repository answers 204 |
Other owner: S20-A entry-point step (merged). |
| AC-23 | destructive=false on a non-empty repository answers 409 |
Other owner: S20-A entry-point step (merged). |
| AC-24 | The 409 message names the remedy | Other owner: S20-A entry-point step (merged). |
| AC-25 | destructive=true on a non-empty repository answers 202 and enqueues |
Other owner: S20-A entry-point step (merged). |
| AC-26 | destructive=true on an empty repository answers 204 |
Other owner: S20-A entry-point step (merged). |
| AC-27 | After the 202, every client route naming the repository answers 404 | Serial, passes today: TestTombstonedRepositoryIntegration_OCIRoutes404 (row POST blobs/uploads). Tombstone committed after resolve, new: TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert (initiate, single_post, mount_fallback). The from= sub-clause, passes today: TestBlobMountLivenessRace_TombstonedSourceKeepsIts202. npm routes: parts 3 and 4. |
| AC-28 | Create at a tombstoned name: 409, then 201 | Other owner: S20-A entry-point step (merged). |
| AC-29 | The finalizer cannot succeed early | Other owner: S20-A finalizer step (merged). |
| AC-30 | The finalizer removes the row, the child row and the links | Other owner: S20-A finalizer step (merged). |
| AC-31 | Sweep re-queues within one interval | Other owner: S20-A sweep step (merged). |
| AC-32 | Sweep reads oldest-first | Other owner: S20-A sweep step (merged). |
| AC-33 | Orphaned npm_packages tombstone is enumerated | Other owner: S20-A sweep step (merged). |
| AC-34 | No re-queue while a live job exists | Other owner: S20-A sweep step (merged). |
| AC-35 | Re-queue after a terminal job | Other owner: S20-A sweep step (merged). |
| AC-36 | A permanent purge failure stays counted and re-queued | Other owner: S20-A sweep step (merged). |
| AC-37 | The sweep registers its worker | Other owner: S20-A sweep step (merged). |
| AC-38 | A frozen namespace is not enqueued by the sweep | Other owner: S20-A sweep step (merged). |
| AC-39 | FindByNameInNamespace hides a tombstoned repository (docker, oci) | Other owner: S20-A visibility step (merged). |
| AC-40 | RepositoryStore.FindByID hides a tombstoned repository | Other owner: S20-A visibility step (merged). |
| AC-41 | RepositoryStore.Update refuses a tombstoned repository | Other owner: S20-A visibility step (merged). |
| AC-42 | Delete, decrements, MarkRepositoryLastUpdated and Missing still act | Other owner: S20-A visibility step (merged). |
| AC-43 | parentRepositoryIsActive fails on a mid-transaction tombstone | Other owner: S20-A visibility step (merged). |
| AC-44 | The four container_images gates hide a tombstoned image | Other owner: S20-A image-tier step (merged). |
| AC-45 | /v2/ image-tier 404 on a tombstoned image |
Other owner: S20-A image-tier step (merged). |
| AC-46 | Mount answers 202 when every source image is tombstoned | Other owner: S20-A image-tier step (merged). This unit changes the fallback only on a tombstoned destination repository; the live-destination 202 stays pinned by TestInitiateLivenessRace_RolledBackTombstoneOpensTheSession/mount_fallback. |
| AC-47 | A tombstoned image named in from= still mounts with 201 |
Other owner: S20-A image-tier step (merged). |
| AC-48 | Mount destination: image tier 201; repository tier answered by the Error Cases row | Repository tier, new: TestMountFallbackDestinationTombstonedReturns404 (source_missing, source_denied), TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert/mount_fallback. Image tier 201: other owner, unchanged. Dropping "until the initiate is re-verified too" from the text: implementer (docs). |
| AC-49 | UpsertContainerImage does not resurrect a tombstoned image | Other owner: S20-A image-tier step (merged), TestContainerImageStore_Upsert_WithASoftDeletedSibling. The new method delegates to the same upsert. |
| AC-50 | ContainerImageStore.Delete still acts on a tombstoned row | Other owner: S20-A image-tier step (merged). |
| AC-51 | Destructive DELETE emits no delta at the tombstone | Other owner: S20-A accounting step (merged). |
| AC-52 | Size and component decrements fire at the purger | Other owner: S20-A accounting step (merged). |
| AC-53 | Every emit site commits before it emits | Other owner: S20-A accounting step (merged). |
| AC-54 | A purge chunk emits from its committed aggregate | Other owner: S20-A accounting step (merged). |
| AC-55 | No raw foreign-key cascade | Other owner: S20-A accounting step (merged). |
| AC-56 | The three index migrations are in structure.sql | Other owner: S20-A schema step (merged). |
| AC-57 | No column added or dropped, no CHECK on soft_deleted_at | Other owner: S20-A schema step (merged). |
| AC-58 | unique_container_images_ns_id_cr_id_name is partial | Other owner: S20-A schema step (merged). |
| AC-59 | Config accepts and rejects retention_window and sweep_interval | Other owner: S20-A config step (merged). |
| AC-60 | Maven metadata tombstone discovered with its package live | Other owner: S20-A Maven metadata step (merged). |
| AC-61 | Maven metadata retention-window filter | Other owner: S20-A Maven metadata step (merged). |
| AC-62 | Maven metadata partial index and its migration | Other owner: S20-A Maven metadata step (merged). |
| AC-63 | Republished Maven metadata survives a purge | Other owner: S20-A Maven metadata step (merged). |
| AC-64 | Concurrent Maven metadata purges count once | Other owner: S20-A Maven metadata step (merged). |
| AC-65 | Tombstone gauge includes Maven metadata | Other owner: S20-A Maven metadata step (merged). |
Error cases
| # | Condition | Tests |
|---|---|---|
| E-1 | DELETE omits destructive |
Other owner: S20-A entry-point step (merged). |
| E-2 | destructive is not true or false |
Other owner: S20-A entry-point step (merged). |
| E-3 | destructive=false on a non-empty repository |
Other owner: S20-A entry-point step (merged). |
| E-4 | The hard-delete phase parks on the row lock | Other owner: S20-A entry-point step (merged). |
| E-5 | A container write parks on the repositories row lock: each park bounded |
Image-upsert park, new: TestContainerImageStore_UpsertContainerImageInLiveRepository_HonorsCallerDeadline, TestCreateSession_BoundsTheImageUpsertAtTombstoneTimeout, TestCompleteUpload_BoundsTheImageUpsertAtTombstoneTimeout, TestMountBlob_BoundsTheImageUpsertAtTombstoneTimeout, TestMountWriteTimeoutCoversTheImageUpsertBound. Past the bound 500, passes today: TestUploadPOST_UpsertErrorReturns500; 499 with the client gone, passes today: TestUploadShed_CallerGoneTakes499. LinkBlob park: other owner, part 1 (merged). |
| E-6 | An artifact published mid-DELETE | Other owner: S20-A entry-point step (merged). |
| E-7 | A write resolved before the tombstone commits after it | Initiate, single-request POST and mount fallback, new: TestContainerImageStore_UpsertContainerImageInLiveRepository_RepositoryGate, _TombstonedMidTransaction, _RolledBackTombstoneUpserts, _RunsAtReadCommitted, TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert, TestInitiateLivenessRace_RolledBackTombstoneOpensTheSession, TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown, TestSessionOpen_TombstoneSentinelAnswers404AfterClientHangup, TestMountFallbackDestinationTombstonedReturns404, TestContainerImageUpserter_ProjectsRowAndWrapsError. Finalize and mount sentinel arms, passes today: TestCompleteUpload_SurfacesRepositoryConcurrentlyDeleted, TestMountBlob_SurfacesRepositoryConcurrentlyDeleted, TestUploadFinalize_RepositoryTombstonedMidLinkReturns404NameUnknown. Live 202 and 201, passes today: TestUploadPOST_InitiateReturns202WithLocation, TestUploadPUT_FinalizeReturns201, TestUploadPOST_SingleShotReturns201, TestMountSucceedsReturns201. Maven-deadline sentence: implementer (docs). npm dist-tag PUT, npm deprecate, upsertContainerTag: parts 3, 4 and 5. |
| E-8 | A package tombstone mid Maven publish | Other owner: S20-A Maven step (merged). |
| E-9 | DELETE on a missing or tombstoned repository | Other owner: S20-A entry-point step (merged). |
| E-10 | Create at a name held by a tombstoned repository | Other owner: S20-A entry-point step (merged). |
| E-11 | Any client route naming a tombstoned repository answers 404 in its own vocabulary | Serial, passes today: TestTombstonedRepositoryIntegration_OCIRoutes404. Session-open race, new: TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert (NAME_UNKNOWN code, message and detail against the unknown-name baseline), TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown, TestMountFallbackDestinationTombstonedReturns404. |
| E-12 | A mount whose from= names a tombstoned repository answers 202 |
Passes today: TestBlobMountLivenessRace_TombstonedSourceKeepsIts202. Unchanged by this unit. |
| E-13 | Purge job: 23503 on a parent delete | Other owner: S20-A purger steps (merged). |
| E-14 | Purge job: manifest refused as referenced | Other owner: S20-A purger steps (merged). |
| E-15 | Purge job: the finalizer aborts | Other owner: S20-A finalizer step (merged). |
| E-16 | Purge job: repository already hard-deleted | Other owner: S20-A purger steps (merged). |
| E-17 | Purge job: retention window changed | Other owner: S20-A purger steps (merged). |
| E-18 | Sweep: enumeration query fails | Other owner: S20-A sweep step (merged). |
| E-19 | Sweep: interval elapses mid-enumeration | Other owner: S20-A sweep step (merged). |
| E-20 | Emit API unavailable | Other owner: S20-A accounting step (merged). |
| E-21 | Parent gate: parent tombstoned before the child INSERT | Other owner: S20-A visibility step (merged). |
| E-22 | A tombstoned coordinate is reoccupied before the reap | Other owner: S20-A npm reap step (merged). |
Security considerations
| # | Concern | Tests |
|---|---|---|
| S-1 | Authorization is unchanged | Other owner: S09. This unit adds no authorization path. Not tested in this unit. |
| S-2 | destructive is a confirmation, not a control |
Other owner: S20-A entry-point step (merged). |
| S-3 | Irreversibility is documented | Other owner: S20-A docs (merged). |
| S-4 | Input validation | Other owner: S20-A entry-point step (merged). Zero-id guard on the new method: TestContainerImageStore_UpsertContainerImageInLiveRepository_RepositoryGate (zero id). |
| S-5 | No injection surface is added | The new method reuses the existing parameterized statements and adds none. Not tested in this unit. |
| S-6 | Error messages do not become an existence oracle | New: TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert (byte-identical to the unknown-name answer, modulo the requested name), TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown, TestMountFallbackDestinationTombstonedReturns404. |
| S-7 | Purge failures are visible | Other owner: S20-A sweep step (merged). |
Card part 2 acceptance
| # | Item | Tests |
|---|---|---|
| A-1 | Initiate on a repository tombstoned after resolve: 404 NAME_UNKNOWN, no image row, no session | TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert/initiate, TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown |
| A-2 | Mount fallback with a tombstoned destination: 404 | TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert/mount_fallback, TestMountFallbackDestinationTombstonedReturns404 |
| A-3 | Single-request POST on a tombstoned repository: 404 at session open | TestInitiateLivenessRace_RepositoryTombstonedAtImageUpsert/single_post, TestSessionOpen_RepositoryTombstonedAtImageUpsertReturns404NameUnknown |
| A-4 | The new method waits; commit gives the sentinel and no row; rollback inserts | TestContainerImageStore_UpsertContainerImageInLiveRepository_TombstonedMidTransaction, _RolledBackTombstoneUpserts, TestInitiateLivenessRace_RolledBackTombstoneOpensTheSession |
| A-5 | The new method refuses a pool handle | Open question Q1: the new method takes no handle. The guard it calls is pinned by TestContainerRepositoryStore_ReVerifyRepositoryAlive_ArgumentGuards (row connection pool), passes today. |
| A-6 | The new method opens its transaction at READ COMMITTED | TestContainerImageStore_UpsertContainerImageInLiveRepository_RunsAtReadCommitted (repeatable-read session default) |
| A-7 | The wait is bounded by TombstoneTimeout; past it 500, or 499 when the client is gone | TestContainerImageStore_UpsertContainerImageInLiveRepository_HonorsCallerDeadline, TestCreateSession_BoundsTheImageUpsertAtTombstoneTimeout; 500 and 499 pass today: TestUploadPOST_UpsertErrorReturns500, TestUploadShed_CallerGoneTakes499 |
| A-8 | Finalize and mount image upserts bounded on their own context; sentinel reaches the 404 arms; mountWriteTimeout includes the bound | TestCompleteUpload_BoundsTheImageUpsertAtTombstoneTimeout, TestMountBlob_BoundsTheImageUpsertAtTombstoneTimeout, TestMountWriteTimeoutCoversTheImageUpsertBound; sentinel arms pass today: TestCompleteUpload_SurfacesRepositoryConcurrentlyDeleted, TestMountBlob_SurfacesRepositoryConcurrentlyDeleted, TestUploadFinalize_RepositoryTombstonedMidLinkReturns404NameUnknown |
| A-9 | Sentinel answers 404 regardless of the hangup check | TestSessionOpen_TombstoneSentinelAnswers404AfterClientHangup |
| A-10 | Live repository: 202 Range 0-0 on initiate, 201 on finalize and mount | Pass today: TestUploadPOST_InitiateReturns202WithLocation, TestUploadPUT_FinalizeReturns201, TestUploadPOST_SingleShotReturns201, TestMountSucceedsReturns201; through the fence: TestInitiateLivenessRace_RolledBackTombstoneOpensTheSession, TestContainerImageStore_UpsertContainerImageInLiveRepository_RepositoryGate |
| A-11 | oci.ImageUpserter and UpsertContainerImage keep their signatures | Compile-time: the unchanged fakes in upload_test.go, emit_gate_internal_test.go, upload_finalize_integration_test.go build against them. |
| A-12 | "until the initiate is re-verified too" removed; oci.md and S12 match | Docs only. Implementer. |
| A-13 | S20-A :1055 states the Maven uploadLockWaitTimeout bound | Docs only. Implementer. |
The tables cover the whole unit, that is, all three MRs of the stack.
These tests in the tables land in !3378 (merged): TestContainerImageStore_UpsertContainerImageInLiveRepository_RepositoryGate, _TombstonedMidTransaction, _RolledBackTombstoneUpserts, _RunsAtReadCommitted and _HonorsCallerDeadline.
All other tests in the tables land in this MR.
Tests changed after the coverage table
The table above was written in the test-first commit of the unit, before the split. Later commits changed these tests, and the third column names the MR that carries each one:
| Test | Change | MR |
|---|---|---|
TestUploadPOST_UpsertWaitPastBoundReturns500Or499 |
Added. A context error out of the image upsert through the initiate answers 500 INTERNAL under a live caller, and 499 with no body when the caller hung up during the upsert (A-7, E-5). |
This MR |
TestUpload_InitiateArmsWriteDeadline |
Added. The initiate arms its own write deadline, sized from the configured session wait, and arms no read deadline. This MR adds the assertion that the deadline outlasts lifecycle.TombstoneTimeout plus the wait. |
!3379 (merged), this MR |
TestUpload_InitiateSetWriteDeadlineFailureReturns500 |
Added. A connection that refuses the write deadline answers 500 before the image upsert runs. |
!3379 (merged) |
TestInitiateWriteTimeoutCoversTheUpsertAndTheSessionWait |
Renamed from TestInitiateWriteTimeoutCoversTheSessionWait (!3379 (merged)). The initiate budget is more than lifecycle.TombstoneTimeout plus the wait, for a configured, a zero, a negative and an unset wait. |
This MR |
TestUpload_BodylessPathsSetNoDeadline |
Changed. The POST initiate row is removed, because the initiate now arms a write deadline. |
!3379 (merged) |
TestMountWriteTimeoutCoversItsOwnBoundedWork |
Removed. It asserted two bounds, and TestMountWriteTimeoutCoversTheImageUpsertBound asserts all three. |
This MR |
TestMountWriteTimeoutCoversTheImageUpsertBound |
Changed. It also asserts that mountWriteTimeout is at least lifecycle.TombstoneTimeout plus 15s plus 5s, which is the fallback arm at the 15s ceiling of storage.upload_session_wait_timeout (E-5, A-8). |
This MR |
TestBuildOCIStoreHandlers_ThreadsUploadSessionWait |
Added. The composition root builds the OCI handlers with a 12s session wait, and the upload handler reports 12s through the UploadSessionWait accessor. |
!3379 (merged) |
TestContainerImageStore_UpsertInLiveRepository_NilContext |
Added. UpsertContainerImageInLiveRepository returns errContainerImageNilContext for a nil context, before it reads the database. |
!3378 (merged) |
TestContainerImageUpserter_ProjectsRowAndWrapsError |
Changed. The pool-call counter, the fake's pool-handle UpsertContainerImage and the NotPanics wrappers are removed, because the narrowed store interface makes the call that they guarded impossible. The fenced-call and argument assertions stay. |
This MR |
TestMountLinkErrorAfterClientHangupReturns499, TestMountLinkErrorWithLiveClientStays500 |
Changed. They assert the new destination image write or blob link failed log text. |
This MR |
e2e scenario catalog
The e2e.oci.setup.delete-repository row in docs/testing/e2e/oci.md now says that a mount whose destination is tombstoned and that takes the fallback arm answers 404 NAME_UNKNOWN at the repository tier, with no Location header.
The row gives the cause as the mechanism, not as an assertion: the fallback writes the image row through a transaction that re-verifies the repository first, so the refused fallback writes no image row and opens no upload session.
This MR adds no row, because the existing row covers the delete-repository journey that changed.
User documentation
No page under docs/user/ describes the race between a repository delete and a write that is in flight, or the answers that the three session-open paths give in that race.
Thus, docs/user/ does not change.
Size
This MR is 1,064 changed lines (876 added, 188 deleted) in 15 files, measured against the head of !3379 (merged) (1cd0a37bc).
| File group | Files | Added | Deleted | Total |
|---|---|---|---|---|
Production Go (internal/, cmd/) |
3 | 66 | 96 | 162 |
| Go tests | 6 | 738 | 42 | 780 |
| Specs, API contracts and docs | 6 | 72 | 50 | 122 |
The production change is 162 lines.
The tests are 73% of the diff, and two test files hold 531 of those lines: internal/format/oci/initiate_liveness_race_test.go (290, new) and internal/format/oci/upload_test.go (241).
This MR does not split further:
- The adapter switch fences every caller at the same time, so the
404arm, the upsert bounds and the largermountWriteTimeoutmust land with it. - The new tests assert the new
404answers, so they cannot merge before the switch that gives those answers. - The upsert-bound comments and the
ImageUpserterdoc describe a row-lock wait that exists only after the switch.
Split validation
Before the split, this MR had head 6b8b295e8.
The stack !3378 (merged) → !3379 (merged) → this MR has head 47bd9b5d7.
git diff 6b8b295e8 47bd9b5d7 is empty: both commits have the tree 890ff79a9c358c810686476e254ee6d6a3abd711.
Thus, the three MRs merged in order give the same code as the earlier head of this MR.
The intermediate states also build:
| Head | Content | go build, go vet (also with -tags integration) |
Pre-commit hooks |
|---|---|---|---|
e20353dc0 |
!3378 (merged) | Pass | Pass |
1cd0a37bc |
!3378 (merged) + !3379 (merged) | Pass | Pass |
47bd9b5d7 |
Full stack | Pass | Pass, on the second run. The first run failed in internal/virtual, a package that the stack does not touch: TestResolve_Integration_ConcurrentResolutions_CoalesceToOneUpstreamGET fails under go test -count=3 on this tree and passes on a single run. |
Verification
The OCI conformance suite did not run locally.
db:setup failed on a port conflict on host port 5432 before the suite started.
The CI conformance:oci:* jobs cover the suite on this MR.
Database review
!3378 (merged) holds the database review of UpsertContainerImageInLiveRepository: its three statements, their locks and the earlier plans.
This MR adds the callers on the request path.
Before this MR, the upsert ran as one statement on the pool handle, outside a transaction.
Now each image upsert of the initiate, the single-request POST, the finalize and the mount runs in its own transaction, after the share lock on the parent repositories row.
The transaction adds a BEGIN, the re-verification and a COMMIT, so each upsert makes three more round trips.
The share lock waits behind a tombstone UPDATE of the same row, and the caller bounds that wait at lifecycle.TombstoneTimeout.
The cost of the share lock.
Each image upsert now takes a FOR SHARE lock on the parent repositories row, in a transaction of its own.
LinkBlob already took the same lock before this MR.
Thus, a pushed layer takes this lock 3 times: in the initiate upsert, in the finalize upsert and in the link.
A mount takes it 2 times: in the destination image upsert and in the link.
The k6 push_small scenario ran in the test:perf:caproni job on the branch and on main, with 5 virtual users for 20 s against one repository.
| Run | Commit | Pipeline | Job | Blob upload data phase, p95 | manifest_put, p95 |
|---|---|---|---|---|---|
| Branch | fd209a75e |
2956117669 | 17241442123 | 96.37 ms | 84.68 ms |
main |
4b95807b9 |
2955951868 | 17241499561 | 111.97 ms | 99.67 ms |
The blob upload data phase is the metric http_req_waiting{op:blob_upload,phase:data}.
Both blob upload values are less than the 150 ms threshold, and both manifest_put checks passed.
Each run used one runner instance, so the difference between the two runs is noise.
At 5 virtual users, no regression is measurable.
The runs sampled no database lock statistics.
The MR author accepts this cost for the closed beta.
The k6 runs above used the head fd209a75e from before the split.
The split did not change the code that they measured.
Merge order
Merge this MR after !3378 (merged) and !3379 (merged). The stack targets enforce this order: GitLab retargets this MR when the MR below it merges. This MR widens no predicate ahead of the code that serves it: three paths now refuse where they answered success before.
The merged-tree check at head fd209a75e against the !3317 (merged) head 960a04162 found text conflicts in two paths.
After the split, they fall on these MRs:
internal/format/oci/upload_test.go: this MR. Both branches add new tests after the last function of that file.docs/dev/configuration-reference.md: !3379 (merged) and this MR. One hunk covers thestorage.max_concurrent_upload_sessionsrow, which !3317 (merged) rewrites, and thestorage.upload_session_wait_timeoutrow beside it, which both MRs of this stack change.
The MR that merges second needs a rebase.
Still open in issue 1122: the npm dist-tag PUT (part 3), the npm deprecate (part 4) and the management-API upsertContainerTag (part 5).
Related to #1122 (closed)