feat(storage): cap concurrent upload sessions, part 1 of 3 (upload-session-concurrency-cap plan: 11/12)
Turns the upload-session admission gate on at all five NewSession sites. Every open charges one unit and hands back a release the calling function defers, so the sessions this process opens are bounded by storage.max_concurrent_upload_sessions. The chunked OCI upload is charged only for its initiating POST; see What this part does not reach. A refused open answers ErrUploadSessionsBusy, which each format renders as a retryable 503.
Part 1 of a three-MR stack. It carries every production line, the docs and sidecar .md edits, the test edits the signature change forces, the gate's own white-box cases, the stack identity assertions, and one test a review round added (TestFetcher_Fetch_CancelPanic_StillReleasesTheAdmissionUnit). The per-site release coverage lands in parts 2 and 3, and so do the test helpers it calls.
The stack, in merge order
| # | MR | Target | Reviewable LOC | Contents |
|---|---|---|---|---|
| gate | !2881 (merged) | main |
83 | plan amendment authorising this split (merged) |
| gate | !2905 (merged) | main |
275 | plan amendment re-pinning the soak load and moving the docker push items to Step 12 (open) |
| 1 | !2882 (merged) (this one) | main |
2,418 | production, docs, forced test edits |
| 2 | !2883 (merged) | part 1's branch | 1,228 | storage and npm test functions |
| 3 | !2884 (merged) | part 2's branch | 1,117 | OCI, remote fill, Maven and config test functions |
Reviewable LOC is added + removed, each row measured against its own target's merge base, at the heads 05408e2be, fb571a91a, 5a078d5b6, e9536ca98 and 368126733. The rows are read per part and never summed, because the nested targets make a sum double-count churn a later part rewrites. Part 2's figure is measured against ab6a259da and part 3's against 334a50ccf, the commits each is stacked on, so those two rows are re-derived once they rebase. Both will grow then: the test code f4d40e6d1 and 5a078d5b6 removed from this part moves to the part whose tests call it.
Merge order, and the gate on this MR
handbook!21192 and !2881 (merged) have merged. Merge !2905 (merged), then this MR, then !2883 (merged), then !2884 (merged).
!2905 (merged) is a merge gate on this MR, set as a merge request dependency. It pins the staging soak's load to the paths this part charges for the whole transfer, and moves the docker push arithmetic and the concurrent-push scenario from the plan's Step 11 to Step 12. The soak it pins runs on this branch, and its result is recorded on this MR before it merges.
handbook!21192 is a merge gate on this MR rather than a follow-up. The plan's ## Dependencies states that no step MR of this plan which turns the acquire on may merge before the handbook ADR-003 amendment has merged, and this MR is the one that turns it on. It merged on 2026-09-22 at 11:53 UTC, so that gate is clear.
!2883 (merged) targets this branch and !2884 (merged) targets !2883 (merged)'s, so GitLab will report them mergeable before this one. Merging them first would put their tests on a branch without the code under test. The order above is the only correct one.
Why this diff is 2,418 LOC
Past the 500-LOC ceiling in docs/dev/development-model.md by 1,918, and the plan's own forecast for the step was ~1,075. Split by file group:
| Group | LOC |
|---|---|
| Go production | 727 |
Go tests (_test.go) |
1,101 |
Go test support (internal/storage/testsuites, stub, storagetest) |
189 |
docs and sidecar .md |
401 |
The 727 production lines are what decide anything. Most of the 1,101 test lines are not new cases: they are the widened NewSession signature on every fake that implements storage.BlobStore, plus new rows and assertions inside test functions that already existed. The new test functions are the gate's white-box cases and benchmark, the two stack assertions, and the review-round test above. The 189 test-support lines are the shared suites, stubs and store helpers those tests call. A Go method signature changes for every implementer and every narrowed redeclaration in one commit or the tree does not build, so none of it can be staged.
The split this MR is part of is what keeps the reviewable surface this small. It goes no lower: the acquire and its release have to land in the same commit at every site, or main carries a cap that leaks.
What this part does not reach
The new leak-path coverage is not here. The admission integration suite, the per-site release tables and the config derivation case arrive in !2883 (merged) and !2884 (merged). Between this MR merging and !2884 (merged) merging, main carries the gate with the widened fakes, the gate's own white-box cases and the stack identity assertion, but not the per-site release coverage. That deviates from this step's Acceptance in the plan, and !2881 (merged) records the deviation and its window.
ResumeSession charges nothing. It takes no unit and hands back no release, and the OCI create arm releases when its 202 returns, so the PATCH and PUT chain a docker push stages its bytes through runs uncharged. The cap bounds the sessions this process opens, not every session holding a staging buffer. docs/dev/storage.md states this in the admission section, the max_concurrent_upload_sessions row in docs/dev/configuration-reference.md says the cap does not bound a docker push's layer uploads, and the resume acquire is Step 12's. This part therefore leaves out the docker push arithmetic, because it is not true until Step 12. !2905 (merged) moves it from the plan's Step 11 to Step 12.
There is no per-tenant fairness. The gate is one semaphore per pod, so one tenant holding units can refuse every other tenant's upload. The per-user and per-namespace control that ADR-004 calls "concurrent upload sessions" is a different limit and is not implemented.
A default S3 install floors at a cap of 2. The derivation gives 209715200 / 135266304 = 1, floored to 2, so two concurrent uploads saturate the gate and the third sheds after its wait. Lowering defaultS3ChunkSize and repricing the S3 per-slot reservation are owned by #1171 (closed); no line here touches either.
A configured cap below 2 is honoured as set. The loader refuses only a non-positive value, so max_concurrent_upload_sessions: 1 boots and then refuses every npm packument rebuild for the process's life, answering 503 inline_build_timeout on every cold packument read. Boot logs a warning naming both values, and it is the only boot-time signal: the refusal spends no wait budget and books no shed, and after boot each refused rebuild logs its own dropped warning. docs/dev/alerting.md alerts on the capacity gauge reading between 0 and 2 for exactly this.
One of the six fill-drain panic windows is closed here, and five are not. drainBlobFill (internal/format/oci/remote_blob.go:329) now also closes the fill body from a defer, so a panic inside the drain still returns the fill's admission unit. Its twin drainRemoteManifestFill (internal/format/oci/remote_manifest.go:306) and the four Maven fill consumers (two in remote_artifact.go, one each in remote_metadata.go and remote_sidecar.go) keep the old shape. All six belong to #1384, which stays open for the other five.
Observability
Four series ship: gitlab_artifact_registry_storage_upload_session_{slots_in_use,capacity,hold_duration_seconds,admission_sheds_total}. docs/dev/observability.md gains a catalog row for each, and docs/dev/alerting.md gains the three required alert rules with every family named in full, because a rule transcribed from a bare suffix selects nothing.
op="NewSession" on blobstore_call_duration_seconds now opens its span before the acquire, so it folds up to storage.upload_session_wait_timeout of gate parking into every outcome including ok. Both catalog rows say so.
e2e scenarios and user docs
One e2e scenario is added: e2e.maven.publish.upload-slots-exhausted in docs/testing/e2e/maven.md holds every slot with Maven deploys and asserts the refused deploy's 503, its Retry-After and the maven_upload shed count. The concurrent-push scenario needs the chunked PATCH and PUT to hold a unit, which Step 12 adds, so it lands there. docs/user/troubleshooting.md gains the 503 section covering installs and pulls as well as publishes, since the refusal is customer-visible on every upload path.
Verification
At 5a078d5b6: go vet, golangci-lint and the unit tests green for internal/config, internal/remote and cmd/artifact-registry, the tagged unused check clean on the last two, comment caps clean, and the full pre-commit chain passed on each of the three commits since f4d40e6d1.
At f4d40e6d1: go vet ./... clean untagged and with -tags=integration, golangci-lint run ./... 0 issues untagged, and the unit tests of every package that commit touches green. With --build-tags=integration, unused reports no issue on a line this MR adds; the one it reports across the module, credentialTablePartitionCount in internal/datastore/credential_table_integration_test.go, is already on main. CI never compiles the integration-tagged files, so that run was done by hand.
At 4dc727eff, the head before f4d40e6d1 removed test code: the full unit suite green, comment caps clean, gofmt clean on every changed Go file, and .claude/skills/run-artifact-registry/driver.sh smoke re-run from zero state on a fresh build: 74 passed, 0 failed.
Related to #1035