fix(config): lower the storage.s3.chunk_size default to 5MiB
Stacked MRs
| Part | MR | Scope | Reviewable LOC | Target |
|---|---|---|---|---|
| 1/2 | !3047 (merged) (merged) | clamp the S3 upload buffer, one slot ratio for both drivers | 451 | main |
| 2/2 | this MR | lower the storage.s3.chunk_size default to 5MiB |
198 | main |
!3047 (merged) merged on 2026-10-12, so this MR is rebased onto main and targets it. This MR's 5MiB default derives a cap of 28 only with part 1's 1.2 ratio. On the old ratio of 2 it derives 18. LOC is added plus removed, less generated .pb.go files: part 1 at its squash commit 55fea6ee4, this MR at 1b0190c1c..39484bdc2.
Summary
At the 64 MiB default the derived upload-session cap on S3 is 2. So a third concurrent Maven deploy, npm publish, remote cache fill or single-POST blob upload waits 5s and gets a 503, and one npm packument rebuild takes both slots. A docker push joins that count once the cap plan's ResumeSession acquire (its step 12) lands, and one push has five layer uploads.
This MR lowers the default to 5MiB, the AWS per-part minimum. Under the 400 MiB fallback that derives 28, the same as the GCS default, and clears the 25 sessions of five concurrent pushes. 6MiB would derive 24.
On S3 it also refuses, at startup, a container.blob_max_size above 10,000 x chunk_size, the largest object one S3 multipart upload holds. The lower default brings that limit down to 52.4 GB, and without the check a larger cap loaded cleanly and a push failed only at part 10,001.
It also updates every place outside docs/plans/ and docs/adr/ that stated the old S3 figures: the configuration reference, docs/dev/storage.md, config.example.yaml, the run recipe, the S3Config.chunk_size proto comment, the caproni rig's comment on the cap it sets, the rendering-ceiling paragraph in internal/format/npm/packument_cache.md, and three npm test names and comments that called 2 "the S3 default cap".
Trade-offs of 5MiB parts:
- Object size. S3 accepts at most 10,000 parts per upload, so
5MiBparts cap an object at 52.4 GB. That clears the50GBcontainer.blob_max_sizedefault (9,537 parts). Startup refuses acontainer.blob_max_sizeabove 10,000 xchunk_sizeon S3 and names both values, so an install that raised it past 52.4 GB fails at boot, not at part 10,001 of a push. - Request count. About 13x more part uploads per byte. The S3 request limiter (
max_requests_per_second, 350 by default) then caps one process's part uploads near 1.7 GiB/s. - Resumes. A resumed upload whose last part is short, like a multi-
PATCHOCI upload, re-copies its object on the server inchunk_sizeparts. At5MiBthat is about 13x moreUploadPartCopycalls per resume. A client that sends a layer in severalPATCHrequests pays it on each one. - Shared request limiter. The re-copy runs up to
multipart_copy.max_concurrency(100)UploadPartCopycalls at once, and each waits on the one S3 request limiter the process holds (max_requests_per_second, 350 by default). So a re-copy delays every other S3 call on that pod, for every tenant. Example: a 10 GB blob sent in ten 1 GBPATCHrequests re-copies 45 GB in total, about 675 calls and 2 s of limiter time at64MiB, and about 8,600 calls and 25 s at5MiB. The cost grows with the square of thePATCHcount. The hazard exists onmain, and this default makes it about 13x cheaper to trigger. It is accepted here, and #1589 (closed) tracks re-copying inmultipart_copy.chunk_sizeparts instead. - Explicit settings stay. An install that sets
chunk_sizekeeps its value.config.example.yamlused to set"64MiB"explicitly and now sets"5MiB".
Governing ADRs
scripts/adr-freshness.sh: the mirror is current.
- ADR 003 (
docs/adr/003_system_requirements.md): its storage-throughput paragraph says the S3 driver at its shipped defaults "admits too few sessions to reach the projection". With this MR the S3 default admits what the GCS default does, so it conforms, and that sentence goes stale (see below). - ADR 004 (
docs/adr/004_data_and_application_limits.md): container blobs up to 50 GB.5MiBparts hold 52.4 GB, so the ADR's limit still fits. Maven and npm are 5 GB.
Testing
- Config tests: the S3 default loads as
5242880bytes and derives 28. Setting the default back to 64 MiB reddens both. - The S3 part-limit check: one byte past 10,000 parts of the
5MiBdefault is refused, the exact figure loads, a64MiBchunk_sizeholds600GB, and GCS is not bound by it. A unit test pins that 10,000 parts of the S3 default hold the50GBblob default. Removing the check, changing>to>=, dropping the S3-only guard, or raising the blob default to60GBeach turns its test red. - The renamed npm tests pass (
TestSetRebuildRenderingCap_ClampsTheCeilingDownNeverUp,TestRebuildStreamedKinds_ReturnsBothUnitsWhenTheWindowFails,TestReserveRenderingSlot_ShedRecordsTheCeilingInForce). - Unit tests pass for
cmd/artifact-registry,internal/config,internal/format/npm,internal/storage/...,internal/accountingandinternal/managementapi, the packages that loadconfig.example.yamlor read the cap. driver.sh smokeatdd06f688a: 103 passed, 0 failed. Its boot line readsdrivers3,chunk_size_bytes5242880,max_concurrent_upload_sessions28,floor_boundfalse. The startup check landed after that run. The recipe sets noblob_max_size, so it boots on the50GBdefault, which 10,0005MiBparts hold.golangci-lint2.14 reports 0 issues oninternal/configandinternal/format/npm, and the full pre-commit chain, including the comment-caps gate andbuf generate freshness, passes on every file the branch changes.
The run recipe changes in this MR because guardrail 20 asks for it in the same MR as the boot change. It is a one-sentence fact correction, so it carries no skill benchmark.
No e2e scenario is added or affected: e2e.maven.publish.upload-slots-exhausted sets its own cap of 2, and no scenario relies on the S3 default. No docs/user/ page covers storage.* settings, so no user documentation is affected.
Not changed here
npm.max_tarball_sizehas no upper bound and is not checked against the S3 part limit: the startup check coverscontainer.blob_max_sizeonly. Maven'smax_artifact_sizeis capped at 5 GB, which any S3chunk_sizeholds.- ADR 003's S3 shortfall paragraph needs a handbook ADR update once this stack lands.
docs/plans/2026-09-03-upload-session-concurrency-cap.mdstill records the 64 MiB default and cap of 2 as its basis, including the Step 12 arithmetic part 1 repriced for the 1.2 ratio. Left alone: this MR is not a step of that plan.
Closes #1171 (closed)