fix(config): lower the GCS chunk-size default and bound it at 64MiB
Stack
Merge in order. This MR targets !2183 (merged)'s branch, so its diff shows only its own changes.
| # | MR | What |
|---|---|---|
| 1 | !2183 (merged) | fix(storage): grow the upload buffer instead of sizing it to chunk_size, plus GOMEMLIMIT |
| 2 | this MR | fix(config): lower the GCS chunk_size default to the driver's own constant and bound it at 64MiB |
What
Two constants both claimed to be the GCS chunk-size default and disagreed by 12.8x:
internal/storage/driver/gcs/gcs.go:81,gcs.DefaultChunkSize= 5 MiB, documenting itself as "matching the Container Registry default".internal/config/storage.go:37,defaultGCSChunkSize= 64 MiB, and this one won.
It won because sizeOrDefault always substitutes a value for an unset field,
which also makes the chunkSize <= 0 fallback to the driver constant in
buildGCSDriver (cmd/artifact-registry/wire_storage.go:313-315) unreachable
in any configuration the loader produced.
The consequence worth fixing independently of !2183 (merged): the GCS conformance suite runs 5 MiB while production runs 64 MiB, so the size production actually uses was never exercised by the suite that certifies the driver.
The config default is now the driver's constant itself, so the two cannot diverge again.
The upper bound
storage.gcs.chunk_size gains a maximum, which the S3 equivalent has had all
along (s3ChunkSizeMax, 5 GiB, AWS's per-part cap).
This one is a memory bound, not a protocol one, and that is worth saying plainly because it is policy rather than an external constraint: GCS caps no chunk size, so any value is protocol-legal. What makes a bound necessary is that every open session grows a staging buffer toward this value, so without one a config change alone can price a pod out of memory, which is the shape of #1023 (closed).
64MiB is the boundary, chosen so that no deployment which took the old
default becomes invalid. A deployment that had explicitly set something above
64 MiB would now fail to start; that is intentional, and it is the only
backwards-incompatible edge here.
The constant lives in the driver, like the default and the alignment floor:
gcs.MaxChunkSize (256 * MinChunkSize) is exported from
internal/storage/driver/gcs/gcs.go and gcsChunkSizeMax aliases it, the same
shape as s3ChunkSizeMax = s3.MaxChunkSize. Enforcement is unchanged and stays
at config load; gcs.New checks positivity and alignment only, exactly as
s3.New does.
The npm comment blocks
Two comment blocks in internal/format/npm/packument_cache_internal_test.go
still described the driver as preallocating the chunk size, which !2183 (merged) made
false. Correcting either in place trips the comment-caps gate: the blocks are 13
and 15 lines, blank-separated segments count as one block, and the test-file cap
is two lines, so any edit forces the whole block down.
Each is therefore compressed to a pointer at
internal/format/npm/packument_cache.md, which already carries the prose in
full under three headings: "Fence consultations: what each site buys",
"rebuildMaxRendering: the memory cap", and "reserveRenderingSlot: why it
waits rather than sheds". I checked each heading covers the claim it is replacing
before deleting anything, so no reasoning is lost, only relocated to the file
that already held it.
That file's memory arithmetic is also re-derived for the new default:
rebuildMaxRendering * (2 * chunk_size) is about 40 MiB at four slots and
the 5 MiB default, and 512 MiB only at the 64 MiB maximum. rebuildMaxRendering
itself is unchanged and now bounds a more conservative ceiling than it was sized
against.
Configuration companions
Per the configuration guardrail, every surface that states this default moves with it:
proto/artifactregistry/config/v1/config.protocomment, andgen/artifactregistry/config/v1/config.pb.goregenerated throughscripts/proto-generate.sh(comment-only, one line).docs/dev/configuration-reference.md: thechunk_sizerow's default and type, and the per-fill memory prose, which previously said bothchunk_sizefields carry a 64 MiB default. Only S3's does now.config.example.yamlis unchanged and correct: it configures the S3 backend only and carries nogcs:block, so itschunk_size: "64MiB"is S3's, whose default this MR does not touch.- A Research Findings note on
docs/plans/2026-08-14-npm-packument-streaming-generation.md, whose Context computes a ceiling from this default.
Testing
TestLoad_Storage_GCSChunkSizeMaxcovers both sides of the bound. The over-bound case uses66048KiB, which is 64 MiB plus two alignment units (512 KiB) and therefore aligned, so only the new bound can reject it, and the assertion matches the bound's own wording rather than justchunk_size(the alignment error names that field too, so a laxer assertion would pass against the wrong check).- The "omit every optional gcs field" subtest now asserts
5_242_880with a message naming where the default comes from. TestLoad_Storage_GCSHappyPathis deliberately unchanged: its fixture setschunk_size: "64MiB"explicitly, which is still valid at exactly the bound, so it keeps covering the parse path.go build ./...clean../internal/config/...,./internal/storage/...,./internal/format/npm/...pass.golangci-lint(pinned 2.13.2) 0 issues. Comment-caps gate passes../cmd/artifact-registry/needsGOOGLE_APPLICATION_CREDENTIALSset for the threeTestWireStorage_CloudCDN*tests; with it they pass. That is a local environment requirement, not a change here.
e2e scenarios
No scenario in docs/testing/ is added or affected. Chunk size changes how many resumable-upload chunks a large blob is split into, not any request or response the client sees, and the GCS conformance suite already runs at 5 MiB.
Why Related to and not Closes
#1023 (closed) asks for a heap pprof profile of the upload path, which is not possible
until /debug/pprof/* is exposed
(#58), and
for staging confirmation that 512Mi is comfortable, which wants an observation
after this deploys. Both remain after this merges, as does the stale comment in
.runway/fairway.yaml pointing at the closed
#447.
Related to #1023 (closed)