fix(config): lower the GCS chunk-size default and bound it at 64MiB

Stack

Merge in order. This MR targets !2183 (merged)'s branch, so its diff shows only its own changes.

# MR What
1 !2183 (merged) fix(storage): grow the upload buffer instead of sizing it to chunk_size, plus GOMEMLIMIT
2 this MR fix(config): lower the GCS chunk_size default to the driver's own constant and bound it at 64MiB

What

Two constants both claimed to be the GCS chunk-size default and disagreed by 12.8x:

  • internal/storage/driver/gcs/gcs.go:81, gcs.DefaultChunkSize = 5 MiB, documenting itself as "matching the Container Registry default".
  • internal/config/storage.go:37, defaultGCSChunkSize = 64 MiB, and this one won.

It won because sizeOrDefault always substitutes a value for an unset field, which also makes the chunkSize <= 0 fallback to the driver constant in buildGCSDriver (cmd/artifact-registry/wire_storage.go:313-315) unreachable in any configuration the loader produced.

The consequence worth fixing independently of !2183 (merged): the GCS conformance suite runs 5 MiB while production runs 64 MiB, so the size production actually uses was never exercised by the suite that certifies the driver.

The config default is now the driver's constant itself, so the two cannot diverge again.

The upper bound

storage.gcs.chunk_size gains a maximum, which the S3 equivalent has had all along (s3ChunkSizeMax, 5 GiB, AWS's per-part cap).

This one is a memory bound, not a protocol one, and that is worth saying plainly because it is policy rather than an external constraint: GCS caps no chunk size, so any value is protocol-legal. What makes a bound necessary is that every open session grows a staging buffer toward this value, so without one a config change alone can price a pod out of memory, which is the shape of #1023 (closed).

64MiB is the boundary, chosen so that no deployment which took the old default becomes invalid. A deployment that had explicitly set something above 64 MiB would now fail to start; that is intentional, and it is the only backwards-incompatible edge here.

The constant lives in the driver, like the default and the alignment floor: gcs.MaxChunkSize (256 * MinChunkSize) is exported from internal/storage/driver/gcs/gcs.go and gcsChunkSizeMax aliases it, the same shape as s3ChunkSizeMax = s3.MaxChunkSize. Enforcement is unchanged and stays at config load; gcs.New checks positivity and alignment only, exactly as s3.New does.

The npm comment blocks

Two comment blocks in internal/format/npm/packument_cache_internal_test.go still described the driver as preallocating the chunk size, which !2183 (merged) made false. Correcting either in place trips the comment-caps gate: the blocks are 13 and 15 lines, blank-separated segments count as one block, and the test-file cap is two lines, so any edit forces the whole block down.

Each is therefore compressed to a pointer at internal/format/npm/packument_cache.md, which already carries the prose in full under three headings: "Fence consultations: what each site buys", "rebuildMaxRendering: the memory cap", and "reserveRenderingSlot: why it waits rather than sheds". I checked each heading covers the claim it is replacing before deleting anything, so no reasoning is lost, only relocated to the file that already held it.

That file's memory arithmetic is also re-derived for the new default: rebuildMaxRendering * (2 * chunk_size) is about 40 MiB at four slots and the 5 MiB default, and 512 MiB only at the 64 MiB maximum. rebuildMaxRendering itself is unchanged and now bounds a more conservative ceiling than it was sized against.

Configuration companions

Per the configuration guardrail, every surface that states this default moves with it:

  • proto/artifactregistry/config/v1/config.proto comment, and gen/artifactregistry/config/v1/config.pb.go regenerated through scripts/proto-generate.sh (comment-only, one line).
  • docs/dev/configuration-reference.md: the chunk_size row's default and type, and the per-fill memory prose, which previously said both chunk_size fields carry a 64 MiB default. Only S3's does now.
  • config.example.yaml is unchanged and correct: it configures the S3 backend only and carries no gcs: block, so its chunk_size: "64MiB" is S3's, whose default this MR does not touch.
  • A Research Findings note on docs/plans/2026-08-14-npm-packument-streaming-generation.md, whose Context computes a ceiling from this default.

Testing

  • TestLoad_Storage_GCSChunkSizeMax covers both sides of the bound. The over-bound case uses 66048KiB, which is 64 MiB plus two alignment units (512 KiB) and therefore aligned, so only the new bound can reject it, and the assertion matches the bound's own wording rather than just chunk_size (the alignment error names that field too, so a laxer assertion would pass against the wrong check).
  • The "omit every optional gcs field" subtest now asserts 5_242_880 with a message naming where the default comes from.
  • TestLoad_Storage_GCSHappyPath is deliberately unchanged: its fixture sets chunk_size: "64MiB" explicitly, which is still valid at exactly the bound, so it keeps covering the parse path.
  • go build ./... clean. ./internal/config/..., ./internal/storage/..., ./internal/format/npm/... pass. golangci-lint (pinned 2.13.2) 0 issues. Comment-caps gate passes.
  • ./cmd/artifact-registry/ needs GOOGLE_APPLICATION_CREDENTIALS set for the three TestWireStorage_CloudCDN* tests; with it they pass. That is a local environment requirement, not a change here.

e2e scenarios

No scenario in docs/testing/ is added or affected. Chunk size changes how many resumable-upload chunks a large blob is split into, not any request or response the client sees, and the GCS conformance suite already runs at 5 MiB.

Why Related to and not Closes

#1023 (closed) asks for a heap pprof profile of the upload path, which is not possible until /debug/pprof/* is exposed (#58), and for staging confirmation that 512Mi is comfortable, which wants an observation after this deploys. Both remain after this merges, as does the stale comment in .runway/fairway.yaml pointing at the closed #447.

Related to #1023 (closed)

Edited by Dzmitry (Dima) Meshcharakou

Merge request reports

Loading
Loading