fix(storage): clamp the S3 upload buffer and reprice its slot

Stacked MRs

Part MR Scope Reviewable LOC Target
1/2 this MR clamp the S3 upload buffer, one slot ratio for both drivers 417 main
2/2 !3048 (merged) lower the storage.s3.chunk_size default to 5MiB 62 dmeshcharakou/1171-s3-buffer-clamp

Merge this MR first. Part 2's 5MiB default derives a cap of 28 only with this MR's 1.2 ratio. On today's ratio of 2 it derives 18. LOC is added plus removed, measured at f0521b22f and d324415e6.

Summary

The S3 writer staged each part in a bytes.Buffer. Its growth doubles past chunk_size, and Reset keeps the capacity, so a session held more than one chunk for its whole life. At the 64 MiB default with 100000-byte reads that is 104 MiB per session. GCS already stops at chunk_size.

This MR:

  • moves the GCS growth rule into a shared chunkbuf.GrowCap (internal/storage/driver/internal/chunkbuf). GCS behavior does not change.
  • replaces the S3 bytes.Buffer with a small stagingBuffer that grows through GrowCap. It keeps the Len, Bytes, Reset, Write and ReadFrom calls the writer makes, so the ported Container Registry Write and re-upload code do not change. ReadFrom covers the re-upload path's tail read, which also grew past the chunk. It returns an error once the buffer holds chunk_size bytes, which a compliant tail never reaches, so a backend that ignores Range fails the re-upload instead of spinning the Write loop.
  • prices an S3 upload slot like a GCS one, 1.2 x chunk_size + 1MiB, and merges the two ratio constants into one.

Capacity one session keeps with the old S3 buffer, from replaying the writer's loop:

chunk_size 100000-byte reads 1448-byte reads 64 KiB reads
64 MiB 1.625x (104 MiB) 1.5x 1.0x
5 MiB 1.3x 1.2x 1.6x

With the clamp, every row is at most 1.0x.

The derived cap on S3, at the 400 MiB GOMEMLIMIT fallback (200 MiB budget):

S3 chunk_size Before After
64 MiB (default) 2 from the floor, 258 MiB reserved 2 derived, 156 MiB reserved, no floor warning
5 MiB 18 28
128 MiB 2 from the floor 2 from the floor

So a default S3 install no longer reserves more than its budget, but it still gets 2 slots. Part 2 fixes that.

Governing ADRs

scripts/adr-freshness.sh: the mirror is current.

  • ADR 003 (docs/adr/003_system_requirements.md): its storage-throughput paragraph says the S3 driver at its shipped defaults "admits more staging memory than the budget it was derived from". This MR removes that over-admission, so it conforms, and that sentence goes stale (see below).
  • ADR 004 (docs/adr/004_data_and_application_limits.md): no limit changes here.

Testing

  • TestGrowCap: the doubling, target, clamp and past-chunk cases. Removing any of the three terms reddens it, and the untouched GCS growth tests redden on two of them, so the move is checked from both sides.
  • TestStagingBuffer* and TestWriterStagesWithinChunkSize: capacity stays within chunk_size at 100000-byte, 1448-byte and 64 KiB writes and reads, content is kept, Reset keeps the capacity, and a read error is wrapped. Without the clamp every row reddens except the power-of-two chunk fed power-of-two writes, the shape that hid the defect.
  • Config tests: the S3 default derives 2 with no floor warning, and a 128MiB S3 chunk still fires it with a 162109849-byte reservation. Setting the ratio back to 2 reddens both.
  • S3 integration suite against a local Garage, the emulator CI uses (-tags=integration -short): 53 conformance subtests pass, and 2 skip under the same -short and large-write settings CI uses. TestContinueStreamAppendSmall goes through the new ReadFrom, and breaking ReadFrom fails it.
  • golangci-lint 2.13 (the pinned one, through mise) reports 0 issues on the changed packages, and the comment-caps gate is clean.

No e2e scenario is added or affected: the change sits inside the driver and the boot derivation, and no scenario asserts the S3 cap. No docs/user/ page covers storage.* settings, so no user documentation is affected.

Plan and spec amendments

  • docs/plans/2026-09-03-upload-session-concurrency-cap.md, Step 12 (8179e732, 1acc266e): both bullets that gave 18 at the 5 MiB minimum chunk_size now give 28, which clears the 25 sessions. The floor warning's note bullet now says that under the 400 MiB GOMEMLIMIT fallback the warning no longer fires at the S3 default, and names the configurations that still raise it, a GOMEMLIMIT of 311 MiB or less among them. Step 11, ### The derived cap at the defaults and the ## Dependencies bullet on the shipped S3 default still describe the unclamped driver, as the plan's recorded basis.

Not changed here

  • ADR 003's S3 shortfall paragraph needs a handbook ADR update once this stack lands. The amendment is recorded on #1171 (closed).

Related to #1171 (closed)

Edited by Dzmitry (Dima) Meshcharakou

Merge request reports

Loading
Loading