fix(storage): clamp the S3 upload buffer and reprice its slot
Stacked MRs
| Part | MR | Scope | Reviewable LOC | Target |
|---|---|---|---|---|
| 1/2 | this MR | clamp the S3 upload buffer, one slot ratio for both drivers | 417 | main |
| 2/2 | !3048 (merged) | lower the storage.s3.chunk_size default to 5MiB |
62 | dmeshcharakou/1171-s3-buffer-clamp |
Merge this MR first. Part 2's 5MiB default derives a cap of 28 only with this MR's 1.2 ratio. On today's ratio of 2 it derives 18. LOC is added plus removed, measured at f0521b22f and d324415e6.
Summary
The S3 writer staged each part in a bytes.Buffer. Its growth doubles past chunk_size, and Reset keeps the capacity, so a session held more than one chunk for its whole life. At the 64 MiB default with 100000-byte reads that is 104 MiB per session. GCS already stops at chunk_size.
This MR:
- moves the GCS growth rule into a shared
chunkbuf.GrowCap(internal/storage/driver/internal/chunkbuf). GCS behavior does not change. - replaces the S3
bytes.Bufferwith a smallstagingBufferthat grows throughGrowCap. It keeps theLen,Bytes,Reset,WriteandReadFromcalls the writer makes, so the ported Container RegistryWriteand re-upload code do not change.ReadFromcovers the re-upload path's tail read, which also grew past the chunk. It returns an error once the buffer holdschunk_sizebytes, which a compliant tail never reaches, so a backend that ignoresRangefails the re-upload instead of spinning theWriteloop. - prices an S3 upload slot like a GCS one,
1.2 x chunk_size + 1MiB, and merges the two ratio constants into one.
Capacity one session keeps with the old S3 buffer, from replaying the writer's loop:
chunk_size |
100000-byte reads | 1448-byte reads | 64 KiB reads |
|---|---|---|---|
| 64 MiB | 1.625x (104 MiB) | 1.5x | 1.0x |
| 5 MiB | 1.3x | 1.2x | 1.6x |
With the clamp, every row is at most 1.0x.
The derived cap on S3, at the 400 MiB GOMEMLIMIT fallback (200 MiB budget):
S3 chunk_size |
Before | After |
|---|---|---|
| 64 MiB (default) | 2 from the floor, 258 MiB reserved | 2 derived, 156 MiB reserved, no floor warning |
| 5 MiB | 18 | 28 |
| 128 MiB | 2 from the floor | 2 from the floor |
So a default S3 install no longer reserves more than its budget, but it still gets 2 slots. Part 2 fixes that.
Governing ADRs
scripts/adr-freshness.sh: the mirror is current.
- ADR 003 (
docs/adr/003_system_requirements.md): its storage-throughput paragraph says the S3 driver at its shipped defaults "admits more staging memory than the budget it was derived from". This MR removes that over-admission, so it conforms, and that sentence goes stale (see below). - ADR 004 (
docs/adr/004_data_and_application_limits.md): no limit changes here.
Testing
TestGrowCap: the doubling, target, clamp and past-chunk cases. Removing any of the three terms reddens it, and the untouched GCS growth tests redden on two of them, so the move is checked from both sides.TestStagingBuffer*andTestWriterStagesWithinChunkSize: capacity stays withinchunk_sizeat 100000-byte, 1448-byte and 64 KiB writes and reads, content is kept,Resetkeeps the capacity, and a read error is wrapped. Without the clamp every row reddens except the power-of-two chunk fed power-of-two writes, the shape that hid the defect.- Config tests: the S3 default derives 2 with no floor warning, and a
128MiBS3 chunk still fires it with a162109849-byte reservation. Setting the ratio back to 2 reddens both. - S3 integration suite against a local Garage, the emulator CI uses (
-tags=integration -short): 53 conformance subtests pass, and 2 skip under the same-shortand large-write settings CI uses.TestContinueStreamAppendSmallgoes through the newReadFrom, and breakingReadFromfails it. golangci-lint2.13 (the pinned one, through mise) reports 0 issues on the changed packages, and the comment-caps gate is clean.
No e2e scenario is added or affected: the change sits inside the driver and the boot derivation, and no scenario asserts the S3 cap. No docs/user/ page covers storage.* settings, so no user documentation is affected.
Plan and spec amendments
docs/plans/2026-09-03-upload-session-concurrency-cap.md, Step 12 (8179e732, 1acc266e): both bullets that gave 18 at the 5 MiB minimumchunk_sizenow give 28, which clears the 25 sessions. The floor warning'snotebullet now says that under the 400 MiBGOMEMLIMITfallback the warning no longer fires at the S3 default, and names the configurations that still raise it, aGOMEMLIMITof 311 MiB or less among them. Step 11,### The derived cap at the defaultsand the## Dependenciesbullet on the shipped S3 default still describe the unclamped driver, as the plan's recorded basis.
Not changed here
- ADR 003's S3 shortfall paragraph needs a handbook ADR update once this stack lands. The amendment is recorded on #1171 (closed).
Related to #1171 (closed)