ci: disable SeaweedFS auto-vacuum to fix write 500 flakes
What
Fixes intermittent failures in the s3:seaweedfs integration job where writes fail with a transient HTTP 500 InternalError, most often TestS3DriverSuite/TestValidPaths.
Original failing job: https://gitlab.com/gitlab-org/container-registry/-/jobs/16497005395
s3_v2: operation error S3: PutObject, exceeded maximum number of attempts,
StatusCode: 500, api error InternalError: We encountered an internal error, please try again.Root cause
SeaweedFS auto-vacuum. During a volume compaction the volume goes read-only and writes to it fail with HTTP 500 for the whole duration. The job set -master.garbageThreshold=0.1, so a compaction was triggered aggressively.
The trigger is a test that mass-writes then deletes (the vacuum / remove-manifests tests, which run just before TestValidPaths alphabetically). That leaves one or more volumes almost entirely garbage. The master's periodic vacuum loop then fires ~20s later and compacts for ~35s, and TestValidPaths (tiny 64-byte writes) hits a wall of 500s the whole time. No retry budget can win a race that long.
Fix
Scoped to the s3:seaweedfs job only; no driver or production defaults change.
-master.garbageThreshold=1.1to disable auto-vacuum for the run. The master vacuum check isresp.GarbageRatio >= garbageThreshold, andGarbageRatioreaches1.0for a fully-deleted volume, so0.99and even1.0still fire. A value above1.0is unreachable, so the check is always false. Safe here: the backend is short-lived, disk is ample onsaas-linux-large-amd64, and the large-write tests that could exhaust disk are already skipped.- Kept
S3_MAX_RETRIES=10as a safety net for brief 500s such as volume rollover. It is exported after theS3|AWSunset loop or it would be wiped.
What I tried first (did not work)
- Larger volumes (
volumeSizeLimitMB=1024) + more retries. Job https://gitlab.com/gitlab-org/container-registry/-/jobs/16497953843 still failed, and the stall grew from ~20s to ~36s because a bigger volume takes longer to compact. This pinpointed vacuum as the cause. Volume size was reverted to 256. garbageThreshold=0.99. Job https://gitlab.com/gitlab-org/container-registry/-/jobs/16498319302 still failed the same way (1.25 and 1.26 passed, 1.27 did not), because the>=comparison against a1.0ratio still triggers vacuum. Raised to1.1.
Notes
The tests already delete everything they write and assert / is empty in TearDownTest, so this is not a live-data accumulation problem. SeaweedFS is log-append, so deletes only reclaim space via vacuum, which is exactly the operation that was stalling writes.