ci: disable SeaweedFS auto-vacuum to fix write 500 flakes

What

Fixes intermittent failures in the s3:seaweedfs integration job where writes fail with a transient HTTP 500 InternalError, most often TestS3DriverSuite/TestValidPaths.

Original failing job: https://gitlab.com/gitlab-org/container-registry/-/jobs/16497005395

s3_v2: operation error S3: PutObject, exceeded maximum number of attempts,
StatusCode: 500, api error InternalError: We encountered an internal error, please try again.

Root cause

SeaweedFS auto-vacuum. During a volume compaction the volume goes read-only and writes to it fail with HTTP 500 for the whole duration. The job set -master.garbageThreshold=0.1, so a compaction was triggered aggressively.

The trigger is a test that mass-writes then deletes (the vacuum / remove-manifests tests, which run just before TestValidPaths alphabetically). That leaves one or more volumes almost entirely garbage. The master's periodic vacuum loop then fires ~20s later and compacts for ~35s, and TestValidPaths (tiny 64-byte writes) hits a wall of 500s the whole time. No retry budget can win a race that long.

Fix

Scoped to the s3:seaweedfs job only; no driver or production defaults change.

  • -master.garbageThreshold=1.1 to disable auto-vacuum for the run. The master vacuum check is resp.GarbageRatio >= garbageThreshold, and GarbageRatio reaches 1.0 for a fully-deleted volume, so 0.99 and even 1.0 still fire. A value above 1.0 is unreachable, so the check is always false. Safe here: the backend is short-lived, disk is ample on saas-linux-large-amd64, and the large-write tests that could exhaust disk are already skipped.
  • Kept S3_MAX_RETRIES=10 as a safety net for brief 500s such as volume rollover. It is exported after the S3|AWS unset loop or it would be wiped.

What I tried first (did not work)

  1. Larger volumes (volumeSizeLimitMB=1024) + more retries. Job https://gitlab.com/gitlab-org/container-registry/-/jobs/16497953843 still failed, and the stall grew from ~20s to ~36s because a bigger volume takes longer to compact. This pinpointed vacuum as the cause. Volume size was reverted to 256.
  2. garbageThreshold=0.99. Job https://gitlab.com/gitlab-org/container-registry/-/jobs/16498319302 still failed the same way (1.25 and 1.26 passed, 1.27 did not), because the >= comparison against a 1.0 ratio still triggers vacuum. Raised to 1.1.

Notes

The tests already delete everything they write and assert / is empty in TearDownTest, so this is not a live-data accumulation problem. SeaweedFS is log-append, so deletes only reclaim space via vacuum, which is exactly the operation that was stalling writes.

Edited by Stan Hu

Merge request reports

Loading
Loading