[FF] http_io_retry_transient_errors -- retry transient object storage failures in Gitlab::HttpIO

Summary

Roll out retries for transient object storage failures in Gitlab::HttpIO currently behind the http_io_retry_transient_errors feature flag.

  • DRI: @hfyngvason
  • Team Slack channel: #g_pipeline-execution

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

Gitlab::HttpIO reads remote files from object storage (archived CI job traces, remote artifacts opened via GitlabUploader#open) as a series of 128 KB range requests. With the flag on, a failed chunk request is retried instead of failing the whole read. The added retry loop handles transport errors such as TLS errors, connection resets, DNS failures, EOF and malformed responses, plus response codes 429, 500, 502, 503, 504. It does not retry open/read/write timeout errors or other 4xx responses. Existing Net::HTTP retry behavior, including its read-timeout retry, is unchanged. The budget is 3 retries shared across every chunk an HttpIO instance fetches, with a 100ms base delay doubled per retry and jittered. With the flag off, the pre-flag code path runs unchanged, in a separate method.

  1. A read against an unavailable endpoint can take longer to fail: up to 3 additional HttpIO attempts per instance, on top of existing Net::HTTP retries. Using all 3 adds 0.35–1.05 seconds of backoff, plus network time. Timeout errors stop the added retry loop, but retriable errors can occur after slow I/O and a subsequent attempt can time out, so the budget is not a wall-clock limit. This affects user-facing job trace requests and Sidekiq workers that parse artifacts.
  2. Retrying a 5xx or 429 adds requests to an endpoint that may already be struggling. The shared budget of 3 per HttpIO instance and the jittered backoff are what bound this; if object storage is genuinely saturated rather than failing occasionally, retries make the load worse rather than better.
  3. Errors that used to surface immediately now surface later. Gitlab::HttpIO::FailedToGetChunkError and OpenSSL::SSL::SSLError volume should fall; if it does not, the retries are not helping and the flag should go back off.

Rollback is disabling the flag, which restores the pre-flag path exactly. Watch the new gitlab_http_io_chunk_retries_total counter, broken down by its reason label, to see whether retries fire at all and whether transport errors or 5xx responses dominate. Also watch the 500 rate on GET /api/:version/projects/:id/jobs/:job_id/trace, external_http_duration_s percentiles, and Sentry volume for Gitlab::HttpIO::FailedToGetChunkError, OpenSSL::SSL::SSLError and Net::OpenTimeout.

The flag is gitlab_com_derisk, so it is off for self-managed and GitLab Dedicated. The Dedicated escalation that motivated this work therefore does not benefit until the flag is removed.

Rollout

Run all production /chatops in #production and cross-post the results to #g_pipeline-execution. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set http_io_retry_transient_errors 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set http_io_retry_transient_errors true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set http_io_retry_transient_errors <percentage> --actors

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252069 19.4
/chatops gitlab run feature delete http_io_retry_transient_errors --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set http_io_retry_transient_errors false                                         # production
/chatops gitlab run feature set http_io_retry_transient_errors false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete http_io_retry_transient_errors --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Hordur Freyr Yngvason