Retry transient HttpIO chunk failures behind a feature flag

What does this MR do and why?

Gitlab::HttpIO reads remote files (archived CI job traces, job artifacts) from object storage as a series of 128 KB HTTP range requests over one keep-alive session.

Object storage fails a small fraction of requests: refused or dropped TLS connections, transient 5xx. Because one read issues many chunk requests, a small per-request failure rate compounds into a much higher per-read failure rate, and today any single failed chunk fails the whole read and surfaces to the user as a 500.

On GitLab Dedicated this shows up as SSL handshake failures (SSL_connect returned=6 errno=107 state=SSLv3/TLS write client hello) on reads of GET /api/:version/projects/:id/jobs/:job_id/trace.

Net::HTTP already retries once, but does not cover these cases:

• The TLS handshake in Net::HTTP.start happens outside Net::HTTP#transport_request.
• Response status codes are not errors to it at all.

This MR adds retry handling behind the feature flag http_io_retry_transient_errors, disabled by default. With the flag off, the original code path runs unchanged: the two paths are separate methods, so disabling the flag is a real kill switch rather than a no-op branch through new code.

What it retries:

• Transport errors: TLS errors, connection resets, DNS failures, EOF mid-body, malformed responses.
• Response codes 429, 500, 502, 503, 504.

What the added retry loop does not retry:

• Other 4xx. An expired presigned object storage URL returns 403, and retrying only adds latency before the same failure.
• Open/read/write timeout errors. Repeating these waits would add latency to user-facing requests. Existing Net::HTTP retry behavior, including its read-timeout retry, is unchanged.

Budget: up to 3 additional HttpIO attempts shared across every chunk an instance fetches, on top of existing Net::HTTP retries. The budget persists across calls to read and readline; it does not reset per chunk or per read call.

Backoff: 100ms base, doubled per retry and jittered, for 0.35–1.05 seconds of total sleep when all 3 retries are used. Network time is additional: retriable errors can occur after slow I/O, and a subsequent attempt can time out. This bounds retry attempts and sleep, not total read duration.

This MR also adds a new Prometheus counter, gitlab_http_io_chunk_retries_total, labelled by reason (the error class, or http_<code>), because no existing metric distinguishes a clean request from one that needed a retry. This is what makes the rollout readable.

How to set up and validate locally

Run:

bundle exec rspec spec/lib/gitlab/http_io_spec.rb

Validated on commit 842a69311c03d3d082ac2d29e4ddc63e2d7e9097:

• RSpec: 63 examples, 0 failures, including enabled and disabled flag paths.
• bundle exec rubocop lib/gitlab/http_io.rb spec/lib/gitlab/http_io_spec.rb: 2 files inspected, no offenses.
• Latest merged-results pipeline: passed.

The specs cover the retry paths, including one run against a real keep-alive HTTP server on localhost, because WebMock intercepts above the socket layer and cannot exercise real transport failures.

References

• Reported problem: #604514
• Parent epic for HttpIO work: https://gitlab.com/groups/gitlab-org/-/work_items/23204
• Feature flag rollout issue: #623429

Screenshots or screen recordings

Not applicable: no visible UI changes.

MR acceptance checklist

• Prior Duo findings are addressed: preserve the original EOFError as the cause, keep existing single-attempt failure specs on the disabled path, and isolate retry metrics and backoff in specs.
• The feature flag is disabled by default, with rollout and rollback tracked in the linked rollout issue.
• No changelog entry is needed while the behavior is behind a default-off flag.
• No user documentation changes are needed for this internal retry behavior.