Backport of 'Retry transient HttpIO chunk failures' (19.3)

What does this MR do and why?

Status

This is an ordinary patch-release backport into a supported release (19.3), and needs no exception.

On the deployment checklist: the retry code merged to master in !252069 (merged) on 2026-09-11 and is deployed to GitLab.com, and its feature flag (http_io_retry_transient_errors) has been at 100% of actors there since 2026-09-15 (rollout issue #623429 (closed)), so the behaviour has real production exposure. The separate flag-removal commit (!255927 (merged)) is still labelled workflow::staging as of 2026-09-18. On GitLab.com, removing a flag that is already at 100% is behaviourally a no-op, but the removal commit itself should reach production before this merges, so the "deployed to GitLab.com" box below is left unchecked until then.

What this backports

• !252069 (merged) — "Retry transient HttpIO chunk failures behind a feature flag" (merged to master 2026-09-11, squashed as 842a69311c03).
• !255927 (merged) — "Remove http_io_retry_transient_errors feature flag" (merged to master 2026-09-17, squashed as 4561fc6d9e3e).

19.3 never carried the flag or the retry code, so both commits are combined into a single flag-free commit on this branch instead of landing as two separate cherry-picks. The resulting lib/gitlab/http_io.rb and spec/lib/gitlab/http_io_spec.rb are byte-identical to current master, and this branch never ends up with a flag definition file.

What it changes

Gitlab::HttpIO reads remote files from object storage (archived CI job traces, and remote artifacts opened via GitlabUploader#open) as a series of 128 KB HTTP range requests over one keep-alive session. Object storage fails a small fraction of requests, and because a single read issues many chunk requests, that per-request failure rate compounds into a much higher per-read failure rate. Previously, any one failed chunk failed the entire read and surfaced to the user as a 500.

A failed chunk request is now retried instead of failing the whole read:

• Retried: transport errors (TLS errors, connection resets, DNS failures, EOF mid-body, malformed responses) and response codes 429, 500, 502, 503, 504.
• Not retried: other 4xx (an expired presigned object storage URL returns 403, and retrying only adds latency before the same failure), and open/read/write timeout errors (repeating those waits would add latency inside a user-facing request). Existing Net::HTTP retry behaviour, including its read-timeout retry, is unchanged.

The retry budget is 3, shared across every chunk one HttpIO instance fetches, on top of existing Net::HTTP retries. The budget persists across calls to read and readline; it does not reset per chunk or per read call. Backoff is 100ms base, doubled per retry and jittered, for 0.35–1.05 seconds of total sleep when all 3 retries are used.

A new Prometheus counter, gitlab_http_io_chunk_retries_total, is added, labelled by reason (the error class, or http_<code>).

Why this version needs it

19.3 does not have this code at all. Without this backport, GitLab Dedicated and self-managed instances on 19.3 keep failing entire reads (job trace downloads, remote artifact reads) on a single transient chunk failure — the exact behaviour behind incident inc-11214, which showed up on Dedicated as SSL handshake failures on job trace reads.

Backport notes

Both cherry-picks (842a69311c03 then 4561fc6d9e3e) applied with no conflicts, and are squashed into the single commit on this branch described above.

Related backports

Sibling HttpIO stable-branch backports, for reference:

Version	MR(s)	Notes
19.2	!251864 (merged), !251865 (merged)	Connection reuse, previous-chunk cache
19.3	!251898 (merged), !251899 (merged)	Connection reuse, previous-chunk cache
19.3	this MR	Retry transient chunk failures (this fix)
19.4	!256444 (merged)	Retry transient chunk failures (this fix)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
• The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
• The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
• Required labels have been applied to this merge request
  • severity label and bug subtype labels (if applicable)
  • If this MR fixes a bug that affects customers, the customer label has been applied.
• This MR has been approved by a maintainer (only one approval is required).
• Ensure the E2E Omnibus GitLab EE pipeline (triggered by thee2e:test-on-omnibus-ee job) has succeeded. If it has failed (or shows warnings), investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.

Note to the merge request author and maintainer

If you have questions about the patch release process, please:

• Refer to the patch release runbook for engineers and maintainers for guidance.
• Ask questions on the #releases Slack channel (internal only).
• Once the backport has been merged, the commit changes will be automatically deployed to a release environment that can be used for manual validation. See after merging runbook for details.