Backport of 'Reuse HTTP connections and add timeouts in Gitlab::HttpIO' (19.3)
What does this MR do and why?
Status
Not blocked on anything upstream: both flag-removal MRs, !251684 (merged) and !251685 (merged), merged to master on 2026-08-25 (labelled workflow::production) and are deployed to GitLab.com.
This is an ordinary patch-release backport into the current stable release (19.3) and needs no exception; it should merge before !251899 (merged), which is stacked on it.
What this backports
This backports the flag removal !251684 (merged) (squashed as commit 21d3c03e). 19.3.0 already ships the code from !246918 (merged), so this MR only deletes the flag and its definition. The backport branch is a single commit, squashed to the state after the feature flag was removed, so the stable branch never carries the flag; the resulting lib/gitlab/http_io.rb is byte-identical to current master.
What it changes
Gitlab::HttpIO opened a new Net::HTTP connection for every 128KB range GET, so reading a 64MB archived trace paid DNS, TCP and TLS setup 512 times. It now reuses one keep-alive session per HttpIO instance, sets explicit open/read/write timeouts from Gitlab::HTTP::DEFAULT_TIMEOUT_OPTIONS instead of Net::HTTP's 60s defaults, sets ignore_eof: false so a connection dying mid-body raises instead of returning a silently truncated chunk, and implements a real #close. Without the timeouts, one unresponsive object storage endpoint could stall a high-urgency Sidekiq worker for up to 60s per chunk.
Rollout evidence
The http_io_persistent_connections feature flag has been at 100% on GitLab.com since 2026-08-10. It was switched off for part of that day during gitlab-com/gl-infra/production#22680 (closed), a wave of TCP connect failures to storage.googleapis.com, and re-enabled the same day with no measurable difference either way.
Backport notes
Clean cherry-pick: lib/gitlab/http_io.rb and its spec on 19-3-stable-ee are identical to the master commit the removal was written against.
Why this version needs it
19.3 is the current stable release. Both flags are gitlab_com_derisk type and default to disabled, so 19.3.0 as released ships both fixes switched off for every self-managed and Dedicated installation. This MR is what turns connection reuse on for those installations.
The full set
| Version | Connection reuse | Previous-chunk cache |
|---|---|---|
| 19.2 (N-1, GitLab Dedicated) | !251864 (merged) | !251865 (merged) (stacked) |
| 19.3 (current stable) | !251898 (merged) | !251899 (merged) (stacked) |
Shared context: https://gitlab.com/gitlab-org/gitlab/-/work_items/605350 (Ci::BuildFinishedWorker latency) and #597765 (severity::2, open since 2026-04-22, actively paging a GitLab Dedicated tenant).
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
- This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
- The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
- The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
- Required labels have been applied to this merge request
- severity label and bug subtype labels (if applicable)
- If this MR fixes a bug that affects customers, the customer label has been applied.
- This MR has been approved by a maintainer (only one approval is required).
- Ensure the
e2e:test-on-omnibus-eejob has succeeded, or if it has failed, investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.
Note to the merge request author and maintainer
If you have questions about the patch release process, please:
- Refer to the patch release runbook for engineers and maintainers for guidance.
- Ask questions on the
#releasesSlack channel (internal only). - Once the backport has been merged, the commit changes will be automatically deployed to a release environment that can be used for manual validation. See after merging runbook for details.