[FF] http_io_persistent_connections -- reuse HTTP connections and set timeouts in Gitlab::HttpIO

Summary

Roll out persistent connections and explicit timeouts in Gitlab::HttpIO currently behind the http_io_persistent_connections feature flag.

• DRI: @hfyngvason
• Team Slack channel: #g_pipeline-execution

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

The flag changes how Gitlab::HttpIO reads remote files from object storage (archived CI traces, remote artifacts opened via GitlabUploader#open): one persistent keep-alive connection per opened file with explicit timeouts (open 10s / read 20s / write 30s), instead of a fresh untimed connection per 128KB chunk (Net::HTTP defaults: 60s each).

Two behavior changes to watch:

1. Reads through an unresponsive object storage endpoint now raise Net::OpenTimeout/Net::ReadTimeout after seconds instead of hanging for up to 60s per chunk. On the coverage-extraction path these are swallowed (existing rescue StandardError); on the artifact-parsing path they surface to Sidekiq retries — intended, but a misbehaving storage backend becomes visible as errors rather than latency.
2. Chunked reads reuse one keep-alive session. Net::HTTP transparently reconnects and retries the idempotent GET once if the server drops the connection between chunks.

Rollback is disabling the flag. Watch Ci::BuildFinishedWorker error rates and duration percentiles, and Gitlab::HttpIO::FailedToGetChunkError / Net::OpenTimeout occurrences in Sentry/Kibana.

Rollout

Run all production /chatops in #production and cross-post the results to #g_pipeline-execution. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set http_io_persistent_connections 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set http_io_persistent_connections true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set http_io_persistent_connections <percentage> --actors

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

• Docs + version history updated
• Breaking changes announced, if any
• Change management issue opened, if required
• External API consumers handled with a fail-open mechanism, if applicable

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246918 19.3
/chatops gitlab run feature delete http_io_persistent_connections --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set http_io_persistent_connections false                                         # production
/chatops gitlab run feature set http_io_persistent_connections false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete http_io_persistent_connections --dev --pre --staging --staging-ref --production  # remove entirely