Gitlab::HttpIO#get_chunk should attempt retry on 5xx

Summary

We are facing issues on the network path from webservice to S3. When a lot of load is added on */projects/25302/jobs/*/trace* endpoint we start seeing 5xx errors with OpenSSL::SSL::SSLError or Gitlab::HttpIO::FailedToGetChunkError

Ideally the Gitlab::HttpIO#get_chunk should simply retry on such errors rather than surfacing them to users.

Impact

• inc-11214: https://app.incident.io/gitlab/incidents/01KVBT5C4462KDJ4HKQ332Y1PY
• Spanning across multiple tenants and summarized here

Recommendation

• Retry mechanism with exponential backoff should be implement based on AWS recommendation - https://repost.aws/knowledge-center/http-5xx-errors-s3
• Use connection pooling instead of initiating a new connection for every single chunk read

Verification