Cache CI build trace chunk object storage connection

Related to https://gitlab.com/gitlab-com/request-for-help/-/work_items/5311

What does this MR do and why?

Ci::BuildTraceChunks::Fog#connection (in app/models/ci/build_trace_chunks/fog.rb) cached its Fog::Storage connection in Gitlab::SafeRequestStore, which is request-scoped. That store is cleared at the end of every request, so every request rebuilt the Fog connection and its googleauth (Signet) credential from scratch.

A fresh credential has no cached OAuth2 access token, so it fetches a new token from the Google token endpoint (oauth2.googleapis.com) on almost every call. size(model) reads the chunk (data -> files.get -> get_object, a real API call) on nearly every PUT /api/:version/jobs/:id job status update, through end_offset in the trace read path. This made it the dominant source of object storage OAuth2 token exchanges, about 60% of them by caller_id, with job artifacts and job trace making up most of the rest.

When a token endpoint TCP connect stalls, for example during a network or egress issue, the request hangs until the 60s Rack::Timeout fires and returns a 500. We've seen these stalls in production.

This MR reuses one Fog connection per process, keyed by the connection credentials, the same way CarrierWave::Storage::Fog already caches connections for the upload path with its connection_cache. The googleauth credential then keeps its access token in memory and refreshes about once an hour instead of once per request. This is implemented with a class-level Concurrent::Map (connections) and a cached_connection class method.

In a container running the production gem versions, 20 job-update-style calls went from 20 token fetches (request-scoped) to 1 (process-level cache). With the flag off, behavior is unchanged.

Thread safety

The cached connection is shared across Puma threads, so it must tolerate concurrent use.

• GCS (googleauth): safe. apply! reads a single token ivar into a local, so there is no torn read; a concurrent refresh yields either the old-but-still-valid token or the new one, both usable. Validated with 50 threads and 5000 concurrent operations, plus 400 threads racing on a force-expired token, with zero errors.
• S3 with static keys: safe. Credentials never refresh, so the connection is immutable after construction.
• S3 with an IAM instance profile: not safe to share, so it is excluded. fog-aws refreshes STS credentials in place on the connection (access key, secret, session token, and signer) without synchronization, and a request reads the session token and the signer at different points with no lock. A shared connection can therefore sign a request with a token and signer from different refresh generations, which AWS rejects with a 403. This path is gated out with object_store_config.use_iam_profile? and keeps the previous request-scoped connection. GitLab.com uses GCS and is unaffected. The gem-level fix is https://github.com/fog/fog-aws/pull/759; the guard can be removed once it ships.

Feature flag

cache_ci_build_trace_chunk_fog_connection, type gitlab_com_derisk, default disabled. When disabled, the old request-scoped SafeRequestStore behavior is kept, so we can roll out and revert quickly.

Rollout issue: #623936

How to set up and validate locally

Requires object storage enabled for artifacts (gitlab_rails['object_store']), so build trace chunks use the Fog store.

1. Enable the feature flag: Feature.enable(:cache_ci_build_trace_chunk_fog_connection)
2. Launch a CI job (any pipeline whose job produces trace output; more than one 128 KB chunk is ideal so chunks flush to object storage).
3. Verify CI job traces still work: open the job and confirm the log renders in full, both while the job runs and after it finishes.

Unit tests: bundle exec rspec spec/models/ci/build_trace_chunks/fog_spec.rb

Validated end to end on a real S3-backed Omnibus install with the flag enabled: a job streamed about 600 KB of logs into 5 fog trace chunks, and the full trace read back correctly (all lines, start and end markers present) through the cached connection.

Tests

Added a #connection block to spec/models/ci/build_trace_chunks/fog_spec.rb covering:

• flag on: one connection is reused across instances and built only once
• flag off: the process-level cache is not populated
• object storage disabled: connection returns nil

Full file passes: 23 examples, 0 failures.

No changelog entry, since the change is behind a feature flag that is disabled by default.

Follow-up

size(model) still downloads the whole chunk with get_object on every status update, even once the token is cached. A stored size, for example a DB column, or a metadata HEAD request, would remove that object read entirely. Out of scope for this MR.