[FF] cache_ci_build_trace_chunk_fog_connection -- cache CI build trace chunk object storage connection

Summary

Roll out the feature currently behind the cache_ci_build_trace_chunk_fog_connection feature flag.

  • DRI: @stanhu
  • Team Slack channel: #g_pipeline_execution

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

The flag makes Ci::BuildTraceChunks::Fog reuse one object storage connection per process instead of rebuilding it per request. Blast radius is CI job trace reads and writes to object storage, which run on essentially every PUT /api/:version/jobs/:id. If the shared connection misbehaved, job trace reads/writes could error or slow down.

The known thread-safety risk (the fog-aws IAM instance-profile credential refresh race) is excluded by construction: the cache is skipped when object_store_config.use_iam_profile? is true. GitLab.com uses GCS, so it is unaffected. Watch the job API error rate and latency, and object storage error rates, on https://dashboards.gitlab.net while rolling out.

Rollout

Run all production /chatops in #production and cross-post the results to #g_pipeline_execution. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set cache_ci_build_trace_chunk_fog_connection 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set cache_ci_build_trace_chunk_fog_connection true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set cache_ci_build_trace_chunk_fog_connection <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss cache_ci_build_trace_chunk_fog_connection true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com cache_ci_build_trace_chunk_fog_connection true
/chatops gitlab run feature set --user=stanhu cache_ci_build_trace_chunk_fog_connection true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete cache_ci_build_trace_chunk_fog_connection --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set cache_ci_build_trace_chunk_fog_connection false                                         # production
/chatops gitlab run feature set cache_ci_build_trace_chunk_fog_connection false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete cache_ci_build_trace_chunk_fog_connection --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Stan Hu