[FF] ci_pipeline_otlp_trace_correlation -- CI pipeline OTLP trace correlation

Summary

Roll out the feature currently behind the ci_pipeline_otlp_trace_correlation feature flag.

The flag gates the deterministic TraceContext-based trace/span ID correlation for CI pipeline observability OTLP export (traces, logs, and metrics), and the associated pipeline webhook payload additions (root_pipeline_id, the bridges collection, and source_pipeline.bridge_id).

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

This flag gates additions to the CI pipeline webhook / integration payload (lib/gitlab/data_builder/pipeline.rb), which is high-traffic code that runs on every pipeline completion with active hooks/integrations. A previous unflagged version of this change (gitlab-org/gitlab!240239) caused an incident, which is why the change is now gated and rolled out incrementally.

Blast radius: pipeline webhook/integration payload serialization. Watch the Sidekiq and webhook/integration dashboards on https://dashboards.gitlab.net during rollout. The flag is a one-switch disable — set it to false to immediately revert to the prior payload shape.

Rollout

Run all production /chatops in #production and cross-post the results to #embody-team. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set ci_pipeline_otlp_trace_correlation 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set ci_pipeline_otlp_trace_correlation true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set ci_pipeline_otlp_trace_correlation <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss ci_pipeline_otlp_trace_correlation true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com ci_pipeline_otlp_trace_correlation true
/chatops gitlab run feature set --user=reneehendricksen ci_pipeline_otlp_trace_correlation true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check <merge-request-url> <milestone>
/chatops gitlab run feature delete ci_pipeline_otlp_trace_correlation --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set ci_pipeline_otlp_trace_correlation false                                         # production
/chatops gitlab run feature set ci_pipeline_otlp_trace_correlation false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete ci_pipeline_otlp_trace_correlation --dev --pre --staging --staging-ref --production  # remove entirely