[FF] ci_observe_pipelines_finished -- Emit pipelines finished Prometheus metrics

Summary

Roll out the feature currently behind the ci_observe_pipelines_finished feature flag.

  • DRI: @hfyngvason
  • Team Slack channel: #g_pipeline-execution

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

Introduced in !244288 (merged). This is a gitlab_com_derisk flag gating the emission of two Prometheus metrics on pipeline completion (pipelines_finished_total and gitlab_ci_pipeline_time_to_finished_seconds), checked with the pipeline's project as actor. The flag is temporary: once emission proves stable on GitLab.com, it is removed and the metrics become unconditional.

What could go wrong?

No user-facing behavior changes; the flag only gates two metric emissions inside a run_after_commit block on pipeline terminal transitions (success/failed/canceled).

  • Blast radius: a counter increment and a histogram observation per finished pipeline, executed in the process performing the transition (mostly Sidekiq). Label cardinality is bounded (source × status × partition_id).
  • No data-loss risk.
  • Watch Sidekiq apdex/error rates for pipeline-processing workers on https://dashboards.gitlab.net after enabling, and confirm the new series appear and look sane.

Rollout

Run all production /chatops in #production and cross-post the results to #g_pipeline-execution. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set ci_observe_pipelines_finished 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set ci_observe_pipelines_finished true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set ci_observe_pipelines_finished <percentage> --actors

Or target specific actors first:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss ci_observe_pipelines_finished true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

  • Docs intentionally not added while the flag exists — the metrics are documented in the Prometheus metrics reference as part of the flag-removal MR (see Cleanup)
  • No breaking changes
  • Change management issue opened, if required
  • Not used by external API consumers

Cleanup

As a gitlab_com_derisk flag, this must be removed promptly once deemed stable — the metrics then emit unconditionally. The flag-removal MR also adds pipelines_finished_total and gitlab_ci_pipeline_time_to_finished_seconds to the Prometheus metrics reference (doc/administration/monitoring/prometheus/gitlab_metrics.md). See cleaning up. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/244288 19.2
/chatops gitlab run feature delete ci_observe_pipelines_finished --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set ci_observe_pipelines_finished false                                         # production
/chatops gitlab run feature set ci_observe_pipelines_finished false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete ci_observe_pipelines_finished --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Hordur Freyr Yngvason