Add pipelines finished counter, histogram and UX SLI

What does this MR do and why?

Adds three observability signals emitted when a pipeline reaches a finished status (success, failed, or canceled), all gated behind the new ci_observe_pipelines_finished derisk feature flag (checked with the pipeline's project as actor):

  • pipelines_finished_total counter, labeled source/partition_id/status — the finished-side counterpart of pipelines_created_total (labels match so finished-vs-created ratio and traffic-cessation queries line up).
  • gitlab_ci_pipeline_time_to_finished_seconds histogram, labeled source/status, buckets 5m/25m/125m — wall-clock time from created_at to finished_at.
  • run_pipeline user experience SLI, observed over the same wall-clock interval — its structured log events carry gl_pipeline_id/gl_project_id (LabKit canonical field names) and pipeline_source/pipeline_status, enabling per-customer impact quantification during incidents that aggregate metrics cannot provide, plus the standard gitlab_user_experience_* counters. The custom 300s apdex threshold deliberately matches the histogram's first bucket: wall-clock duration is workload-dominated, so the apdex tracks the short-pipeline population, where a systemic processing slowdown is most visible.

Pipeline completion is not directly observable today: pipelines_created_total has no finished-side counterpart, and the duration attribute is the union of job running periods, which excludes queue and Sidekiq processing time — the very gaps where pipeline-processing incidents manifest. Wall clock from creation deliberately includes them.

Scope notes:

  • These signals observe at completion, so they quantify impact and confirm throughput (completion rate, distribution shift out of the low buckets); they do not directly detect stalls, because stuck pipelines never observe. Stall detection is being pursued separately (state-machine reconciliation discussion in the work item).
  • Retried pipelines finish again and are observed again anchored on the original created_at, folding in the user's decide-to-retry time. Accepted and documented on the metric definition.
  • skipped is excluded intentionally, matching the model's finished scope; finished_at is never set for skipped pipelines.
  • The Prometheus metrics reference is intentionally not updated here: the flag is a temporary derisk flag, so the docs entries ship with the flag-removal MR, when the metrics become generally available.

How to set up and validate locally

  1. Feature.enable(:ci_observe_pipelines_finished) in the Rails console.
  2. Run a small pipeline to completion.
  3. Check the metrics endpoint for pipelines_finished_total and gitlab_ci_pipeline_time_to_finished_seconds_bucket, and log/application_json.log for the run_pipeline user experience events.

References

Edited by Hordur Freyr Yngvason

Merge request reports

Loading