Add pipelines finished counter, histogram and UX SLI
What does this MR do and why?
Adds three observability signals emitted when a pipeline reaches a finished status (success, failed, or canceled), all gated behind the new ci_observe_pipelines_finished derisk feature flag (checked with the pipeline's project as actor):
pipelines_finished_totalcounter, labeledsource/partition_id/status— the finished-side counterpart ofpipelines_created_total(labels match so finished-vs-created ratio and traffic-cessation queries line up).gitlab_ci_pipeline_time_to_finished_secondshistogram, labeledsource/status, buckets 5m/25m/125m — wall-clock time fromcreated_attofinished_at.run_pipelineuser experience SLI, observed over the same wall-clock interval — its structured log events carrygl_pipeline_id/gl_project_id(LabKit canonical field names) andpipeline_source/pipeline_status, enabling per-customer impact quantification during incidents that aggregate metrics cannot provide, plus the standardgitlab_user_experience_*counters. The custom 300s apdex threshold deliberately matches the histogram's first bucket: wall-clock duration is workload-dominated, so the apdex tracks the short-pipeline population, where a systemic processing slowdown is most visible.
Pipeline completion is not directly observable today: pipelines_created_total has no finished-side counterpart, and the duration attribute is the union of job running periods, which excludes queue and Sidekiq processing time — the very gaps where pipeline-processing incidents manifest. Wall clock from creation deliberately includes them.
Scope notes:
- These signals observe at completion, so they quantify impact and confirm throughput (completion rate, distribution shift out of the low buckets); they do not directly detect stalls, because stuck pipelines never observe. Stall detection is being pursued separately (state-machine reconciliation discussion in the work item).
- Retried pipelines finish again and are observed again anchored on the original
created_at, folding in the user's decide-to-retry time. Accepted and documented on the metric definition. skippedis excluded intentionally, matching the model'sfinishedscope;finished_atis never set for skipped pipelines.- The Prometheus metrics reference is intentionally not updated here: the flag is a temporary derisk flag, so the docs entries ship with the flag-removal MR, when the metrics become generally available.
How to set up and validate locally
Feature.enable(:ci_observe_pipelines_finished)in the Rails console.- Run a small pipeline to completion.
- Check the metrics endpoint for
pipelines_finished_totalandgitlab_ci_pipeline_time_to_finished_seconds_bucket, andlog/application_json.logfor therun_pipelineuser experience events.
References
Edited by Hordur Freyr Yngvason