# Problem When operating complex CI/CD pipelines at scale, not only do we need to track pipeline and job runtimes, but also runtimes, success rates etc of separate job sections like: * Prepare stage setup and docker image pull times * Repository clone operations times and success rates * Cache hit/miss rates and download/upload times * Artifact upload times, success rates, sizes etc These metrics help identify various problems early and identify impact much more easily without manually diving in to separate job logs. Currently no real solution exists on how to do this. We have some instrumentation that runs from within the jobs themselves but it's very brittle and hard to maintain due to the fact that it depends on pipeline implementation itself and doesn't support some of the actions that happen after the `after_script` block has finished. This adhoc tools also do not benefit other users of GitLab given these observability features don't end up as part of the product even if it somewhat works. # Customer Verbatim 1. "To analyze a specific job, some more detailed information would be required for the job in addition to the pipeline as described above. The following parameters would help here: * Duration (split into the sections of the log to identify the possible bottlenecks, and with the custom collapsible sections, it is possible to split the job into separate steps to analyze)" (See https://gitlab.com/groups/gitlab-org/-/epics/11835#note_1631621923) 2. Currently, users can only see the what's happening on the application layer with the job log. To locate the root cause of the issue, users need to correlate what's happening at the GitLab job level with what's happening on the runner and infrastructure side at the same moment. (See [customer](https://gitlab.com/gitlab-com/user-interviews/-/work_items/29#note_1569976852)) # Solution ### MVC Proposal ### Metrics Collection and Storage (Proposed) See the list of in-scope system metrics - https://gitlab.com/groups/gitlab-org/quality/analytics/-/epics/22#system-level-metrics-primary-focus GitLab extracts timing and metadata from these sections and stores them in **ClickHouse** ### System-Level Metrics (PRIMARY FOCUS) Operations that happen _outside_ user control—before and after script execution: | Category | Metrics | |----------|---------| | All sections individually inside CI jobs (e.g. `before_script`, `after_script`, `artifacts upload`) | Duration, Start time, Finish time | | Job Preparation | • Docker image name<br>• Docker image pull duration<br>• Error messages (nice to have) | | Git Operations | • Kind (git pull, git fetch, git clone, nothing, ...)<br>• Params passed to the git operation (depth, ...)<br>• Duration | | Cache Operations | Cache key, Upload times, Download times, Hit rates | | Artifact Operations | • Artifact download duration (relatively less critical if we need to look at scope)<br>• Artifact upload duration<br>• Artifact key | #### Example metrics payload for a finished job <details> <summary>Example payload that could be ingested into ClickHouse</summary> ```json { "schema_version": "1.0", "reported_at": "2025-01-12T14:32:45.123Z", "functions": [ { "name": "pull_image", "start_time": "2025-01-12T14:29:58.150Z", "finish_time": "2025-01-12T14:30:00.000Z", "metadata": { "docker_image": "registry.gitlab.com/mygroup/myproject/ci-image:v2.3.1" } }, { "name": "prepare_executor", "start_time": "2025-01-12T14:30:00.000Z", "finish_time": "2025-01-12T14:30:02.150Z" }, { "name": "prepare_script", "start_time": "2025-01-12T14:30:02.150Z", "finish_time": "2025-01-12T14:30:02.400Z" }, { "name": "get_sources", "start_time": "2025-01-12T14:30:02.400Z", "finish_time": "2025-01-12T14:30:08.900Z", "metadata": { "git_kind": "fetch", "git_depth": 50, "git_filter": "blob:none" } }, { "name": "restore_cache", "start_time": "2025-01-12T14:30:08.900Z", "finish_time": "2025-01-12T14:30:12.100Z", "metadata": { "cache_key": "ruby-gems-a1b2c3d4e5f6", "cache_hit": true } }, { "name": "restore_cache", "start_time": "2025-01-12T14:30:12.100Z", "finish_time": "2025-01-12T14:30:12.150Z", "metadata": { "cache_key": "node-modules-f6e5d4c3b2a1", "cache_hit": false } }, { "name": "download_artifacts", "start_time": "2025-01-12T14:30:12.150Z", "finish_time": "2025-01-12T14:30:14.600Z", "metadata": { "artifact_key": "compiled-assets", "artifact_type": "archive" } }, { "name": "step_script", "start_time": "2025-01-12T14:30:14.600Z", "finish_time": "2025-01-12T14:32:30.000Z" }, { "name": "after_script", "start_time": "2025-01-12T14:32:30.000Z", "finish_time": "2025-01-12T14:32:31.200Z" }, { "name": "archive_cache", "start_time": "2025-01-12T14:32:31.200Z", "finish_time": "2025-01-12T14:32:35.500Z", "metadata": { "cache_key": "node-modules-f6e5d4c3b2a1" } }, { "name": "upload_artifacts", "start_time": "2025-01-12T14:32:35.500Z", "finish_time": "2025-01-12T14:32:45.000Z", "metadata": { "artifact_key": "rspec-junit-report", "artifact_type": "junit" } } ] } ``` </details> ### UI (Customer 0 - DevExp) **Deliverable**: Teams can visualize these job-level system metrics and identify trends Option 1: No in-product visualization - DevExp Team will create their own dashboards (Grafana) using the data available in ClickHouse > Currently, we have our [own Grafana instance](https://dashboards.devex.gitlab.net/) and our own ClickHouse instance that we query from Grafana dashboards. Assuming we could get access to the ClickHouse instance we're pushing this data from with a Grafana instance, we could make the dashboards ourselves to start with (we call this our [data silver path](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/developer-experience/data-golden-path/#-silver-path-good-alternative) :second_place: ) <details> <summary> Out of Scope (Post MVC) </summary> ### Out of Scope (Post MVC) #### User-Level Telemetry (PARTIALLY SOLVED) Custom metrics from within user scripts (test coverage, build times, etc.) can already be captured via the existing `metrics_reports` feature at an MR level. Limitation: The data being trapped in individual MR contexts means we can't do trend analysis or pipeline-level insights. **Create a separate epic.** **Deliverable**: Users can instrument the jobs with custom metrics and are able to track and report on those custom metrics. #### UI (TBD) Option 2: Show the above metrics in Project Level -\> CI/CD Analytics Dashboard Option 3: Show the custom collapsible sections in CI/CD job log UI </details> ## Technical Proposal **Full architecture design document**: [CI Job Telemetry Reporting](/handbook/engineering/architecture/design-documents/ci_job_telemetry/) ### Key Architectural Decisions | Decision | Resolution | |----------|------------| | **Telemetry format** | OpenTelemetry Protocol (OTLP) - industry standard with span hierarchy support | | **Transport** | gRPC streaming - real-time telemetry as CI Functions complete | | **Routing** | Runner → KAS → Data Insights Platform (DIP) → ClickHouse | | **Authentication** | KAS validates runner/job tokens, forwards to DIP as trusted source | ### MVC Cross-Team Dependencies | Team | Responsibility | Status | |------|----------------|--------| | **Runner Core** (Verify) | Runner instrumentation to collect and stream telemetry spans | Design phase | | **Runner Core** (Verify) | KAS telemetry proxy (gRPC endpoint, DIP forwarding) | Design phase | | **CI Platform** (Verify) | Rails auth endpoint (`/api/v4/internal/jobs/verify_telemetry_auth`) | Not started | | **Platform Insights** | DIP instance setup, OTEL Ingester, CI Telemetry Exporter | [Needs development](https://gitlab.com/gitlab-org/analytics-section/platform-insights/core/-/issues/124) | ### Post-MVC Phases 1. **Phase 2** - Job Router telemetry (KAS scheduling spans) 2. **Phase 3** - Rails integration (job lifecycle spans) 3. **Phase 4** - Self-Managed and Dedicated deployment 4. **Phase 5** - Alerting 5. **Phase 6** - CI Functions DAG telemetry 6. **Phase 7** - In-product visualization ## Success Metric (Proposed) Decrease internal GitLab Pipelines Failure Rate by X %