## Overview Instrument GitLab Runner to collect and stream telemetry spans for CI job execution using OpenTelemetry Protocol (OTLP), pushing directly to the OTEL Collector via [LabKit](https://gitlab.com/gitlab-org/labkit). ## Parent Epic &20632 ## Phased Delivery ### Basic instrumentation (end-to-end integration + built-in stage spans) The goal is to validate the full pipeline — Runner → OTEL Collector → ClickHouse → Grafana — with a `job_execution` span and built-in stage child spans. | Issue | Status | Description | |-------|--------|-------------| | gitlab-org/gitlab-runner#39231+ | ✅ Closed (Apr 2026) | Feature negotiation (`features.tracing` from job payload), OTLP export client (via LabKit), and first `job_execution` span covering the full job lifecycle | | gitlab-org/gitlab-runner#39230+ | Ready for development | Spans for `prepare_executor`, `pull_image`, `get_sources`, `restore_cache`, `step_script`, `after_script`, `archive_cache`, `upload_artifacts` with stage-specific metadata | **Dependencies**: Rails-side feature negotiation (gitlab-org/gitlab#590588+, ✅ closed) and trace context initialization (gitlab-org/gitlab#590587+, ✅ closed) landed first. **Estimate**: ~1 week (per @ash2k) — basic instrumentation merged Apr 2026; stage spans pending. ### CI Functions spans Instrument CI Function invocations as child spans under `job_execution`. | Issue | Status | Description | |-------|--------|-------------| | gitlab-org/gitlab-runner#39271+ | Open | Spans for each CI Function invocation with function name, version, and status | **Estimate**: ~2 weeks (conservative) **Total Runner estimate: ~3 weeks** (basic instrumentation: ~1 week, CI Functions spans: ~2 weeks) ## Key Implementation Details - **LabKit integration**: Use [LabKit](https://gitlab.com/gitlab-org/labkit) for OTEL SDK integration — aligns with the rest of GitLab's instrumentation - **OTEL Collector endpoint from Rails**: The endpoint URL comes from Rails in `features.tracing.otel_endpoints` (single entry for MVC) — no static runner manager `config.toml` configuration needed. The Runner configures one OTLP exporter per endpoint. BYO OTLP destinations (additional customer-configured endpoints) are deferred to [future work](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/ci_job_telemetry/#future-work-byo-otlp-endpoints). - **OIDC/workload identity auth**: GitLab.com hosted runners authenticate directly using OIDC tokens - **`AlwaysOn` SDK sampling**: Override LabKit's default `TraceIDRatioBased(0.01)` sampler with `Config{SampleRate: 1.0}`. Sampling decisions are made by Rails (not the SDK) — if `features.tracing` is present, the pipeline was already sampled and the Runner instruments everything. - **Graceful degradation**: Telemetry failures must never fail jobs - **Streaming**: Spans are pushed as stages/functions complete (not batched at job end) ## Architecture Reference <https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/ci_job_telemetry/#gitlab-runner-changes>