## Summary W3C Trace Context (`TRACEPARENT` / `TRACESTATE`) is **not available to CI job scripts** today. The observability export system (`lib/gitlab/observability/pipeline_to_traces.rb`) generates trace data for pipelines, but uses random trace IDs and never exposes trace context to user scripts. This means: - No `TRACEPARENT` environment variable is injected into CI jobs - No `traceparent` header can be propagated to external HTTP calls - No `TRACESTATE` support exists - Cross-pipeline and job-to-external-service trace correlation is impossible - Each pipeline export gets a random `SecureRandom.hex(16)` trace ID — even parent/child pipelines are unrelated traces This issue tracks the plan to: 1. Extract trace ID / span ID derivation into a **shared CE module** (`lib/gitlab/ci/trace_context.rb`) so that all systems — observability export, CI variables, and the existing EE job telemetry — produce identical trace context values 2. Make `PipelineToTraces` use **deterministic** trace IDs derived from the pipeline hierarchy 3. Expose `TRACEPARENT` and `TRACESTATE` as **predefined CI variables** so user scripts can propagate trace context to downstream services 4. Add trace context correlation to `PipelineToLogs` and `PipelineToMetrics` so all three OTLP signal types (traces, logs, metrics) can be cross-referenced in observability backends **Use cases:** UC-9 (cross-pipeline correlation), UC-10 (TRACEPARENT propagation to external services) **Origin:** Ericsson Co-Create Sessions 3 & 5 --- ## Current State ### Observability Export (`lib/gitlab/observability/`) The export pipeline works but has no trace correlation: | Component | File | What It Does | |---|---|---| | `ExportService` | `app/services/ci/observability/export_service.rb` | Triggered on pipeline completion; checks `GITLAB_OBSERVABILITY_EXPORT` CI variable; orchestrates conversion + export | | `PipelineToTraces` | `lib/gitlab/observability/pipeline_to_traces.rb` | Converts pipeline data to OTLP trace format. **Uses `SecureRandom.hex(16)` for trace_id** (line 376-378) and `SecureRandom.hex(8)` for span_id (line 386-388). | | `OtelExporter` | `lib/gitlab/observability/otel_exporter.rb` | HTTP POST to OTLP collector endpoint | | `DataBuilder::Pipeline` | `lib/gitlab/data_builder/pipeline.rb` | Builds pipeline data hash including `source_pipeline` info | ### What's missing ```mermaid sequenceDiagram participant Rails as GitLab Rails participant Runner as GitLab Runner participant Env as Job Environment participant Script as User Script Rails->>Runner: POST /api/v4/jobs/request<br/>(job payload + variables) Note over Rails,Runner: No TRACEPARENT in variables Runner->>Env: Set env vars:<br/>CI_*, GITLAB_*, custom vars Note over Env: No TRACEPARENT env var Runner->>Script: Execute script Script->>Script: $ curl https://deploy.example.com Note over Script: No traceparent header<br/>on outbound HTTP call ``` And on the export side, each pipeline gets an isolated random trace: ```mermaid graph TB subgraph "Parent Pipeline Export" PP["pipeline span<br/>traceId: random_A"] PJ1["job span<br/>traceId: random_A"] PJ2["bridge job<br/>(no span emitted)"] PP --> PJ1 PP --> PJ2 end subgraph "Child Pipeline Export" CP["pipeline span<br/>traceId: random_B<br/>attr: source_pipeline.pipeline_id=parent_id"] CJ1["job span<br/>traceId: random_B"] CP --> CJ1 end PJ2 -.->|"triggers<br/>(no trace link)"| CP style PJ2 fill:#e24329,color:#fff style CP fill:#fca326,color:#000 ``` --- ## Shared CE Module: `Gitlab::Ci::TraceContext` ### Why a shared module is needed An existing EE system (`ee/lib/gitlab/ci/job_telemetry/span_ids.rb`) already derives deterministic trace IDs and span IDs for CI pipelines: ```ruby # ee/lib/gitlab/ci/job_telemetry/span_ids.rb (existing EE code) def trace_id_for(root_pipeline_id) format("%032x", root_pipeline_id) # => "00000000000000000000000000000042" end def for_job(root_pipeline_id, job_id, kind) input = "#{root_pipeline_id}:#{job_id}:#{kind}" OpenSSL::Digest::SHA256.hexdigest(input)[0, 16] # => "a1b2c3d4e5f67890" end ``` If we reimplement these formulas independently in `lib/gitlab/observability/`, the two systems will produce the same values *today* — but there's no guarantee they stay in sync. If either side changes the formula, EE customers get **two unrelated trace trees** per pipeline (one from `SpanEmitter`, one from `PipelineToTraces` / TRACEPARENT) with no compile-time error. ### Proposed: extract to a shared CE module Create `lib/gitlab/ci/trace_context.rb` in CE with the canonical derivation formulas: ```ruby # lib/gitlab/ci/trace_context.rb (NEW — shared CE module) # frozen_string_literal: true module Gitlab module Ci module TraceContext TRACE_ID_HEX_LENGTH = 32 SPAN_ID_HEX_LENGTH = 16 class << self # Deterministic trace ID from the root pipeline's database ID. # All spans across the pipeline hierarchy (same-project parent-child) # share this trace_id. # # Matches the W3C trace-id format: 32 lowercase hex characters. def trace_id_for(root_pipeline_id) format("%0#{TRACE_ID_HEX_LENGTH}x", root_pipeline_id) end # Deterministic span ID for a job. # `kind` disambiguates when multiple spans exist for the same job # (e.g., the observability export span vs the TRACEPARENT variable). # # Matches the W3C parent-id format: 16 lowercase hex characters. def span_id_for_job(root_pipeline_id, job_id, kind = :default) input = "#{root_pipeline_id}:#{job_id}:#{kind}" ::OpenSSL::Digest::SHA256.hexdigest(input)[0, SPAN_ID_HEX_LENGTH] end # Span ID for a pipeline span (used by PipelineToTraces). def span_id_for_pipeline(root_pipeline_id, pipeline_id) input = "pipeline:#{root_pipeline_id}:#{pipeline_id}" ::OpenSSL::Digest::SHA256.hexdigest(input)[0, SPAN_ID_HEX_LENGTH] end # Span ID for a bridge (trigger) job, used to link child pipeline # spans to their triggering job. def span_id_for_bridge(bridge_id) format("%0#{SPAN_ID_HEX_LENGTH}x", bridge_id) end # Build a W3C traceparent value. # version=00, flags=01 (sampled). def build_traceparent(root_pipeline_id, job_id, kind = :default) tid = trace_id_for(root_pipeline_id) sid = span_id_for_job(root_pipeline_id, job_id, kind) "00-#{tid}-#{sid}-01" end end end end end ``` ### Then refactor EE `SpanIds` to delegate ```ruby # ee/lib/gitlab/ci/job_telemetry/span_ids.rb (refactored) module Gitlab module Ci module JobTelemetry module SpanIds class << self def trace_id_for(root_pipeline_id) ::Gitlab::Ci::TraceContext.trace_id_for(root_pipeline_id) end def for_job(root_pipeline_id, job_id, kind) ::Gitlab::Ci::TraceContext.span_id_for_job(root_pipeline_id, job_id, kind) end def for_bridge(bridge_id) ::Gitlab::Ci::TraceContext.span_id_for_bridge(bridge_id) end # This method stays EE-only — it's specific to the phantom parent pattern def for_pipeline_phantom(root_pipeline_id) ::OpenSSL::Digest::SHA256.hexdigest("pipeline:#{root_pipeline_id}")[0, SPAN_ID_HEX_LENGTH] end end end end end end ``` ### All consumers use the same module ```mermaid graph TB TC["lib/gitlab/ci/trace_context.rb<br/>(shared CE module)"] PT["PipelineToTraces<br/>lib/gitlab/observability/"] VB["Variables Builder<br/>lib/gitlab/ci/variables/"] SI["EE SpanIds<br/>ee/lib/gitlab/ci/job_telemetry/"] TC --> PT TC --> VB TC --> SI PT -.->|"trace_id_for + span_id_for_job"| EXPORT["OTLP Export"] VB -.->|"build_traceparent"| VAR["$TRACEPARENT CI variable"] SI -.->|"trace_id_for + for_job"| EMIT["EE SpanEmitter"] style TC fill:#34a853,color:#fff style EXPORT fill:#4285f4,color:#fff style VAR fill:#4285f4,color:#fff style EMIT fill:#4285f4,color:#fff ``` **Guarantee:** All three systems produce identical trace_id and span_id values for the same pipeline/job, enforced by a single source of truth. A shared spec asserts `TraceContext.trace_id_for(id) == SpanIds.trace_id_for(id)` for the EE case. --- ## Proposed Architecture ### 1. Deterministic trace IDs in `PipelineToTraces` Replace `SecureRandom.hex(16)` with a derivation from `pipeline.root_ancestor.id`: ```ruby # lib/gitlab/observability/pipeline_to_traces.rb def pipeline_trace_id Gitlab::Ci::TraceContext.trace_id_for(root_pipeline_id) end def root_pipeline_id # pipeline_data[:object_attributes][:root_pipeline_id] populated # by DataBuilder::Pipeline from pipeline.root_ancestor.id pipeline_data.dig(:object_attributes, :root_pipeline_id) || pipeline[:id] end ``` > **Note:** `pipeline.root_ancestor` is a CE method on `Ci::Pipeline` (line 1323). It walks same-project parent-child links via `object_hierarchy(project_condition: :same)`. No EE dependency. ### 2. Inject TRACEPARENT as a predefined CI variable ```ruby # lib/gitlab/ci/variables/builder/pipeline.rb def predefined_traceparent_variables return Gitlab::Ci::Variables::Collection.new unless traceparent_enabled? root_id = pipeline.root_ancestor.id traceparent = Gitlab::Ci::TraceContext.build_traceparent(root_id, job.id) Gitlab::Ci::Variables::Collection.new.tap do |variables| variables.append(key: 'TRACEPARENT', value: traceparent) variables.append(key: 'TRACESTATE', value: "gitlab=pipeline:#{pipeline.id};job:#{job.id}") end end ``` ### 3. End-to-end flow ```mermaid sequenceDiagram participant Rails as GitLab Rails participant Runner as GitLab Runner participant Env as Job Environment participant Script as User Script participant Ext as Deploy Target Rails->>Rails: TraceContext.build_traceparent(root_ancestor.id, job.id) Rails->>Runner: POST /api/v4/jobs/request<br/>variables: [..., TRACEPARENT=00-{tid}-{sid}-01] Runner->>Env: Set env vars including<br/>TRACEPARENT and TRACESTATE Note over Env: Available to all scripts Runner->>Script: Execute script Script->>Script: $ curl -H "traceparent: $TRACEPARENT" \<br/> https://deploy.example.com Script->>Ext: HTTP request with traceparent Note over Ext: Continues the trace<br/>(same trace_id as pipeline export) ``` --- ## W3C Trace Context Format ```mermaid graph LR TP["traceparent header / env var"] TP --> V["version<br/>'00'"] TP --> TID["trace-id<br/>32 hex chars<br/>(128-bit)"] TP --> PID["parent-id<br/>16 hex chars<br/>(64-bit span id)"] TP --> FL["flags<br/>'01' = sampled<br/>'00' = not sampled"] style TID fill:#4285f4,color:#fff style PID fill:#34a853,color:#fff ``` Example: ``` TRACEPARENT=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 +--+ +----------------+----------------+ +------+-------+ +-+ version trace-id parent-span-id flags ``` --- ## Design Decision: Cross-Pipeline Trace Propagation When Pipeline A triggers Pipeline B (via `trigger:` keyword), should B's jobs get the same `trace_id` as A's jobs? ### How `root_ancestor` vs `upstream_root` work Both are CE methods on `Ci::Pipeline` (`app/models/ci/pipeline.rb`): - **`pipeline.root_ancestor`** (line 1323) — walks up **same-project** parent-child links only (`project_condition: :same`). Cross-project boundaries stop the traversal. - **`pipeline.upstream_root`** (line 1330) — walks up **all** upstream links including cross-project. ### Correlation matrix | Scenario | `root_ancestor` (recommended) | `upstream_root` (alternative) | |---|---|---| | Parent -> Child (same project) | :white_check_mark: Same trace | :white_check_mark: Same trace | | A -> B -> C (all same project) | :white_check_mark: All share root A | :white_check_mark: All share root A | | Project X -> Project Y (cross-project) | :x: Separate traces | :white_check_mark: Same trace | | X -> Y -> Z (all different projects) | :x: Three separate traces | :white_check_mark: All share root X | | X -> Y child -> Y grandchild | Y subtree shares trace; X separate | :white_check_mark: All share root X | ### Design options | Option | Approach | Pros | Cons | |---|---|---|---| | **A. Use `root_ancestor.id`** | Same-project parent-child correlation | Safe — no cross-project info leak. Simple. Covers the most common case (parent-child pipelines). | Multi-project pipelines get isolated traces. | | **B. Use `upstream_root.id`** | Full cross-project correlation | Complete end-to-end tracing across project boundaries. | Project Y's TRACEPARENT reveals Project X's pipeline ID. Security review needed. | | **C. Propagate TRACEPARENT through bridge variables** | Parent pipeline's TRACEPARENT flows as a variable to child pipeline | Most flexible — opt-in, follows distributed tracing conventions (caller passes context to callee). | Requires bridge job to emit a span (not implemented yet). | **Recommendation:** Start with **Option A** (`root_ancestor.id`). It's safe, requires no cross-project security decisions, and covers same-project parent/child pipelines which is the most common hierarchy. Option C is the ideal long-term target. ### Bridge job gap For cross-pipeline traces to render correctly, the **bridge (trigger) job** needs to emit its own span so the trace tree connects. Currently: - `PipelineToTraces#build_spans` only emits pipeline and job spans — bridge jobs that trigger child pipelines are included as regular job spans but have **no parent-child link** to the child pipeline's spans - The child pipeline's `source_pipeline.pipeline_id` attribute is informational only — it's not an OTel span link - Trace visualizers will show a gap between parent and child pipeline spans ```mermaid graph TB subgraph "Parent Pipeline (trace_id: T1)" PJ1["job: build<br/>span_id: S1"] PJ2["bridge: trigger_child<br/>span_id: S2<br/>(no child link)"] end subgraph "Child Pipeline (trace_id: T1, same project)" CJ1["job: test<br/>parentSpanId: pipeline_span<br/>(should be S2 but isn't)"] CJ2["job: deploy<br/>parentSpanId: pipeline_span"] end PJ2 -.->|"triggers<br/>(trace link missing)"| CJ1 style PJ2 fill:#e24329,color:#fff style CJ1 fill:#fca326,color:#000 style CJ2 fill:#fca326,color:#000 ``` **Fix:** In `PipelineToTraces`, when a child pipeline is exported: if `source_pipeline` exists and shares the same `root_ancestor`, set the child pipeline span's `parentSpanId` to a deterministic derivation of the bridge job's ID. This connects the trace tree without requiring both pipelines to be exported simultaneously. --- ## Consolidated Gap Analysis | Priority | Component | Gap | Current State | Recommended Fix | |---|---|---|---|---| | **High** | New CE module | No shared trace context derivation | EE `SpanIds` has formulas; CE has none; risk of divergence | Create `lib/gitlab/ci/trace_context.rb` as single source of truth; refactor EE `SpanIds` to delegate | | **High** | `PipelineToTraces` | Random trace_id per pipeline | `SecureRandom.hex(16)` — no correlation | Call `TraceContext.trace_id_for(root_ancestor.id)` — deterministic, hierarchical | | **High** | Variables builder | `TRACEPARENT` not available as CI variable | No CI variable exists | Inject as predefined variable using same trace_id derivation as export | | **High** | Variables builder | Per-attempt span_id needed for retries | Not implemented | `SHA256(root_id:job_id)` — deterministic per job; for retries, include attempt number | | **High** | `PipelineToTraces` | Span IDs are random | `SecureRandom.hex(8)` per span | Call `TraceContext.span_id_for_job(root_id, job_id)` so TRACEPARENT and export use same span_id | | **Medium** | Variables builder | `TRACESTATE` not available | No vendor context | Inject `TRACESTATE=gitlab=pipeline:{id};job:{id}` alongside TRACEPARENT | | **Medium** | `PipelineToTraces` | Child pipeline spans not linked to parent | Only `source_pipeline.pipeline_id` attribute, no span link | Set child pipeline span's `parentSpanId` to deterministic derivation of bridge job ID when `root_ancestor` matches | | **Medium** | `PipelineToTraces` | Bridge jobs not distinguished | Treated as regular job spans | Mark bridge spans; use their span_id as parent for child pipeline span | | **Medium** | `DataBuilder::Pipeline` | `root_pipeline_id` not in pipeline data hash | Not included | Add `root_pipeline_id: pipeline.root_ancestor.id` to `object_attributes` | | **Medium** | Docs | No user-script propagation examples | Users won't know to forward the header | CI templates and snippets: `curl -H "traceparent: $TRACEPARENT"` | | **Low** | Runner | No per-section TRACEPARENT | All sections share one span_id | Runner emits fresh child span per section (Pattern C, long-term) | | **Low** | Runner | Trace context not preserved across retries | No link to previous attempts | Add OTel `span_link` to previous attempt's span | | **Low** | Rails | No `traceparent` in webhook payloads | Webhooks carry no trace context | Add `X-GitLab-Traceparent` header to CI-originated webhooks | | **High** | `PipelineToLogs` / `PipelineToMetrics` | No trace context on log records or metric data points | Log records have no `traceId`/`spanId`; metrics have no exemplars; all three OTLP signal types are uncorrelated | Add `traceId`/`spanId` to log records; add exemplars + `gitlab.trace_id` resource attribute to metrics (#94) | | **Low** | All three converters | Duplicated `build_resource`, `build_scope`, `service_name`, trace context helpers | ~6-8 identical methods copy-pasted across `PipelineToTraces`, `PipelineToLogs`, `PipelineToMetrics` | Extract shared `PipelineConverterBase` concern module (deleted — folded into #94) | --- ## Security Considerations ### Trace ID reversibility The deterministic trace_id (`format('%032x', root_pipeline_id)`) is trivially reversible — anyone with the TRACEPARENT can recover the pipeline ID by converting the hex back to an integer. **This is acceptable** because `CI_PIPELINE_ID` is already a public, non-sensitive predefined variable. The trace_id exposes no new information. ### Variable sensitivity classification TRACEPARENT should use the default variable flags (`public: true, masked: false`), consistent with `CI_PIPELINE_ID`, `CI_JOB_ID`, `CI_COMMIT_SHA`, and other non-sensitive predefined variables. ### Forked pipeline behavior TRACEPARENT should be available in fork pipelines since it contains no secrets. The trace_id will be derived from the fork pipeline's own `root_ancestor.id`. ### Cross-project trace context (if Option B or C is adopted later) If TRACEPARENT propagates across project boundaries, Project Y's jobs would receive a trace_id derived from Project X's root pipeline ID. This is comparable to the already-existing `CI_UPSTREAM_PIPELINE_ID` variable which already reveals the parent's pipeline ID. Still, changing trace_id scope from `root_ancestor` to `upstream_root` should require explicit security review. ### Existing cross-pipeline CI variables Jobs in child/downstream pipelines already receive: - `CI_UPSTREAM_PIPELINE_ID` — the parent pipeline's ID - `CI_UPSTREAM_PROJECT_ID` — the parent project's ID - `CI_UPSTREAM_JOB_ID` — the bridge job's ID (Defined in `lib/gitlab/ci/variables/builder/pipeline.rb`, lines 128-134) TRACEPARENT would not expose anything beyond what these already reveal. --- ## Integration with Section Spans If TRACEPARENT is injected **per section span** rather than per job, downstream services can be correlated to the specific build section that called them: ```mermaid graph TB P["pipeline span<br/>trace_id: T1"] J["job span<br/>trace_id: T1<br/>span_id: J1<br/>parent: pipeline"] SC["section: step_script<br/>trace_id: T1<br/>span_id: S1<br/>parent: J1"] DEPLOY["External: deploy.example.com<br/>traceparent: 00-T1-S1-01<br/>(child of section span)"] P --> J --> SC SC -.->|"HTTP call with<br/>traceparent header"| DEPLOY style P fill:#4285f4,color:#fff style J fill:#e24329,color:#fff style SC fill:#7b1fa2,color:#fff style DEPLOY fill:#fabc05,color:#000 ``` This requires the Runner to re-emit TRACEPARENT per section (Pattern C — Runner-side OTel SDK). This is a long-term goal, not part of the initial phases. --- ## TRACESTATE for Vendor Context `TRACESTATE` carries vendor-specific keys that follow the trace through the system: ``` TRACESTATE=gitlab=pipeline:12345;job:67890 ``` > **Note:** The TRACESTATE format must comply with the [W3C Tracestate specification](https://www.w3.org/TR/trace-context/#tracestate-header). The vendor value must only contain characters allowed by the spec grammar. Validate before implementation. --- ## Failure Modes & Mitigations | Failure | Behavior | Mitigation | |---------|----------|------------| | TRACEPARENT not yet available (older GitLab) | Job receives unset env var | Job scripts must tolerate missing var (`${TRACEPARENT:-}`) | | `root_ancestor` returns `self` for cross-project pipeline | Each project-boundary pipeline gets its own trace_id | Expected under Option A; document for users | | User script doesn't forward the header | External services start a new trace | Document best practice; provide CI template snippets | | Runner caches stale TRACEPARENT across retries | Wrong span context propagated | TRACEPARENT must be regenerated per job execution attempt | | Pipeline re-exported (e.g. after retry) | Deterministic trace_id produces same value | Correct behavior — idempotent | | Child pipeline exported before parent | Child trace_id matches parent's (same root_ancestor) | Correct — spans connect regardless of export order | | Formula drift between TraceContext and EE SpanIds | Traces from different systems diverge silently | Shared spec in `ee/spec/` asserts `TraceContext.trace_id_for(id) == SpanIds.trace_id_for(id)` for all derivations | --- ## Phased Delivery Plan ### Dependency DAG ``` #90 (TraceContext module) ─┬─► #91 (deterministic export IDs) ─┬─► #93 (cross-pipeline linking) │ └─► #94 (trace context on logs/metrics) └─► #92 (TRACEPARENT CI variable + docs) ``` ### Phase 1: Shared Module + Deterministic Trace IDs + TRACEPARENT CI Variable **No Runner changes required.** CI variables flow through the existing variable pipeline to the Runner automatically. | MR | Task | Depends on | Status | |----|------|-----------|--------| | #90 | Create `lib/gitlab/ci/trace_context.rb` + refactor EE `SpanIds` to delegate | Nothing — foundation | Open | | #91 | Deterministic trace IDs in `PipelineToTraces` + add `root_pipeline_id` to `DataBuilder::Pipeline` | #90 | Open | | #92 | Inject `TRACEPARENT` + `TRACESTATE` as predefined CI variables + update predefined variables docs | #90 | Open | ### Phase 2: Cross-Signal Correlation | MR | Task | Depends on | Status | |----|------|-----------|--------| | #93 | Cross-pipeline span linking — child pipeline `parentSpanId` → bridge job span; OTel span links for cross-project | #91 | Open | | #94 | Add `traceId`/`spanId` to `PipelineToLogs` log records + `PipelineToMetrics` exemplars + `gitlab.trace_id` resource attribute | #91 | Open | ### Phase 3: Runner-Side Enhancements (requires Runner release, future) - [ ] Per-section TRACEPARENT — Runner re-emits per build section (Pattern C / Runner OTel SDK) - [ ] Optional auto-inject feature flag for outbound HTTP `traceparent` header - [ ] `X-GitLab-Traceparent` header on webhook deliveries from CI jobs - [ ] Retry span links — OTel `span_link` to previous attempt's span --- ## Key Code References | File | What It Does | Action | |---|---|---| | `lib/gitlab/ci/trace_context.rb` | **NEW** — shared CE module for trace_id/span_id derivation | **Create** — single source of truth for all trace context formulas | | `ee/lib/gitlab/ci/job_telemetry/span_ids.rb` | EE deterministic span ID derivation | **Refactor** — delegate to `TraceContext`; keep EE API unchanged | | `lib/gitlab/observability/pipeline_to_traces.rb` | Converts pipeline to OTLP traces; random trace_id | **Modify** — call `TraceContext` for deterministic trace_id + span_id | | `lib/gitlab/observability/pipeline_to_logs.rb` | Converts pipeline to OTLP logs; no trace context on log records | **Modify** — add `traceId`/`spanId` to log records (#94) | | `lib/gitlab/observability/pipeline_to_metrics.rb` | Converts pipeline to OTLP metrics; no trace context on data points | **Modify** — add exemplars + `gitlab.trace_id` resource attribute (#94) | | `lib/gitlab/ci/variables/builder/pipeline.rb` | Pipeline predefined vars (`CI_PIPELINE_ID`, `CI_UPSTREAM_*`) | **Modify** — add `TRACEPARENT` + `TRACESTATE` using `TraceContext` | | `lib/gitlab/ci/variables/builder.rb` | Assembles all CI variables for a job | **Modify** — concat traceparent variables into `scoped_variables` | | `lib/gitlab/data_builder/pipeline.rb` | Builds pipeline data hash for export | **Modify** — add `root_pipeline_id` to `object_attributes` | | `app/services/ci/observability/export_service.rb` | Orchestrates pipeline export on completion | **Reference** — triggers `PipelineToTraces` | | `app/models/ci/pipeline.rb` | Pipeline model; `root_ancestor` (line 1323), `upstream_root` (line 1330) | **Use** — `root_ancestor` for trace_id derivation | | `app/models/ci/bridge.rb` | Bridge job model; triggers child/downstream pipelines | **Reference** — bridge ID used for child span linking | | `app/models/ci/build.rb` | Build model; `variables` method assembles CI vars | **Reference** — variable assembly chain | --- ## References - [W3C Trace Context specification](https://www.w3.org/TR/trace-context/) - [W3C Tracestate specification](https://www.w3.org/TR/trace-context/#tracestate-header) - [OTel Span Context specification](https://opentelemetry.io/docs/specs/otel/trace/api/#spancontext) - Co-Create Session 3: OTel Compliance & Export (UC-9, UC-10) - Co-Create Session 5: Build Section Spans & Real-Time Monitoring (UC-15, UC-16) - `04-traceparent-propagation.md` in co-create-documentation/part-5/