feat: Open Telemetry protocol support for Labkit::Tracing
This MR adds OpenTelemetry Protocol (OTLP) support to Labkit::Tracing while maintaining backward compatibility with the existing OpenTracing/Jaeger implementation. The library now supports both tracing protocols through a unified API, automatically detecting which protocol to use based on the GITLAB_TRACING connection string.
What?
This implementation adds first-class OpenTelemetry support to labKit-ruby, allowing GitLab services to migrate from the deprecated OpenTracing standard to the modern OpenTelemetry standard without code changes.
Key Features:
- Dual Protocol Support: Both OpenTracing (opentracing://) and OpenTelemetry (otlp://) connection strings
- Unified API: All existing tracing APIs work identically with both protocols via adapter pattern
- OpenTelemetry SDK Configurability: Optional configuration block for customizing instrumentation, span processors, and resources
- Console Exporter: Development-friendly console exporter for local debugging without external collectors
- Auto-instrumentation when
GITLAB_TRACINGenvironment variable is set
Changes
- Extended Factory to support OTLP
- Built protocol adapter layer
- Refactored OpenTracing-specific code
- Abstracted connection string recognition
- Made TracingUtils protocol-agnostic
- Added OTel SDK configuration block
- Created comprehensive tracing README - documented usage, migration guide, configuration options, and troubleshooting
- Instantiating and assigning global tracer automatically for consistency
-
Implemented console exporter for OTel - added
otlp://consoleendpoint for development/testing without external collectors -
Exposing concrete
tracerimplementation to provide more options for power users
Architecture Changes
Before: Direct OpenTracing API usage throughout the codebase
After: Protocol-agnostic adapter layer with automatic protocol detection
┌─────────────────────────────────────┐
│ Public API (Labkit::Tracing) │
│ with_tracing, sampled?, etc. │
└──────────────┬──────────────────────┘
│
┌──────────────▼──────────────────────┐
│ TracingUtils (adapter routing) │
└──────────────┬──────────────────────┘
│
┌───────┴───────┐
▼ ▼
┌──────────────┐ ┌──────────────────┐
│ OpenTracing │ │ OpenTelemetry │
│ Adapter │ │ Adapter │
└──────────────┘ └──────────────────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────────┐
│ Jaeger │ │ OTLP Exporters │
│ Client │ │ (HTTP/gRPC/ │
│ │ │ Console) │
└──────────────┘ └──────────────────┘
Links
Resolves gitlab-com/gl-infra/observability/team#4430 (closed).
The changes implemented here can be seen in action in this example MR https://gitlab.com/gitlab-com/gl-infra/observability/tracing-app-ruby/-/merge_requests/1.
I did a quick update pointing the GDK to this version of labkit, and it worked fine.
Click to expand the collector output
2026-02-09T17:37:45.317Z info service@v0.144.0/service.go:239 Starting otelcol... {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "Version": "0.144.0", "NumCPU": 4}
2026-02-09T17:37:45.317Z info extensions/extensions.go:40 Starting extensions... {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}}
2026-02-09T17:37:45.317Z info otlpreceiver@v0.144.0/otlp.go:120 Starting GRPC server {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "[::]:4317"}
2026-02-09T17:37:45.317Z info otlpreceiver@v0.144.0/otlp.go:178 Starting HTTP server {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "[::]:4318"}
2026-02-09T17:37:45.317Z info service@v0.144.0/service.go:262 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}}
2026-02-09T17:38:23.253Z info Traces {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 4}
2026-02-09T17:38:23.253Z info ResourceSpans #0
Resource SchemaURL:
Resource attributes:
-> service.name: Str(labkit-service)
-> process.pid: Int(902)
-> process.command: Str(bin/rails)
-> process.runtime.name: Str(ruby)
-> process.runtime.version: Str(3.3.10)
-> process.runtime.description: Str(ruby 3.3.10 (2025-10-23 revision 343ea05002) [arm64-darwin25])
-> telemetry.sdk.name: Str(opentelemetry)
-> telemetry.sdk.language: Str(ruby)
-> telemetry.sdk.version: Str(1.10.0)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope OpenTelemetry::Instrumentation::PG 0.34.1
Span #0
Trace ID : 2b1220658269e44dd468b64465693584
Parent ID :
ID : 69c2b1ded5f78f75
Name : gitlabhq_development
Kind : Client
Start time : 2026-02-09 17:38:19.221473 +0000 UTC
End time : 2026-02-09 17:38:19.221528 +0000 UTC
Status code : Unset
Status message :
DroppedAttributesCount: 0
DroppedEventsCount: 0
DroppedLinksCount: 0
Attributes:
-> db.system: Str(postgresql)
-> db.user: Str(hercules)
-> db.name: Str(gitlabhq_development)
-> net.sock.family: Str(unix)
-> net.peer.name: Str(/Users/hercules/GitLab/gdk/postgresql)
-> db.statement: Str(;)
Span #1
Trace ID : 60e27f8c8adaa4e8565c0b5f4644c581
Parent ID :
ID : 182595a99d70b96b
Name : connect
Kind : Client
Start time : 2026-02-09 17:38:20.011638 +0000 UTC
End time : 2026-02-09 17:38:20.012997 +0000 UTC
Status code : Unset
Status message :
DroppedAttributesCount: 0
DroppedEventsCount: 0
DroppedLinksCount: 0
Attributes:
-> db.system: Str(postgresql)
-> db.name: Str(gitlabhq_development)
-> db.user: Str(hercules)
-> net.sock.family: Str(unix)
-> net.peer.name: Str(/Users/hercules/GitLab/gdk/postgresql)
Span #2
Trace ID : f59d4b222c376b475e3c580f881235fa
Parent ID :
ID : a833d48a20bd4cbf
Name : gitlabhq_development
Kind : Client
Start time : 2026-02-09 17:38:20.314299 +0000 UTC
End time : 2026-02-09 17:38:20.314349 +0000 UTC
Status code : Unset
Status message :
DroppedAttributesCount: 0
DroppedEventsCount: 0
DroppedLinksCount: 0
Attributes:
-> db.system: Str(postgresql)
-> db.user: Str(hercules)
-> db.name: Str(gitlabhq_development)
-> net.sock.family: Str(unix)
-> net.peer.name: Str(/Users/hercules/GitLab/gdk/postgresql)
-> db.statement: Str(;)
ScopeSpans #1
ScopeSpans SchemaURL:
InstrumentationScope OpenTelemetry::Instrumentation::ActiveRecord 0.11.1
Span #0
Trace ID : 8bc96d9aa922eb7e20f82411916a658b
Parent ID :
ID : 48513446362da09b
Name : ApplicationSetting query
Kind : Internal
Start time : 2026-02-09 17:38:19.417454 +0000 UTC
End time : 2026-02-09 17:38:19.421399 +0000 UTC
Status code : Unset
Status message :
DroppedAttributesCount: 0
DroppedEventsCount: 0
DroppedLinksCount: 0
{"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces"}
2026-02-09T17:38:28.318Z info Traces {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}
2026-02-09T17:38:28.318Z info ResourceSpans #0
Resource SchemaURL:
Resource attributes:
-> service.name: Str(labkit-service)
-> process.pid: Int(902)
-> process.command: Str(bin/rails)
-> process.runtime.name: Str(ruby)
-> process.runtime.version: Str(3.3.10)
-> process.runtime.description: Str(ruby 3.3.10 (2025-10-23 revision 343ea05002) [arm64-darwin25])
-> telemetry.sdk.name: Str(opentelemetry)
-> telemetry.sdk.language: Str(ruby)
-> telemetry.sdk.version: Str(1.10.0)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope OpenTelemetry::Instrumentation::PG 0.34.1
Span #0
Trace ID : 92589843ac4fa8591585574778fe224e
Parent ID : e90cf7e464c980eb
ID : 696189537cbd2676
Name : connect
Kind : Client
Start time : 2026-02-09 17:38:24.041462 +0000 UTC
End time : 2026-02-09 17:38:24.043073 +0000 UTC
Status code : Unset
Status message :
DroppedAttributesCount: 0
DroppedEventsCount: 0
DroppedLinksCount: 0
Attributes:
-> db.system: Str(postgresql)
-> db.name: Str(gitlabhq_development)
-> db.user: Str(hercules)
-> net.sock.family: Str(unix)
-> net.peer.name: Str(/Users/hercules/GitLab/gdk/postgresql)
{"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces"}
2026-02-09T17:38:33.998Z info Traces {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 1}
2026-02-09T17:38:33.998Z info ResourceSpans #0
Resource SchemaURL:
Resource attributes:
-> service.name: Str(labkit-service)
-> process.pid: Int(902)
-> process.command: Str(bin/rails)
-> process.runtime.name: Str(ruby)
-> process.runtime.version: Str(3.3.10)
-> process.runtime.description: Str(ruby 3.3.10 (2025-10-23 revision 343ea05002) [arm64-darwin25])
-> telemetry.sdk.name: Str(opentelemetry)
-> telemetry.sdk.language: Str(ruby)
-> telemetry.sdk.version: Str(1.10.0)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope OpenTelemetry::Instrumentation::Net::HTTP 0.26.1
Span #0
Trace ID : 15bc235238334881ea6bc9ce8fc25563
Parent ID :
ID : 859e475562984c9c
Name : connect
Kind : Internal
Start time : 2026-02-09 17:38:28.901583 +0000 UTC
End time : 2026-02-09 17:38:28.903431 +0000 UTC
Status code : Error
Status message : Unhandled exception of type: Errno::ECONNREFUSED
DroppedAttributesCount: 0
DroppedEventsCount: 0
DroppedLinksCount: 0
Attributes:
-> net.peer.name: Str(gdk.test)
-> net.peer.port: Int(9091)
Events:
SpanEvent #0
-> Name: exception
-> Timestamp: 2026-02-09 17:38:28.903416 +0000 UTC
-> DroppedAttributesCount: 0
-> Attributes::
-> exception.type: Str(Errno::ECONNREFUSED)
-> exception.message: Str(Failed to open TCP connection to gdk.test:9091 (Connection refused - connect(2) for "gdk.test" port 9091))
-> exception.stacktrace: Str(/Users/hercules/GitLab/gdk/gitlab/gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `initialize': Failed to open TCP connection to gdk.test:9091 (Connection refused - connect(2) for "gdk.test" port 9091) (Errno::ECONNREFUSED)
from /Users/hercules/GitLab/gdk/gitlab/gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `open'
from /Users/hercules/GitLab/gdk/gitlab/gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `block in connect'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/timeout-0.4.3/lib/timeout.rb:185:in `block in timeout'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/timeout-0.4.3/lib/timeout.rb:192:in `timeout'
from /Users/hercules/GitLab/gdk/gitlab/gems/gitlab-http/lib/net_http/connect_patch.rb:50:in `connect'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-instrumentation-net_http-0.26.1/lib/opentelemetry/instrumentation/net/http/patches/old/instrumentation.rb:76:in `block in connect'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/trace/tracer.rb:37:in `block in in_span'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/trace.rb:70:in `block in with_span'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/context.rb:88:in `with_value'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/trace.rb:70:in `with_span'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/trace/tracer.rb:37:in `in_span'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-instrumentation-net_http-0.26.1/lib/opentelemetry/instrumentation/net/http/patches/old/instrumentation.rb:75:in `connect'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:1642:in `do_start'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:1631:in `start'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:2375:in `request'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-instrumentation-net_http-0.26.1/lib/opentelemetry/instrumentation/net/http/patches/old/instrumentation.rb:22:in `request'
from /Users/hercules/GitLab/labkit-ruby/lib/labkit/net_http_publisher.rb:40:in `request'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:326:in `http_post'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:247:in `send_requests_with_post'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:234:in `send_requests'
from /Users/hercules/GitLab/gdk/gitlab/lib/gitlab/tracking/snowplow_logging_emitter.rb:14:in `send_requests'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:427:in `block in consume'
from <internal:kernel>:187:in `loop'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:425:in `consume'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:417:in `block (2 levels) in initialize'
/Users/hercules/GitLab/gdk/gitlab/gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `initialize': Connection refused - connect(2) for "gdk.test" port 9091 (Errno::ECONNREFUSED)
from /Users/hercules/GitLab/gdk/gitlab/gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `open'
from /Users/hercules/GitLab/gdk/gitlab/gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `block in connect'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/timeout-0.4.3/lib/timeout.rb:185:in `block in timeout'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/timeout-0.4.3/lib/timeout.rb:192:in `timeout'
from /Users/hercules/GitLab/gdk/gitlab/gems/gitlab-http/lib/net_http/connect_patch.rb:50:in `connect'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-instrumentation-net_http-0.26.1/lib/opentelemetry/instrumentation/net/http/patches/old/instrumentation.rb:76:in `block in connect'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/trace/tracer.rb:37:in `block in in_span'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/trace.rb:70:in `block in with_span'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/context.rb:88:in `with_value'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/trace.rb:70:in `with_span'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-api-1.7.0/lib/opentelemetry/trace/tracer.rb:37:in `in_span'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-instrumentation-net_http-0.26.1/lib/opentelemetry/instrumentation/net/http/patches/old/instrumentation.rb:75:in `connect'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:1642:in `do_start'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:1631:in `start'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/net-http-0.6.0/lib/net/http.rb:2375:in `request'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/opentelemetry-instrumentation-net_http-0.26.1/lib/opentelemetry/instrumentation/net/http/patches/old/instrumentation.rb:22:in `request'
from /Users/hercules/GitLab/labkit-ruby/lib/labkit/net_http_publisher.rb:40:in `request'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:326:in `http_post'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:247:in `send_requests_with_post'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:234:in `send_requests'
from /Users/hercules/GitLab/gdk/gitlab/lib/gitlab/tracking/snowplow_logging_emitter.rb:14:in `send_requests'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:427:in `block in consume'
from <internal:kernel>:187:in `loop'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:425:in `consume'
from /Users/hercules/.local/share/mise/installs/ruby/3.3.10/lib/ruby/gems/3.3.0/gems/snowplow-tracker-0.8.0/lib/snowplow-tracker/emitters.rb:417:in `block (2 levels) in initialize'
)
{"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces"}
^C2026-02-09T17:39:09.295Z info otelcol@v0.144.0/collector.go:376 Received signal from OS {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}, "signal": "interrupt"}
2026-02-09T17:39:09.296Z info service@v0.144.0/service.go:276 Starting shutdown... {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}}
2026-02-09T17:39:09.296Z info extensions/extensions.go:68 Stopping extensions... {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}}
2026-02-09T17:39:09.296Z info service@v0.144.0/service.go:290 Shutdown complete. {"resource": {"service.instance.id": "dc76c52a-0502-49c6-b8c9-9262f3f6700f", "service.name": "otelcol", "service.version": "0.144.0"}}