feat: Auto instrument tracing when GITLAB_TRACING is set

  spec.add_runtime_dependency "grpc", ">= 1.75" # Be sure to update the "grpc-tools" dev_dependency too

  spec.add_runtime_dependency "google-protobuf", ">= 3.25", "< 5.0"

  spec.add_runtime_dependency "jaeger-client", "~> 1.1.0"

  spec.add_runtime_dependency 'json_schemer', '>= 2.3.0', '< 3.0'

  spec.add_runtime_dependency "json_schemer", ">= 2.3.0", "< 3.0"

  spec.add_runtime_dependency "openssl", "~> 3.3.2"

  spec.add_runtime_dependency "opentelemetry-sdk", "~> 1.10"

  spec.add_runtime_dependency "opentelemetry-instrumentation-all", "~> 0.89.1"

  spec.add_development_dependency "pry", "~> 0.12"

  spec.add_development_dependency "pry-byebug", "~> 3.11"

  spec.add_development_dependency "rack", "~> 2.0"

  spec.add_development_dependency "railties", ">= 5.0.0", "< 8.1.0"

  spec.add_development_dependency "rake", "~> 13.2"

  spec.add_development_dependency "rest-client", "~> 2.1.0"

  spec.add_development_dependency "rspec", "~> 3.12.0"

  autoload :HTTPClientPublisher, "labkit/httpclient_publisher"

end

Labkit::Tracing::AutoInitialize.initialize! if defined?(Labkit::Tracing)

# rubocop:enable Naming/FileName

  # Tracing provides distributed tracing functionality

  module Tracing

    autoload :AbstractInstrumenter, "labkit/tracing/abstract_instrumenter"

    autoload :AutoInitialize, "labkit/tracing/auto_initialize"

    autoload :TracingCommon, "labkit/tracing/tracing_common"

    autoload :Factory, "labkit/tracing/factory"

    autoload :GRPC, "labkit/tracing/grpc"

    end

  end

end

require "labkit/tracing/railtie" if defined?(Rails::Railtie)

Example:

```bash

# OpenTelemetry with HTTP endpoint

export GITLAB_TRACING="otlp://localhost:4318"

# OpenTelemetry with HTTP endpoint and service name

export GITLAB_TRACING="otlp://localhost:4318?service_name=my-api"

# Console exporter for development/testing (outputs to stdout)

export GITLAB_TRACING="otlp://console"

export GITLAB_TRACING="otlp://console?service_name=my-api"

# Without service name (defaults to "labkit-service")

export GITLAB_TRACING="otlp://localhost:4318"

```

**Important: Initialization Required**

**Automatic Initialization**

Setting `GITLAB_TRACING` alone is not sufficient to collect traces. You must explicitly call `Labkit::Tracing::Factory.create_tracer` in your application's initialization code (see [Creating a Tracer](#creating-a-tracer) section below).

When `GITLAB_TRACING` is set, LabKit automatically creates and configures a tracer when the gem is loaded. No manual initialization is required in most cases.

Without calling `Factory.create_tracer`:

- Your application runs safely without errors

- Instrumentation middleware creates span objects

- **No trace data is exported** to your backend (OTLP collector)

- A no-op tracer is used internally

Auto-initialization provides:

- Automatic tracer creation with connection string settings

- Service name from `service_name` query parameter (defaults to `"labkit-service"`)

- All available OpenTelemetry instrumentation enabled by default

- Safe fallback if initialization fails (no-op tracer)

### Optional Configuration

  - For `probabilistic`: rate between 0.0 and 1.0 (e.g., `0.1` = 10%)

  - For `const`: `1` (sample all) or `0` (sample none)

- **`service_name`** - Override the service name for this tracer

- **`service_name`** - Set the service name for this tracer (defaults to `"labkit-service"`)

  - The service name identifies your application in traces

  - Appears in trace UI and helps distinguish between services

  - Example: `service_name=checkout-api`

### Examples

```bash

# Development: Console output with full sampling

export GITLAB_TRACING="otlp://console?sampler=const&sampler_param=1"

# Development: Console output with full sampling and service name

export GITLAB_TRACING="otlp://console?service_name=my-app&sampler=const&sampler_param=1"

# Staging: HTTP endpoint with probabilistic sampling (1%)

export GITLAB_TRACING="otlp://collector.staging.example.com:4318?sampler=probabilistic&sampler_param=0.01"

export GITLAB_TRACING="otlp://collector.staging.example.com:4318?service_name=my-app&sampler=probabilistic&sampler_param=0.01"

# Production: HTTP endpoint with custom path and authentication

export GITLAB_TRACING="otlp://user:pass@collector.prod.example.com:4318/v1/traces?sampler=probabilistic&sampler_param=0.001"

export GITLAB_TRACING="otlp://user:pass@collector.prod.example.com:4318/v1/traces?service_name=my-app&sampler=probabilistic&sampler_param=0.001"

# Testing: Console output with no sampling (useful for test verification)

export GITLAB_TRACING="otlp://console?sampler=const&sampler_param=0"

export GITLAB_TRACING="otlp://console?service_name=my-app&sampler=const&sampler_param=0"

```

## Usage

end

```

### Creating a Tracer

### Automatic Initialization (Default Behavior)

When `GITLAB_TRACING` is set, LabKit automatically creates and configures a tracer when the gem loads:

```bash

# Set environment variable with service name

export GITLAB_TRACING="otlp://localhost:4318?service_name=my-api&sampler=probabilistic&sampler_param=0.01"

# Or use default service name "labkit-service"

export GITLAB_TRACING="otlp://localhost:4318"

```

Auto-initialization:

- Creates tracer with connection string settings (sampler, exporter, endpoints)

- Uses service name from `service_name` query parameter (defaults to `"labkit-service"`)

- Enables all available OpenTelemetry instrumentation

- Sets up the global `OpenTelemetry.tracer_provider`

**No application code changes required** - just set the environment variable.

### Automatic Rails Middleware Insertion

Setting the `GITLAB_TRACING` environment variable enables tracing functionality, but **you must explicitly initialize the tracer** to collect and export traces:

When using Rails, the `Labkit::Tracing::RackMiddleware` is automatically inserted into your middleware stack when `GITLAB_TRACING` is set. No manual configuration needed!

#### Basic Initialization

**How it works:**

- Detected automatically via Rails Railtie

- Inserted after `Rails::Rack::Logger` (or at available position if Rails::Rack::Logger not found)

- Only activates when `GITLAB_TRACING` is set

- Skips insertion if middleware is already manually configured

- Logs insertion to Rails logger for visibility

The simplest initialization uses GITLAB_TRACING configuration only:

**Middleware Positioning:**

The middleware is automatically inserted, but you can still customize its position if needed:

```ruby

# In your application initializer (e.g., config/initializers/tracing.rb)

if Labkit::Tracing.enabled?

  # REQUIRED: Create the tracer to actually collect traces

  Labkit::Tracing::Factory.create_tracer("my-service", ENV["GITLAB_TRACING"])

# config/application.rb

module MyApp

  class Application < Rails::Application

    # Move tracing middleware before authentication for earlier tracing

    config.middleware.move_before AuthenticationMiddleware, Labkit::Tracing::RackMiddleware

  # Factory.create_tracer configures the tracer globally by setting

  # OpenTelemetry.tracer_provider. No additional setup needed

    # Or move it after error handling

    config.middleware.move_after ErrorHandlingMiddleware, Labkit::Tracing::RackMiddleware

  end

end

```

#### With Configuration Block (Recommended)

**Note:** The middleware is automatically added, so you only need to use `move_before` or `move_after` if you need custom positioning. Don't use `insert_after` or `use` as that would duplicate the middleware.

You can customize the OpenTelemetry SDK while preserving GITLAB_TRACING settings:

### Manual Initialization (Override Auto-initialization)

You can override auto-initialization by calling `Factory.create_tracer` in your application initializers. This is useful when you need custom configuration:

```ruby

# In your application initializer (e.g., config/initializers/tracing.rb)

# config/initializers/tracing.rb

if Labkit::Tracing.enabled?

  Labkit::Tracing::Factory.create_tracer("my-service", ENV["GITLAB_TRACING"]) do |c|

    # Enable automatic instrumentation

    c.use_all() # Recommended - enables all available instrumentation

    # Or selective instrumentation

  # Override auto-initialization with custom configuration

  Labkit::Tracing::Factory.create_tracer("my-custom-service", ENV["GITLAB_TRACING"]) do |c|

    # Selective instrumentation instead of c.use_all()

    c.use 'OpenTelemetry::Instrumentation::Rails'

    c.use 'OpenTelemetry::Instrumentation::Sidekiq'

end

```

**Benefits:**

- Single initialization point for all tracing configuration

- GITLAB_TRACING settings (sampler, exporter, endpoints) are preserved

- Full access to OpenTelemetry SDK features (instrumentation, span processors, resources)

- Configuration block is optional

**When to use manual initialization:**

- You need selective instrumentation (not `c.use_all()`)

- You need custom span processors

- You need to add resource attributes

- You want to override the service name from code instead of the connection string

**What happens without initialization:**

Falls back to a no-op tracer (the default `ProxyTracerProvider` is detected and skipped). Spans are created but not exported, allowing your application to run safely while producing no trace data.

**Note:** Manual calls to `Factory.create_tracer` reconfigure the global OpenTelemetry tracer provider. The last call wins, so manual initialization overrides auto-initialization.

### Manual Span Creation

end

```

### Configuration Precedence

### Initialization Order and Precedence

When using `Factory.create_tracer` with a configuration block:

**Auto-initialization:**

1. Runs when the gem is loaded (`require 'gitlab-labkit'`)

2. Only runs if `GITLAB_TRACING` environment variable is set

3. Uses `service_name` query parameter or defaults to `"labkit-service"`

4. Attempts to enable all instrumentation with `c.use_all()`

**Manual initialization:**

1. Runs in your application initializer (e.g., `config/initializers/tracing.rb`)

2. Overrides auto-initialization by reconfiguring the global tracer provider

3. Last call to `Factory.create_tracer` wins

**Configuration precedence within a single `Factory.create_tracer` call:**

1. **GITLAB_TRACING connection string** settings are applied first:

   - Service name (from parameter or `service_name` query parameter)

   - Service name (from `service_name` query parameter or method parameter)

   - Sampler type and parameters (`sampler`, `sampler_param`)

   - Exporter endpoint, protocol, and authentication headers

   - Span processors with OTLP exporter

   - Merge additional resource attributes

   - Override service_name if explicitly set in the block

**Important:** Calling `Factory.create_tracer` multiple times will reconfigure the global OpenTelemetry tracer provider each time. The last call wins. This is generally safe but **should be avoided** - initialize tracing once during application startup.

**Important:** Calling `Factory.create_tracer` multiple times will reconfigure the global OpenTelemetry tracer provider each time. Initialize tracing once during application startup.

### Getting Trace URLs

Instrument incoming HTTP requests in Rack/Rails applications:

```ruby

# In config.ru or config/application.rb

# For non-Rails Rack apps (Sinatra, Grape, etc.) - in config.ru

use Labkit::Tracing::RackMiddleware

# For Rails apps: Middleware is automatically inserted when GITLAB_TRACING is set.

# No manual configuration needed! See "Automatic Rails Middleware Insertion" section.

```

This automatically:

## Example: Complete Setup

### Development Environment with Console Exporter

### Development Environment with Console Exporter (Automatic)

```ruby

# config/initializers/tracing.rb

if Labkit::Tracing.enabled?

  Labkit::Tracing::Factory.create_tracer("my-rails-app", ENV["GITLAB_TRACING"])

end

# config/application.rb

module MyApp

  class Application < Rails::Application

    # Add Rack middleware for HTTP request tracing

    config.middleware.insert_after Rails::Rack::Logger, Labkit::Tracing::RackMiddleware

    # Labkit::Tracing::RackMiddleware is automatically inserted!

    # No manual configuration needed.

    # Optional: Customize middleware position if needed

    # config.middleware.move_after SomeOtherMiddleware, Labkit::Tracing::RackMiddleware

  end

end

# .env.development

# GITLAB_TRACING="otlp://console?sampler=const&sampler_param=1"

# GITLAB_TRACING="otlp://console?service_name=my-rails-app&sampler=const&sampler_param=1"

```

This setup outputs all trace spans directly to your development console/logs, making it easy to:

**That's it!** Both the tracer and middleware are automatically configured when the gem loads. This setup outputs all trace spans directly to your development console/logs, making it easy to:

- Debug request flows without external tools

- Verify instrumentation is working correctly

- Test tracing configuration changes

- Develop and debug trace-dependent features

### Production Environment with Automatic Instrumentation

### Production Environment (Automatic with Custom Metadata)

Most applications can use automatic initialization:

```ruby

# config/application.rb

module MyApp

  class Application < Rails::Application

    # Labkit::Tracing::RackMiddleware is automatically inserted!

    # No manual configuration needed.

  end

end

# .env.production

# GITLAB_TRACING="otlp://collector.example.com:4318?service_name=my-rails-app&sampler=probabilistic&sampler_param=0.01"

```

If you need to add custom metadata (deployment environment, version, etc.), override tracer initialization:

```ruby

# config/initializers/tracing.rb

  end

end

# config/application.rb

module MyApp

  class Application < Rails::Application

    # Add Rack middleware for HTTP request tracing

    config.middleware.insert_after Rails::Rack::Logger, Labkit::Tracing::RackMiddleware

  end

end

# .env.production

# GITLAB_TRACING="otlp://collector.example.com:4318?sampler=probabilistic&sampler_param=0.01"

# GITLAB_TRACING="otlp://collector.example.com:4318?service_name=my-rails-app&sampler=probabilistic&sampler_param=0.01"

```

### With Manual Instrumentation Selection

Note: Middleware is still automatically inserted even with custom tracer initialization.

### With Selective Instrumentation (Manual Override)

If you want to control exactly which components are instrumented instead of using `c.use_all()`, override tracer initialization:

```ruby

# config/initializers/tracing.rb

if Labkit::Tracing.enabled?

  # REQUIRED: Initialize the tracer to collect and export traces

  # Without this, instrumentation runs but traces aren't sent anywhere

  Labkit::Tracing::Factory.create_tracer("my-rails-app", ENV["GITLAB_TRACING"])

  # Override auto-initialization for selective instrumentation

  Labkit::Tracing::Factory.create_tracer("my-rails-app", ENV["GITLAB_TRACING"]) do |c|

    # Selective OpenTelemetry instrumentation (instead of c.use_all())

    c.use 'OpenTelemetry::Instrumentation::Rails'

    c.use 'OpenTelemetry::Instrumentation::Sidekiq'

  end

  # Instrument Rails components

  # Additional LabKit-specific instrumentation

  Rails.application.config.after_initialize do

    Labkit::Tracing::Rails::ActiveRecord::Subscriber.instrument

    Labkit::Tracing::Rails::ActionView::Subscriber.instrument

    Labkit::Tracing::Redis.instrument

  end

end

# config/application.rb

module MyApp

  class Application < Rails::Application

    # Add Rack middleware for HTTP request tracing

    config.middleware.insert_after Rails::Rack::Logger, Labkit::Tracing::RackMiddleware

  end

end

```

Note: Middleware is still automatically inserted - no need to configure it manually.

## Troubleshooting

### Tracing Not Working

**First, check if you initialized the tracer** - see [Middleware Works But No Traces Appear](#middleware-works-but-no-traces-appear) below.

If you've confirmed initialization is correct:

1. Verify `GITLAB_TRACING` is set correctly:

   ```ruby

   puts Labkit::Tracing.connection_string

   puts Labkit::Tracing.enabled?

   ```

2. Check for tracer creation errors in logs (warnings are emitted on failure)

2. Check for tracer creation errors in logs (warnings are emitted on failure during auto-initialization)

3. Verify the OTLP collector is reachable

### Middleware Not Tracing Requests (Rails)

If you're not seeing HTTP request traces in Rails:

1. Verify the middleware is in your Rails stack:

   ```ruby

   # In Rails console

   Rails.application.middleware.middlewares

   # Should include Labkit::Tracing::RackMiddleware

   ```

2. Check Rails logs for auto-insertion message:

   ```

   Labkit::Tracing: Automatically inserted RackMiddleware after Rails::Rack::Logger

   ```

3. If using custom middleware positioning, verify the order is correct:

   ```bash

   # View middleware stack with positions

   bundle exec rake middleware

   ```

4. Ensure `GITLAB_TRACING` is set in your environment (not just in `.env` files that might not be loaded)

### Missing Spans

1. Ensure instrumentation is called after dependencies are loaded

If your application runs without errors but you don't see traces in your backend (OTLP collector):

**1. Verify you called `Factory.create_tracer` in your initializer:**

This step is **required** - setting `GITLAB_TRACING` alone is not enough:

**1. Verify the tracer provider is initialized:**

```ruby

# config/initializers/tracing.rb

if Labkit::Tracing.enabled?

  Labkit::Tracing::Factory.create_tracer("my-service", ENV["GITLAB_TRACING"])

end

OpenTelemetry.tracer_provider.class

# Expected: OpenTelemetry::SDK::Trace::TracerProvider

# Problem:  OpenTelemetry::Internal::ProxyTracerProvider (auto-initialization failed)

```

**2. Verify the tracer provider is initialized:**

**2. Check for auto-initialization warnings in logs:**

```ruby

OpenTelemetry.tracer_provider.class

# Expected: OpenTelemetry::SDK::Trace::TracerProvider

# Problem:  OpenTelemetry::Internal::ProxyTracerProvider (means Factory.create_tracer wasn't called)

Auto-initialization warnings indicate what went wrong:

```

Labkit::Tracing auto-initialization failed: <error message>

```

**Why this happens:**

**3. Common causes:**

- `GITLAB_TRACING` environment variable not set or not visible to the process

- Connection string format error (should start with `otlp://`)

- OpenTelemetry gem dependency issues

- OTLP collector not reachable

**4. Test with console exporter:**

Use the console exporter to verify tracing works locally:

```bash

export GITLAB_TRACING="otlp://console?service_name=test&sampler=const&sampler_param=1"

```

Without calling `Factory.create_tracer`, LabKit uses a no-op tracer that creates span objects for instrumentation but doesn't export them to any backend. This design prevents crashes when tracing is misconfigured, but means no trace data is collected.

You should see trace output in your application logs.

### High Overhead

# frozen_string_literal: true

require "cgi"

module Labkit

  module Tracing

    module AutoInitialize

      DEFAULT_SERVICE_NAME = "labkit-service"

      def self.detect_service_name(connection_string)

        return DEFAULT_SERVICE_NAME unless connection_string

        if connection_string =~ /[?&]service_name=([^&]+)/

          CGI.unescape(Regexp.last_match(1))

        else

          DEFAULT_SERVICE_NAME

        end

      end

      def self.initialize!

        connection_string = ENV.fetch("GITLAB_TRACING", nil)

        return if connection_string.nil? || connection_string.empty?

        service_name = detect_service_name(connection_string)

        Factory.create_tracer(service_name, connection_string) do |c|

          require "opentelemetry/instrumentation/all"

          c.use_all

        end

      rescue StandardError => e

        warn "Labkit::Tracing auto-initialization failed: #{e.message}"

      end

    end

  end

end