Verified Commit c923b3e1 authored by Hercules Merscher's avatar Hercules Merscher 🌴
Browse files

docs: Tracing README

parent b5934bb8
Loading
Loading
Loading
Loading
+662 −0
Original line number Diff line number Diff line
# Labkit::Tracing

The `Labkit::Tracing` module provides distributed tracing functionality for Ruby applications using the OpenTracing or OpenTelemetry standards. It enables you to trace requests across multiple services and components, helping you understand application performance and debug issues in distributed systems.

## Overview

Distributed tracing allows you to track requests as they flow through your application and external services. The tracing module integrates with OpenTelemetry (OTLP) and Jaeger (OpenTracing) backends, and provides automatic instrumentation for:

- HTTP requests (Rack/Rails)
- gRPC calls (client and server)
- Redis operations
- External HTTP requests
- Rails components (ActiveRecord, ActionView, ActiveSupport)

## Protocol Support

Labkit-Ruby supports both OpenTracing and OpenTelemetry protocols. The library automatically detects which protocol to use based on the `GITLAB_TRACING` connection string and provides a unified API that works seamlessly with both.

**Note:** OpenTracing is [archived and deprecated](https://opentracing.io/). OpenTelemetry is the recommended tracing protocol for new projects. Labkit-Ruby maintains OpenTracing support for backward compatibility.

### Protocol Selection

- **OpenTelemetry** (Recommended): Use `otlp://` prefix in connection string
- **OpenTracing** (Deprecated): Use `opentracing://` prefix in connection string

All public APIs (`Labkit::Tracing.with_tracing`, `Labkit::Tracing.sampled?`, etc.) work identically regardless of the protocol used.

## Configuration

Tracing is controlled via environment variables:

### Required Configuration

**`GITLAB_TRACING`** - Connection string for the tracing backend

Format: `opentracing://<driver>?<options>` or `otlp://<host:port>?<options>`

Example:
```bash
# OpenTelemetry with HTTP endpoint (Recommended)
export GITLAB_TRACING="otlp://localhost:4318"

# Jaeger with UDP endpoint (OpenTracing - Deprecated)
export GITLAB_TRACING="opentracing://jaeger?udp_endpoint=localhost:6831"

# Jaeger with HTTP endpoint (OpenTracing - Deprecated)
export GITLAB_TRACING="opentracing://jaeger?http_endpoint=https://jaeger.example.com:14268/api/traces"

# Jaeger with sampling configuration (OpenTracing - Deprecated)
export GITLAB_TRACING="opentracing://jaeger?udp_endpoint=localhost:6831&sampler=probabilistic&sampler_param=0.1"
```

**Important: Initialization Required**

Setting `GITLAB_TRACING` alone is not sufficient to collect traces. You must explicitly call `Labkit::Tracing::Factory.create_tracer` in your application's initialization code (see [Creating a Tracer](#creating-a-tracer) section below).

Without calling `Factory.create_tracer`:
- Your application runs safely without errors
- Instrumentation middleware creates span objects
- **No trace data is exported** to your backend (OTLP collector or Jaeger)
- A no-op tracer is used internally

This applies to both OpenTelemetry (`otlp://...`) and OpenTracing (`opentracing://jaeger...`) connection strings.

### Optional Configuration

**`GITLAB_TRACING_URL`** - Template URL for linking to trace views

Supports placeholders:
- `{{ correlation_id }}` - Current correlation ID
- `{{ service }}` - Service name

Example:
```bash
export GITLAB_TRACING_URL="https://jaeger.example.com/trace/{{ correlation_id }}?service={{ service }}"
```

**`GITLAB_TRACING_INCLUDE_STACKTRACE`** - Comma-separated list of operation name prefixes to include stack traces

Example:
```bash
export GITLAB_TRACING_INCLUDE_STACKTRACE="redis,active_record"
```

## OTLP Configuration Options

<!-- TODO: Add detailed OTLP configuration options similar to Jaeger section below.
     Include examples for:
     - HTTP vs gRPC endpoints
     - Sampling configuration
     - Custom paths and authentication
     - Protocol override parameter
-->

## Jaeger Configuration Options

When using Jaeger as the tracing backend, the following query parameters are supported:

### Endpoints

- **`udp_endpoint`** - Jaeger agent UDP endpoint (default port: 6831)
  - Example: `udp_endpoint=localhost:6831`
- **`http_endpoint`** - Jaeger collector HTTP endpoint
  - Example: `http_endpoint=https://jaeger.example.com:14268/api/traces`
  - Supports basic authentication: `http_endpoint=https://user:password@jaeger.example.com/api/traces`

### Sampling

- **`sampler`** - Sampling strategy (`probabilistic` or `const`)
  - `probabilistic` - Sample a percentage of traces (default: 0.1%)
  - `const` - Sample all traces (when `sampler_param=1`) or none (when `sampler_param=0`)

- **`sampler_param`** - Parameter for the sampler
  - For `probabilistic`: rate between 0.0 and 1.0 (e.g., `0.1` = 10%)
  - For `const`: `1` (sample all) or `0` (sample none)

- **`service_name`** - Override the service name for this tracer

Example:
```bash
export GITLAB_TRACING="opentracing://jaeger?udp_endpoint=localhost:6831&sampler=probabilistic&sampler_param=0.01"
```

## Usage

### Checking if Tracing is Enabled

```ruby
if Labkit::Tracing.enabled?
  # Tracing is configured
end
```

### Creating a Tracer

Setting the `GITLAB_TRACING` environment variable enables tracing functionality, but **you must explicitly initialize the tracer** to collect and export traces:

#### Basic Initialization

The simplest initialization uses GITLAB_TRACING configuration only:

```ruby
# In your application initializer (e.g., config/initializers/tracing.rb)
if Labkit::Tracing.enabled?
  # REQUIRED: Create the tracer to actually collect traces
  tracer = Labkit::Tracing::Factory.create_tracer("my-service", ENV["GITLAB_TRACING"])

  # For OpenTracing/Jaeger: Set as global tracer
  OpenTracing.global_tracer = tracer if tracer

  # For OpenTelemetry/OTLP: Factory.create_tracer configures it globally
  # The tracer provider is set automatically - no additional setup needed
end
```

#### With Configuration Block (Recommended for OpenTelemetry)

For OpenTelemetry (OTLP) connections, you can customize the SDK while preserving GITLAB_TRACING settings:

```ruby
# In your application initializer (e.g., config/initializers/tracing.rb)
if Labkit::Tracing.enabled?
  Labkit::Tracing::Factory.create_tracer("my-service", ENV["GITLAB_TRACING"]) do |c|
    # Enable automatic instrumentation
    c.use_all() # Recommended - enables all available instrumentation

    # Or selective instrumentation
    c.use 'OpenTelemetry::Instrumentation::Rails'
    c.use 'OpenTelemetry::Instrumentation::Sidekiq'

    # Add custom span processors
    c.add_span_processor(MyCustomProcessor.new)

    # Add additional resource attributes
    c.resource = c.resource.merge(
      OpenTelemetry::SDK::Resources::Resource.create(
        'deployment.environment' => Rails.env,
        'service.version' => MyApp::VERSION
      )
    )
  end
end
```

**Benefits:**
- Single initialization point for all tracing configuration
- GITLAB_TRACING settings (sampler, exporter, endpoints) are preserved
- Full access to OpenTelemetry SDK features (instrumentation, span processors, resources)
- Configuration block is optional

**Note:** Configuration blocks only work with OpenTelemetry connections (`otlp://`). They are ignored for OpenTracing connections (`opentracing://`) with a warning.

**What happens without initialization:**
- **OpenTelemetry (OTLP):** Falls back to a no-op tracer (the default `ProxyTracerProvider` is detected and skipped)
- **OpenTracing (Jaeger):** Uses the default no-op tracer

In both cases, spans are created but not exported, allowing your application to run safely while producing no trace data.

### Manual Span Creation

Use `Labkit::Tracing.with_tracing` to create custom spans:

```ruby
Labkit::Tracing.with_tracing(
  operation_name: "process_data",
  tags: { "user_id" => user.id, "data_size" => data.size }
) do |span|
  # Your code here
  result = process_data(data)
  # Add additional tags during execution
  span.set_tag("result_count", result.count)
  result
end
```

With parent span context:

```ruby
# Get the current active span (works with both OpenTracing and OpenTelemetry)
parent_span = Labkit::Tracing::TracingUtils.active_span

Labkit::Tracing.with_tracing(
  operation_name: "child_operation",
  tags: { "type" => "background" },
  child_of: parent_span
) do |span|
  # This span will be a child of parent_span
  perform_operation
end
```

### Configuration Precedence

When using `Factory.create_tracer` with a configuration block:

1. **GITLAB_TRACING connection string** settings are applied first:
   - Service name (from parameter or `service_name` query parameter)
   - Sampler type and parameters (`sampler`, `sampler_param`)
   - Exporter endpoint, protocol, and authentication headers
   - Span processors with OTLP exporter

2. **Configuration block** runs second and can:
   - Add automatic instrumentation (`use`, `use_all`)
   - Add additional span processors
   - Merge additional resource attributes
   - Override service_name if explicitly set in the block

**Important:** Calling `Factory.create_tracer` multiple times will reconfigure the global OpenTelemetry tracer provider each time. The last call wins. This is generally safe but **should be avoided** - initialize tracing once during application startup.

**Example showing precedence:**
```ruby
# GITLAB_TRACING="otlp://localhost:4318?sampler=probabilistic&sampler_param=0.01"

Labkit::Tracing::Factory.create_tracer("api-service", ENV["GITLAB_TRACING"]) do |c|
  # These settings are ADDED to GITLAB_TRACING configuration
  c.use 'OpenTelemetry::Instrumentation::Rails'

  # Resource attributes are MERGED
  c.resource = c.resource.merge(
    OpenTelemetry::SDK::Resources::Resource.create('env' => 'production')
  )

  # Sampler from GITLAB_TRACING (probabilistic 1%) is preserved
  # Exporter endpoint (localhost:4318) is preserved
  # Service name ('api-service') is preserved unless you override it
end
```

### Getting Trace URLs

Generate a URL to view the current trace in your tracing UI:

```ruby
if Labkit::Tracing.tracing_url_enabled?
  url = Labkit::Tracing.tracing_url("my-service")
  # Returns: https://jaeger.example.com/trace/abc123?service=my-service
end
```

### Checking if Current Request is Sampled

```ruby
if Labkit::Tracing.sampled?
  # Current request is being traced
  # Safe to add expensive tracing operations
end
```

## Instrumentation

### Rack Middleware

Instrument incoming HTTP requests in Rack/Rails applications:

```ruby
# In config.ru or config/application.rb
use Labkit::Tracing::RackMiddleware
```

This automatically:
- Extracts trace context from incoming requests
- Creates spans for HTTP requests with method and URL
- Adds HTTP status codes to spans
- Sanitizes sensitive parameters in URLs

### Rails Components

#### ActiveRecord (Database Queries)

```ruby
unsubscribe = Labkit::Tracing::Rails::ActiveRecord::Subscriber.instrument

# Later, to stop instrumentation:
unsubscribe.call
```

Traces:
- SQL queries with sanitized statements
- Query fingerprints
- Connection IDs
- Cached query indicators

#### ActionView (Template Rendering)

```ruby
unsubscribe = Labkit::Tracing::Rails::ActionView::Subscriber.instrument

# Later, to stop instrumentation:
unsubscribe.call
```

Traces:
- Template rendering
- Partial rendering
- Collection rendering
- Template identifiers and layouts

#### ActiveSupport (Caching)

```ruby
unsubscribe = Labkit::Tracing::Rails::ActiveSupport::Subscriber.instrument

# Later, to stop instrumentation:
unsubscribe.call
```

Traces:
- Cache reads (with hit/miss information)
- Cache writes
- Cache deletes
- Cache fetch operations
- Cache key information

### Redis

Instrument Redis operations:

```ruby
Labkit::Tracing::Redis.instrument
```

This automatically traces:
- Individual Redis commands
- Pipelined commands (up to 5 commands shown)
- Connection details (host, port, scheme)
- Sanitized command arguments (sensitive commands like AUTH and EVAL are masked)

### gRPC

#### Client-Side Instrumentation

```ruby
# Add to gRPC client configuration
interceptors = [Labkit::Tracing::GRPC::ClientInterceptor.instance]
stub = MyService::Stub.new(address, :this_channel_is_insecure, interceptors: interceptors)
```

Traces outgoing gRPC calls with:
- Method names
- gRPC call types (unary, client_stream, server_stream, bidi_stream)
- Automatic context propagation to downstream services

#### Server-Side Instrumentation

```ruby
# Add to gRPC server configuration
server = GRPC::RpcServer.new(interceptors: [Labkit::Tracing::GRPC::ServerInterceptor.new])
```

Traces incoming gRPC calls with:
- Method names
- gRPC call types
- Automatic context extraction from upstream services

### External HTTP Requests

Instrument outgoing HTTP requests made by Net::HTTP, Excon, and HTTPClient:

```ruby
unsubscribe = Labkit::Tracing::ExternalHttp.instrument

# Later, to stop instrumentation:
unsubscribe.call
```

Automatically traces:
- HTTP method and URL
- Response status codes
- Host, port, and scheme
- Proxy information (if applicable)

## Architecture

### Core Components

- **`Labkit::Tracing`** - Main module with configuration and utility methods
- **`Labkit::Tracing::Factory`** - Creates and configures tracer instances (supports both OpenTracing and OpenTelemetry)
- **`Labkit::Tracing::JaegerFactory`** - Jaeger-specific tracer configuration (OpenTracing)
- **`Labkit::Tracing::OpenTelemetryFactory`** - OpenTelemetry-specific tracer configuration
- **`Labkit::Tracing::TracingUtils`** - Protocol-agnostic utilities for span management (abstracts OpenTracing and OpenTelemetry APIs)
- **`Labkit::Tracing::AbstractInstrumenter`** - Base class for ActiveSupport::Notifications instrumenters

### Instrumentation Pattern

Most instrumenters follow this pattern:

1. Subscribe to `ActiveSupport::Notifications` events
2. Create spans when events start
3. Add tags and metadata to spans
4. Handle exceptions and log them to spans
5. Close spans when events finish

### Span Lifecycle

Each span automatically includes:
- **Correlation ID** - Links traces to logs and other telemetry
- **Common tags** - Component, operation type, etc.
- **Exception handling** - Errors are logged with stack traces
- **Stack traces** - Optional, based on `GITLAB_TRACING_INCLUDE_STACKTRACE`

## Security and Privacy

The tracing module includes built-in sanitization:

- **SQL queries** - Sanitized using `Labkit::Logging::Sanitizer.sanitize_sql`
- **URLs** - Filtered parameters removed (e.g., tokens, passwords)
- **Redis commands** - Sensitive commands (AUTH, EVAL) are masked
- **Redis arguments** - Long arguments are truncated with masking
- **Exception messages** - Sanitized using `Labkit::Logging::Sanitizer.sanitize_field`

## Performance Considerations

### Sampling

Use sampling to control overhead:

```bash
# Trace 1% of requests
export GITLAB_TRACING="opentracing://jaeger?udp_endpoint=localhost:6831&sampler=probabilistic&sampler_param=0.01"

# Trace all requests (high overhead)
export GITLAB_TRACING="opentracing://jaeger?udp_endpoint=localhost:6831&sampler=const&sampler_param=1"
```

### Conditional Expensive Operations

Check if the current request is being sampled before adding expensive tracing data:

```ruby
if Labkit::Tracing.sampled?
  # Only execute expensive operations when trace is being captured
  span.set_tag("expensive_data", compute_expensive_data)
end
```

### Flush Interval

The Jaeger reporter flushes spans every 5 seconds (configurable via `FLUSH_INTERVAL` constant) to prevent UDP packet overflow.

## Integration with Correlation

Tracing integrates with `Labkit::Correlation` to link traces with logs and other telemetry:

```ruby
# Correlation ID is automatically added to all spans
correlation_id = Labkit::Correlation::CorrelationId.current_id
# This ID appears in both logs and traces
```

## Migrating from OpenTracing to OpenTelemetry

To migrate from OpenTracing to OpenTelemetry:

1. **Update the connection string** - Change from `opentracing://` to `otlp://`:
   ```bash
   # Before
   export GITLAB_TRACING="opentracing://jaeger?udp_endpoint=localhost:6831"

   # After
   export GITLAB_TRACING="otlp://localhost:4318"
   ```

2. **No code changes required** - All Labkit tracing APIs (`Labkit::Tracing.with_tracing`, `Labkit::Tracing.sampled?`, etc.) work identically with both protocols.

3. **Update backend configuration** - Ensure your tracing backend (Jaeger, etc.) supports OpenTelemetry OTLP protocol.

That's it! The migration is transparent thanks to Labkit's protocol-agnostic API.

## Deprecations

- **OpenTracing Protocol** - The OpenTracing project is archived and deprecated. Use OpenTelemetry (`otlp://` connection string) for new projects. OpenTracing support (`opentracing://` connection string) is maintained for backward compatibility during migration.
- **`Labkit::Tracing::GRPCInterceptor`** - Deprecated, use `Labkit::Tracing::GRPC::ClientInterceptor` instead

## Example: Complete Setup

### With Automatic Instrumentation (Recommended for OpenTelemetry)

```ruby
# config/initializers/tracing.rb
require 'opentelemetry/instrumentation/all'

if Labkit::Tracing.enabled?
  Labkit::Tracing::Factory.create_tracer("my-rails-app", ENV["GITLAB_TRACING"]) do |c|
    # Enable all available OpenTelemetry instrumentation
    c.use_all()

    # Add deployment metadata
    c.resource = c.resource.merge(
      OpenTelemetry::SDK::Resources::Resource.create(
        'deployment.environment' => Rails.env,
        'service.version' => MyApp::VERSION
      )
    )
  end
end

# config/application.rb
module MyApp
  class Application < Rails::Application
    # Add Rack middleware for HTTP request tracing
    config.middleware.insert_after Rails::Rack::Logger, Labkit::Tracing::RackMiddleware
  end
end

# .env or environment variables
# GITLAB_TRACING="otlp://localhost:4318?sampler=probabilistic&sampler_param=0.01"
```

### With Manual Instrumentation Selection

```ruby
# config/initializers/tracing.rb

if Labkit::Tracing.enabled?
  # REQUIRED: Initialize the tracer to collect and export traces
  # Without this, instrumentation runs but traces aren't sent anywhere
  tracer = Labkit::Tracing::Factory.create_tracer("my-rails-app", ENV["GITLAB_TRACING"])

  # For OpenTracing/Jaeger, set the global tracer
  # For OpenTelemetry/OTLP, this is already configured globally by create_tracer
  OpenTracing.global_tracer = tracer if tracer

  # Instrument Rails components
  Rails.application.config.after_initialize do
    Labkit::Tracing::Rails::ActiveRecord::Subscriber.instrument
    Labkit::Tracing::Rails::ActionView::Subscriber.instrument
    Labkit::Tracing::Rails::ActiveSupport::Subscriber.instrument
    Labkit::Tracing::ExternalHttp.instrument
    Labkit::Tracing::Redis.instrument
  end
end

# config/application.rb
module MyApp
  class Application < Rails::Application
    # Add Rack middleware for HTTP request tracing
    config.middleware.insert_after Rails::Rack::Logger, Labkit::Tracing::RackMiddleware
  end
end
```

## Troubleshooting

### Tracing Not Working

**First, check if you initialized the tracer** - see [Middleware Works But No Traces Appear](#middleware-works-but-no-traces-appear) below.

If you've confirmed initialization is correct:

1. Verify `GITLAB_TRACING` is set correctly:
   ```ruby
   puts Labkit::Tracing.connection_string
   puts Labkit::Tracing.enabled?
   ```

2. Check for tracer creation errors in logs (warnings are emitted on failure)

3. Verify the Jaeger agent/collector is reachable

### Missing Spans

1. Ensure instrumentation is called after dependencies are loaded
2. Verify tracing is properly initialized and enabled:
   ```ruby
   Labkit::Tracing.enabled?
   ```

### Middleware Works But No Traces Appear

If your application runs without errors but you don't see traces in your backend (OTLP collector, Jaeger, etc.):

**1. Verify you called `Factory.create_tracer` in your initializer:**

This step is **required** - setting `GITLAB_TRACING` alone is not enough:

```ruby
# config/initializers/tracing.rb
if Labkit::Tracing.enabled?
  tracer = Labkit::Tracing::Factory.create_tracer("my-service", ENV["GITLAB_TRACING"])
  OpenTracing.global_tracer = tracer if tracer  # For OpenTracing/Jaeger
end
```

**2. For OpenTelemetry (OTLP), verify the tracer provider is initialized:**

```ruby
OpenTelemetry.tracer_provider.class
# Expected: OpenTelemetry::SDK::Trace::TracerProvider
# Problem:  OpenTelemetry::Internal::ProxyTracerProvider (means Factory.create_tracer wasn't called)
```

**3. For OpenTracing (Jaeger), verify the global tracer is set:**

```ruby
OpenTracing.global_tracer.class
# Expected: Jaeger::Client::Tracer
# Problem:  OpenTracing::Tracer (means tracer wasn't initialized/set)
```

**Why this happens:**

Without calling `Factory.create_tracer`, LabKit uses a no-op tracer that creates span objects for instrumentation but doesn't export them to any backend. This design prevents crashes when tracing is misconfigured, but means no trace data is collected.

### High Overhead

1. Reduce sampling rate:
   ```bash
   export GITLAB_TRACING="opentracing://jaeger?udp_endpoint=localhost:6831&sampler=probabilistic&sampler_param=0.001"
   ```

2. Disable stack traces or limit to specific operations:
   ```bash
   export GITLAB_TRACING_INCLUDE_STACKTRACE=""
   ```

## References

- [OpenTelemetry Documentation](https://opentelemetry.io/docs/languages/ruby/)
- [OpenTelemetry Protocol (OTLP) Specification](https://opentelemetry.io/docs/specs/otlp/)
- [OpenTracing Specification](https://opentracing.io/specification/)
- [Jaeger Documentation](https://www.jaegertracing.io/docs/)
- [Rails Instrumentation Guide](https://guides.rubyonrails.org/active_support_instrumentation.html)