fix: Only one OTel provider and selective instrumentation

**Automatic Initialization**

When `GITLAB_TRACING` is set, LabKit automatically creates and configures a tracer when the gem is loaded. No manual initialization is required in most cases.

When `GITLAB_TRACING` is set, LabKit automatically creates and configures a tracer:

- **Rails applications**: Railtie automatically inserts Labkit::Tracing::RackMiddleware into the middleware stack

- **Non-Rails applications**: Works automatically with no additional configuration

Auto-initialization provides:

- Automatic tracer creation with connection string settings

- Automatic tracer creation with connection string settings (sampler, exporter, service name)

- Service name from `service_name` query parameter (defaults to `"labkit-service"`)

- All available OpenTelemetry instrumentation enabled by default (`c.use_all()`)

- LabKit-specific instrumentation enabled automatically:

- Selective OpenTelemetry instrumentation for components that don't conflict with LabKit:

  - `ConcurrentRuby`, `Net::HTTP`, `ActionPack`, `ActionMailer`, `ActiveJob`

- LabKit's own instrumentation for components it handles directly:

  - HTTP request tracing (RackMiddleware for Rails)

  - Rails components (ActiveRecord, ActionView, ActiveSupport) - if Rails is available

  - Redis instrumentation - if Redis gem is loaded

  - External HTTP instrumentation (Net::HTTP, Excon, HTTPClient)

  - **Behavior**: Outputs spans immediately to stdout with no remote export

  - **Benefits**: No external collector required, instant feedback, simple debugging

**HTTPS Enforcement:** For security, non-localhost OTLP connections are automatically upgraded to HTTPS. Localhost addresses (`localhost`, `127.0.0.1`, `::1`) use plain HTTP. For example:

- `otlp://localhost:4318` → `http://localhost:4318`

- `otlp://collector.example.com:4318` → `https://collector.example.com:4318`

Ensure your OTLP collector supports TLS when using non-localhost endpoints.

### Sampling

**Default Behavior:** When no sampler is specified, probabilistic sampling is used with a **0.1% sample rate** (1 in 1000 traces).

### Automatic Initialization (Default Behavior)

When `GITLAB_TRACING` is set, LabKit automatically creates and configures a tracer when the gem loads:

When `GITLAB_TRACING` is set, LabKit automatically creates and configures a tracer:

```bash

# Set environment variable with service name

export GITLAB_TRACING="otlp://localhost:4318"

```

**Rails Applications:**

- Initialization happens automatically during Rails boot via Railtie

- Runs after all other gem initializers (including `opentelemetry-instrumentation-rails`)

- LabKit's TracerProvider is the final configuration used by the application

- **No application code changes required** - just set the environment variable

**Non-Rails Applications (Sinatra, Grape, standalone Ruby):**

- Auto-initialization happens automatically when `require 'gitlab-labkit'` is called, so no manual setup is needed.

Auto-initialization:

- Creates tracer with connection string settings (sampler, exporter, endpoints)

- Uses service name from `service_name` query parameter (defaults to `"labkit-service"`)

- Enables all available OpenTelemetry instrumentation

- Enables selective OpenTelemetry instrumentation (non-conflicting with LabKit's own)

- Enables LabKit instrumentation for ActiveRecord, ActionView, ActiveSupport, Redis, and External HTTP

- Sets up the global `OpenTelemetry.tracer_provider`

**No application code changes required** - just set the environment variable.

### Automatic Rails Middleware Insertion

When using Rails, the `Labkit::Tracing::RackMiddleware` is automatically inserted into your middleware stack when `GITLAB_TRACING` is set. No manual configuration needed!

# config/initializers/tracing.rb

# Override auto-initialization with custom configuration

Labkit::Tracing::Factory.create_tracer("my-custom-service", ENV["GITLAB_TRACING"]) do |c|

  # Selective instrumentation instead of c.use_all()

  c.use 'OpenTelemetry::Instrumentation::Rails'

  # Custom OTel instrumentation

  c.use 'OpenTelemetry::Instrumentation::ConcurrentRuby'

  c.use 'OpenTelemetry::Instrumentation::Sidekiq'

  # Add custom span processors

```

**When to use manual initialization:**

- You need selective instrumentation (not `c.use_all()`)

- You need different OTel instrumentations than the defaults

- You need custom span processors

- You need to add resource attributes

- You want to override the service name from code instead of the connection string

### Initialization Order and Precedence

**Auto-initialization:**

1. Runs when the gem is loaded (`require 'gitlab-labkit'`)

**Auto-initialization (all applications):**

1. Runs automatically when `require 'gitlab-labkit'` is called

2. Only runs if `GITLAB_TRACING` environment variable is set

3. Uses `service_name` query parameter or defaults to `"labkit-service"`

4. Attempts to enable all instrumentation with `c.use_all()`

4. Enables selective OpenTelemetry instrumentation (ConcurrentRuby, Net::HTTP, ActionPack, ActionMailer, ActiveJob) to avoid conflicts with LabKit's own instrumentation

5. Enables LabKit instrumentation for ActiveRecord, ActionView, ActiveSupport, Redis, and External HTTP

**Additional Rails behavior:**

1. Railtie runs after user config initializers (via `initializer ... after: :load_config_initializers`)

2. Automatically inserts `Labkit::Tracing::RackMiddleware` into the middleware stack

**Manual initialization:**

1. Runs in your application initializer (e.g., `config/initializers/tracing.rb`)

   - Span processors with OTLP exporter

2. **Configuration block** runs second and can:

   - Add automatic instrumentation (`use`, `use_all`)

   - Add automatic instrumentation (`use`)

   - Add additional span processors

   - Merge additional resource attributes

   - Override service_name if explicitly set in the block

# config/initializers/tracing.rb

Labkit::Tracing::Factory.create_tracer("my-rails-app", ENV["GITLAB_TRACING"]) do |c|

  # Enable all available OpenTelemetry instrumentation

  c.use_all()

  # Selective instrumentation (avoid conflicting with LabKit's own instrumentation

  # for ActiveRecord, ActionView, ActiveSupport, Redis, Rack, and Sidekiq)

  c.use("OpenTelemetry::Instrumentation::ConcurrentRuby")

  c.use("OpenTelemetry::Instrumentation::Net::HTTP")

  c.use("OpenTelemetry::Instrumentation::ActionPack") if defined?(ActionPack)

  c.use("OpenTelemetry::Instrumentation::ActionMailer") if defined?(ActionMailer)

  c.use("OpenTelemetry::Instrumentation::ActiveJob") if defined?(ActiveJob)

  # Add deployment metadata

  c.resource = c.resource.merge(

Note: Middleware is still automatically inserted even with custom tracer initialization.

### With Selective Instrumentation (Manual Override)

### With Custom Instrumentation (Manual Override)

If you want to control exactly which components are instrumented instead of using `c.use_all()`, override tracer initialization:

If you want to control exactly which OTel components are instrumented, override tracer initialization:

**Important:** When you manually call `Factory.create_tracer`, auto-initialization still runs but is replaced by your manual configuration. The LabKit-specific instrumentation (Rails, Redis, ExternalHttp) is still automatically enabled unless you explicitly disable auto-initialization.

```ruby

# config/initializers/tracing.rb

# Override auto-initialization for selective OpenTelemetry instrumentation

# Override auto-initialization for custom OpenTelemetry instrumentation

Labkit::Tracing::Factory.create_tracer("my-rails-app", ENV["GITLAB_TRACING"]) do |c|

  # Selective OpenTelemetry instrumentation (instead of c.use_all())

  c.use 'OpenTelemetry::Instrumentation::Rails'

  # Only enable the OTel instrumentations you need

  c.use 'OpenTelemetry::Instrumentation::ConcurrentRuby'

  c.use 'OpenTelemetry::Instrumentation::Sidekiq'

end

2. Check Rails logs for auto-insertion message:

    ```

   Labkit::Tracing: Automatically inserted RackMiddleware after Rails::Rack::Logger

    Labkit::Tracing::RackMiddleware automatically inserted after Labkit::Middleware::Rack

    ```

3. If using custom middleware positioning, verify the order is correct:

        # exception_object is the standard exception payload from ActiveSupport::Notifications

        # https://github.com/rails/rails/blob/v6.0.3.1/activesupport/lib/active_support/notifications/instrumenter.rb#L26

        exception = payload[:exception_object].presence || payload[:exception].presence

        Labkit::Tracing::TracingUtils.log_exception_on_span(span, exception)

        span.set_error(exception)

        tags(payload).each do |k, v|

          span.set_tag(k, v)

        service_name = detect_service_name(connection_string)

        require "opentelemetry/instrumentation/concurrent_ruby"

        require "opentelemetry/instrumentation/net/http"

        require "opentelemetry/instrumentation/action_pack" if defined?(ActionPack)

        require "opentelemetry/instrumentation/action_mailer" if defined?(ActionMailer)

        require "opentelemetry/instrumentation/active_job" if defined?(ActiveJob)

        Factory.create_tracer(service_name, connection_string) do |c|

          require "opentelemetry/instrumentation/all"

          c.use_all

          # Enable only non-conflicting instrumentations

          # Exclude: ActiveSupport, ActiveRecord, ActionView, Redis, Rack, Sidekiq

          # (LabKit provides its own instrumentation for these via AbstractInstrumenter)

          # Core instrumentations

          c.use("OpenTelemetry::Instrumentation::ConcurrentRuby")

          c.use("OpenTelemetry::Instrumentation::Net::HTTP")

          # Rails components that don't conflict with LabKit

          c.use("OpenTelemetry::Instrumentation::ActionPack") if defined?(ActionPack)

          c.use("OpenTelemetry::Instrumentation::ActionMailer") if defined?(ActionMailer)

          c.use("OpenTelemetry::Instrumentation::ActiveJob") if defined?(ActiveJob)

        end

        enable_labkit_instrumentation

              yield(c)

            end

            # SDK.configure doesn't expose sampler configuration directly

            # We need to replace the tracer provider to set the sampler

            # This is a known limitation of the OpenTelemetry Ruby SDK

            current_provider = OpenTelemetry.tracer_provider

            return unless current_provider.is_a?(OpenTelemetry::SDK::Trace::TracerProvider)

            # Create new provider with sampler, preserving resource from SDK.configure

            create_and_configure_provider(current_provider.resource, sampler, exporter)

            current_provider.sampler = sampler

          else

            create_and_configure_provider(base_resource, sampler, exporter)

          end

          begin

            yield span

          rescue StandardError => e

            log_exception_on_span(span, e)

            span.set_error(e)

            raise e

          end

        end

        span = tracer.start_span(operation_name, child_of: child_of, tags: tags, start_time: start_time)

        log_common_fields_on_span(span, operation_name)

        log_exception_on_span(span, exception) if exception

        span.set_error(exception)

        span.finish(end_timestamp: end_time)

      end

        span.log_event("stack", stack: caller.join('\n')) if include_stacktrace?(operation_name)

      end

      # Add exception logging to a span

      def self.log_exception_on_span(span, exception)

        span.set_error(exception)

      end

      def self.include_stacktrace?(operation_name)

        @include_stacktrace ||= Hash.new do |result, name|

          result[name] = Tracing.stacktrace_operations.any? { |stacktrace_operation| name.starts_with?(stacktrace_operation) }