feat: Covered Experience (!169) · Merge requests · GitLab.org / ruby / gems / labkit-ruby

Part of gitlab-com/gl-infra/observability/team#4114 (closed).

Initiate registry with config/covered_experiences (from calling application's root) by default
Validate required fields from the definition files using JSON schema
Covered experience events
Instrumenting an experience in development without having it in the registry will led to an error:

Click to expand

  40) Gitlab::Metrics::RequestsRackMiddleware#call application context SLI satisfactory An unknown request falls back request's expectation to default (1 second)
      Failure/Error:
                  ::Labkit::CoveredExperience[:rails_request].start do |xp|
                    xp.push_attributes!(labels_from_context)

                    status, headers, body = @app.call(env)
                    result = [status, headers, body] && break if health_endpoint

                    xp.checkpoint

                    urgency = urgency_for_env(env)
                    if ::Gitlab::Metrics.record_duration_for_status?(status)

      RuntimeError:
        Covered Experience rails_request not found in the registry
      # /Users/hercules/GitLab/labkit-ruby/lib/labkit/covered_experience.rb:37:in `raise_or_null'
      # /Users/hercules/GitLab/labkit-ruby/lib/labkit/covered_experience.rb:28:in `[]'
      # ./lib/gitlab/metrics/requests_rack_middleware.rb:85:in `call'
      # ./spec/lib/gitlab/metrics/requests_rack_middleware_spec.rb:447:in `block (6 levels) in <top (required)>'
      # ./spec/lib/gitlab/metrics/requests_rack_middleware_spec.rb:14:in `block (3 levels) in <top (required)>'
      # ./lib/gitlab/application_context.rb:173:in `block in use'
      # /Users/hercules/GitLab/labkit-ruby/lib/labkit/context.rb:35:in `with_context'
      # ./lib/gitlab/application_context.rb:173:in `use'
      # ./lib/gitlab/application_context.rb:96:in `with_context'
      # ./spec/lib/gitlab/metrics/requests_rack_middleware_spec.rb:13:in `block (2 levels) in <top (required)>'
      # ./spec/spec_helper.rb:472:in `block (3 levels) in <top (required)>'
      # ./lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'
      # ./spec/spec_helper.rb:471:in `block (2 levels) in <top (required)>'
      # ./spec/spec_helper.rb:466:in `block (3 levels) in <top (required)>'
      # ./spec/support/sidekiq_middleware.rb:9:in `with_sidekiq_server_middleware'
      # ./spec/spec_helper.rb:457:in `block (2 levels) in <top (required)>'
      # ./spec/spec_helper.rb:453:in `block (3 levels) in <top (required)>'
      # /Users/hercules/GitLab/labkit-ruby/lib/labkit/context.rb:35:in `with_context'
      # ./lib/gitlab/application_context.rb:100:in `with_raw_context'
      # ./spec/spec_helper.rb:453:in `block (2 levels) in <top (required)>'
      # ./spec/spec_helper.rb:424:in `block (3 levels) in <top (required)>'
      # ./lib/gitlab/ci/config/feature_flags.rb:38:in `ensure_correct_usage'
      # ./spec/spec_helper.rb:423:in `block (2 levels) in <top (required)>'
      # ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
      # ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>'
      # ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
      # ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>'

Other environments other than development and test won't do this.

A "no-op" experience (null) instance will be returned in production environments when the experience is not in the registry
Start the experience using a block so that we can capture errors and complete the experience automatically for the user, and checkpoints within the experience can be instrumented manually as needed.

Here is an example (though this specific one looks terrible to the eyes):

Click to expand

diff --git a/config/covered_experiences/rails_request.yml b/config/covered_experiences/rails_request.yml
new file mode 100644
index 000000000000..1be9226c608d
--- /dev/null
+++ b/config/covered_experiences/rails_request.yml
@@ -0,0 +1,3 @@
+description: "This measures the experience of Ruby On Rails requests"
+feature_category: "scalability"
+urgency: "sync_fast"
diff --git a/lib/gitlab/metrics/requests_rack_middleware.rb b/lib/gitlab/metrics/requests_rack_middleware.rb
index dff9f9cbcb4d..d28e115c1c3e 100644
--- a/lib/gitlab/metrics/requests_rack_middleware.rb
+++ b/lib/gitlab/metrics/requests_rack_middleware.rb
@@ -80,19 +80,30 @@ def call(env)
         status = 'undefined'
 
         begin
-          status, headers, body = @app.call(env)
-          return [status, headers, body] if health_endpoint
-
-          urgency = urgency_for_env(env)
-          if ::Gitlab::Metrics.record_duration_for_status?(status)
-            elapsed = ::Gitlab::Metrics::System.monotonic_time - started
-            self.class.http_request_duration_seconds.observe({ method: method }, elapsed)
-            record_apdex(urgency, elapsed)
-          end
+          result = nil
+
+          ::Labkit::CoveredExperience[:rails_request].start do |xp|
+            xp.push_attributes!(labels_from_context)
+
+            status, headers, body = @app.call(env)
+            result = [status, headers, body] && break if health_endpoint
+
+            xp.checkpoint
 
-          record_error(urgency, status)
+            urgency = urgency_for_env(env)
+            if ::Gitlab::Metrics.record_duration_for_status?(status)
+              elapsed = ::Gitlab::Metrics::System.monotonic_time - started
+              self.class.http_request_duration_seconds.observe({ method: method }, elapsed)
+              record_apdex(urgency, elapsed)
+            end
+
+            record_error(urgency, status)
+            xp.error! if ::Gitlab::Metrics.server_error?(status)
+
+            result = [status, headers, body]
+          end
 
-          [status, headers, body]
+          result
         rescue StandardError
           self.class.rack_uncaught_errors_count.increment
           raise

Most experiences within a simple request will naturally use this design pattern in a much nicer way than this.

Start one experience that needs to handle error and completion manually -- useful for multi-services experiences:

Here is an example:

Click to expand

diff --git a/config/covered_experiences/rails_request.yml b/config/covered_experiences/rails_request.yml
new file mode 100644
index 000000000000..1be9226c608d
--- /dev/null
+++ b/config/covered_experiences/rails_request.yml
@@ -0,0 +1,3 @@
+description: "This measures the experience of Ruby On Rails requests"
+feature_category: "scalability"
+urgency: "sync_fast"
diff --git a/lib/gitlab/metrics/requests_rack_middleware.rb b/lib/gitlab/metrics/requests_rack_middleware.rb
index dff9f9cbcb4d..115a752bc744 100644
--- a/lib/gitlab/metrics/requests_rack_middleware.rb
+++ b/lib/gitlab/metrics/requests_rack_middleware.rb
@@ -80,9 +80,14 @@ def call(env)
         status = 'undefined'
 
         begin
+          xp = ::Labkit::CoveredExperience[:rails_request].start
+          xp.push_attributes!(labels_from_context)
+
           status, headers, body = @app.call(env)
           return [status, headers, body] if health_endpoint
 
+          xp.checkpoint
+
           urgency = urgency_for_env(env)
           if ::Gitlab::Metrics.record_duration_for_status?(status)
             elapsed = ::Gitlab::Metrics::System.monotonic_time - started
@@ -91,12 +96,15 @@ def call(env)
           end
 
           record_error(urgency, status)
+          xp.error! if ::Gitlab::Metrics.server_error?(status)
 
           [status, headers, body]
         rescue StandardError
           self.class.rack_uncaught_errors_count.increment
           raise
         ensure
+          xp.complete
+
           if health_endpoint
             self.class.http_health_requests_total.increment(status: status.to_s, method: method)
           else

Checkpointing can be done multiple times.

The examples are using a simplified version of the rails_request application SLI. The examples are not meant to be merged as-is, they are just examples to make it easy to visualize the usage of Labkit::CoveredExperience's API -- we can either improve this to make it production ready, or throw it away and focus on another SLI to transform in an experience later. I picked this one because it is very self-contained and easy to grasp the flow.

After this one, we can follow up adding log messages, with extra context such as the correlation_id and the meta fields.

Bonus

I've added Labkit::RSpec, which contains custom matchers to verify covered experiences that I'm using to test, but this can also be used by the Rails app to test the real implementations.

It's flexible enough to host more custom matchers for other purposes if needed.

Edited Jul 23, 2025 by Hercules Merscher

feat: Covered Experience

Bonus

Merge request reports