Create an API to define & track SLI ratio metrics in the ruby Application

For &525 we want to figure out a way to easily define, initialize and use SLI (success-rate) type of metrics in the ruby parts of our codebase. This issue is to discuss what that should look like. Ideally this code ends up in Labkit to be abstracted away of the main application.

In this iteration, we want to define an API that allows developers to consistently initialize and increment 2 counters:

1. operation rate: the counter that is incremented on every operation (for example a request)
2. success rate: the counter that is incremented on a successful operation (for example a request with status code 2xx, or a request that was fast enough compared to the threshold)

The API should allow initializing these 2 counters with all the possible label combinations we expect. Though when tracking an unexpected combination, the code should not fail (it could perhaps emit a warning in the logs, now or in a future iteration?)

Later, we could implement the same kind of thing for our go-services, making sure that the metrics emitted look the same.

Ideas

Initialization (Idea 1)

Labkit::Metrics.initialize_sli(:rails_request_apdex, label_combinations:[{ feature_category: 'code_review', endpoint_id: 'Projects::MergeRequestsController#show' }]

This initializes these: 2 counters to 0 with all of the passed label combinations

• labkit_sli:rails_request_apdex:total
• labkit_sli:rails_request_apdex:success_total

Observation (Idea 1)

rails_request_apdex = Labkit::Metrics.sli(:rails_request_apdex)
observation = rails_request_apdex.observe({ feature_category: 'code_review', endpoint_id: 'Projects::MergeRequestsController#show' }) # increments the total
observation.success! if duration < 1# increments the success

Though this splits the observe and success quite a bit

Observation (Idea 2)

Labkit::Metrics.sli(:rails_request_index).observe(
  labels: {...} 
  success: duration < 1 # required argument
)

The :total counter always incremented, the :success_total is only incremented when success: true is passed.