Port essential database metrics to Sidekiq
What does this MR do?
A part of #323163 (closed), a following up MR of #323165 (closed).
In summary, this MR brings those metrics to Sidekiq:
gitlab_database_transaction_seconds
gitlab_sql_<role>_duration_seconds
gitlab_transaction_db_<role>_count_total
gitlab_transaction_db_<role_cached_count_total
Each metric has endpoint_id
and feature_category
labels. The endpoint_id
field is a universal label for endpoints, for exampleProjects::MergesController#index
, /api/v4/users/:id
or PagesWorker
. It is newly introduced recently (!55564 (merged) and gitlab-com/gl-infra/scalability#790 (closed)). It should be considered to be the new standard soon.
As each worker is classified into one feature category, the cardicardinality of the new metrics correlates to the number of workers, which is reasonable comparing to the benefits the metrics yield.
Those metrics are the blockers for &5434 (closed) and essential observabilities for #322452 (closed).
Solutions
This approach follows the approach of decoupling metrics recording and metrics persistence. It means that the instrumenters can focus on make the metrics and directly related information are captured collectly. They then publish into an event queue (I want to be generic, but probaly ActiveSupport::Notifications
). The subscribers listen to the event, accumulate, add platform specific metadata, and persist into different output sources.
RedisInstrumenter | |
| | Prometheus
ARTransactionInstrumenter | |
| ActiveSupport +-------------------+ | Structured Logs
ExternalHTTPInstrumenter +----------------------->| Subscribers +-------->|
| Notifications +-------------------+ | Lograge
Other events public by gems | - Add platform metadata |
| (via transaction objects |
| - Accumulate | Tracing
... | (sum per request for example)|
- Transform events
In detail:
- This MR introduces
Gitlab::Metrics::BackgroundTransaction
(again). This object adds Sidekiq-specific labels (endpoint_id
andfeature_category
) to the metrics before publishing. The subscribers explicitly publish the metrics for desirable runtimes like the following code snippet. As the transaction objects are created in the Rack middleware and sidekiq middleware, the metrics for the opposite runtime are ignored automatically.
web_transaction&.observe(histogram, event.duration / 1000.0) do
buckets DURATION_BUCKET
end
background_transaction&.observe(histogram, event.duration / 1000.0) do
buckets DURATION_BUCKET
end
-
Gitlab::Database
contains a money patch wrapping aroundActiveRecord::Base.transaction
. This patch publishes a transaction event when this block is called. -
Gitlab::Metrics::Subscribers::ActiveRecord
listens totransaction
andsql
events and:- Accumulate data for structured logs and performance bar
- Publish prometheus metrics via
Gitlab::Metrics::WebTransaction
- Publish prometheus metrics via
Gitlab::Metrics::BackgroundTransaction
Screenshots (strongly suggested)
Those screenshots are captured in the metric endpoint served by Sidekiq in my local environment.
Does this MR meet the acceptance criteria?
Conformity
-
📋 Does this MR need a changelog?-
I have included a changelog entry. -
I have not included a changelog entry because _____.
-
-
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team