Port essential database metrics to Sidekiq

What does this MR do?

A part of #323163 (closed), a following up MR of #323165 (closed).

In summary, this MR brings those metrics to Sidekiq:

• gitlab_database_transaction_seconds
• gitlab_sql_<role>_duration_seconds
• gitlab_transaction_db_<role>_count_total
• gitlab_transaction_db_<role_cached_count_total

Each metric has endpoint_id and feature_category labels. The endpoint_id field is a universal label for endpoints, for exampleProjects::MergesController#index, /api/v4/users/:id or PagesWorker. It is newly introduced recently (!55564 (merged) and gitlab-com/gl-infra/scalability#790 (closed)). It should be considered to be the new standard soon.

As each worker is classified into one feature category, the cardicardinality of the new metrics correlates to the number of workers, which is reasonable comparing to the benefits the metrics yield.

Those metrics are the blockers for &5434 (closed) and essential observabilities for #322452 (closed).

Solutions

This approach follows the approach of decoupling metrics recording and metrics persistence. It means that the instrumenters can focus on make the metrics and directly related information are captured collectly. They then publish into an event queue (I want to be generic, but probaly ActiveSupport::Notifications). The subscribers listen to the event, accumulate, add platform specific metadata, and persist into different output sources.

RedisInstrumenter            |                                                      |
                             |                                                      |  Prometheus
ARTransactionInstrumenter    |                                                      |
                             |    ActiveSupport       +-------------------+         |  Structured Logs
ExternalHTTPInstrumenter     +----------------------->|    Subscribers    +-------->|
                             |    Notifications       +-------------------+         |  Lograge
Other events public by gems  |                        - Add platform metadata       |
                             |                          (via transaction objects    |
                             |                        - Accumulate                  |  Tracing
...                          |                         (sum per request for example)|
                                                      - Transform events

In detail:

• This MR introduces Gitlab::Metrics::BackgroundTransaction (again). This object adds Sidekiq-specific labels (endpoint_id and feature_category) to the metrics before publishing. The subscribers explicitly publish the metrics for desirable runtimes like the following code snippet. As the transaction objects are created in the Rack middleware and sidekiq middleware, the metrics for the opposite runtime are ignored automatically.

          web_transaction&.observe(histogram, event.duration / 1000.0) do
            buckets DURATION_BUCKET
          end
          background_transaction&.observe(histogram, event.duration / 1000.0) do
            buckets DURATION_BUCKET
          end

• Gitlab::Database contains a money patch wrapping around ActiveRecord::Base.transaction. This patch publishes a transaction event when this block is called.
• Gitlab::Metrics::Subscribers::ActiveRecord listens to transaction and sql events and:
  • Accumulate data for structured logs and performance bar
  • Publish prometheus metrics via Gitlab::Metrics::WebTransaction
  • Publish prometheus metrics via Gitlab::Metrics::BackgroundTransaction

Screenshots (strongly suggested)

Those screenshots are captured in the metric endpoint served by Sidekiq in my local environment.

Does this MR meet the acceptance criteria?

Conformity

• 📋 Does this MR need a changelog?
  • I have included a changelog entry.
  • I have not included a changelog entry because _____.
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team