Add database transaction application SLIs

What does this MR do and why?

This MR add the database transaction application SLIs. We define 2 thresholds and a default 1s threshold. These values are expected to be refined over time.

See gitlab-com/gl-infra/scalability#3166 (closed) and discussion in gitlab-com/gl-infra/scalability#2863 (comment 1856890752)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

1. Apply the patch

diff --git a/app/workers/chaos/sleep_worker.rb b/app/workers/chaos/sleep_worker.rb
index 149bab5d9d3c..dd45b0886aeb 100644
--- a/app/workers/chaos/sleep_worker.rb
+++ b/app/workers/chaos/sleep_worker.rb
@@ -10,7 +10,13 @@ class SleepWorker # rubocop:disable Scalability/IdempotentWorker
     include ChaosQueue

     def perform(duration_s)
-      Gitlab::Chaos.sleep(duration_s)
+      # Gitlab::Chaos.sleep(duration_s)
+      ApplicationRecord.transaction do
+        sleep(duration_s)
+      end
+      Ci::ApplicationRecord.transaction do
+        sleep(duration_s)
+      end
     end
   end
 end

2. Enable sidekiq exporter in config/gitlab.yml

  monitoring:
    # Sidekiq exporter is a webserver built in to Sidekiq to expose Prometheus metrics
    sidekiq_exporter:
      enabled: true
      address: 127.0.0.1
      port: 3807

3. Run gdk restart rails

4. Curl the exporter

➜  gitlab git:(sc1-db-txn-application-sli) ✗ curl -s 'localhost:3807/metrics' | rg Chaos::SleepWorker | rg gitlab_sli_db
gitlab_sli_db_transaction_apdex_success_total{db_config_name="ci",destination_shard_redis="main",external_dependencies="no",feature_category="not_owned",queue="default",urgency="low",worker="Chaos::SleepWorker"} 0
gitlab_sli_db_transaction_apdex_success_total{db_config_name="main",destination_shard_redis="main",external_dependencies="no",feature_category="not_owned",queue="default",urgency="low",worker="Chaos::SleepWorker"} 0
gitlab_sli_db_transaction_apdex_total{db_config_name="ci",destination_shard_redis="main",external_dependencies="no",feature_category="not_owned",queue="default",urgency="low",worker="Chaos::SleepWorker"} 0
gitlab_sli_db_transaction_apdex_total{db_config_name="main",destination_shard_redis="main",external_dependencies="no",feature_category="not_owned",queue="default",urgency="low",worker="Chaos::SleepWorker"} 0

5. Open a gdk console and enqueue some jobs

Loading development environment (Rails 7.0.8.1)
[1] pry(main)> Chaos::SleepWorker.perform_async(4)
=> "ec8d1cb913836d4fdfb09b33"
[2] pry(main)> Chaos::SleepWorker.perform_async(1)
=> "6c89353e54b85a69d59c2ce6"
[3] pry(main)> Chaos::SleepWorker.perform_async(0.5)
=> "f8c7433a649f8af12cb9ca20"

6. Verify with another curl

➜  gitlab git:(sc1-db-txn-application-sli) ✗ curl -s 'localhost:3807/metrics' | rg Chaos::SleepWorker | rg gitlab_sli_db
gitlab_sli_db_transaction_apdex_success_total{db_config_name="ci",destination_shard_redis="main",external_dependencies="no",feature_category="not_owned",queue="default",urgency="low",worker="Chaos::SleepWorker"} 2
gitlab_sli_db_transaction_apdex_success_total{db_config_name="main",destination_shard_redis="main",external_dependencies="no",feature_category="not_owned",queue="default",urgency="low",worker="Chaos::SleepWorker"} 2
gitlab_sli_db_transaction_apdex_total{db_config_name="ci",destination_shard_redis="main",external_dependencies="no",feature_category="not_owned",queue="default",urgency="low",worker="Chaos::SleepWorker"} 3
gitlab_sli_db_transaction_apdex_total{db_config_name="main",destination_shard_redis="main",external_dependencies="no",feature_category="not_owned",queue="default",urgency="low",worker="Chaos::SleepWorker"} 3