Add instrumentation for read_multi notifications

What does this MR do and why?

This MR instruments read_multi and fetch_multi methods in ActiveSupport::Cache::RedisCacheStore.

Currently we only do not notify for cache_read_multi.active_support (see how key is formed: https://github.com/rails/rails/blob/v6.1.6.1/activesupport/lib/active_support/cache.rb#L726) as Gitlab::Metrics::Subscribers::RailsCache lacks it. In fairness, the Rails documentation does not include cache_read_multi.active_support.

See gitlab-com/gl-infra/scalability#2004 (closed)

Note that this MR implements 2 of 3 points here !104335 (comment 1177638034). Cross-slot detection should be performed at a lower level (redis lib) since keys in the ActiveSupport::Cache stage are not normalized (https://github.com/rails/rails/blob/dc1242fd5a4d91e63846ab552a07e19ebf8716ac/activesupport/lib/active_support/cache.rb#L655)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Local validation is trick since RackMiddleware is omitted in dev and test environments (https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/zz_metrics.rb#L17). I tested by making small modifications below:

Click to expand

diff --git a/app/services/projects/batch_count_service.rb b/app/services/projects/batch_count_service.rb
index 455c7211ab26..c955f97e8ff9 100644
--- a/app/services/projects/batch_count_service.rb
+++ b/app/services/projects/batch_count_service.rb
@@ -14,6 +14,7 @@ def refresh_cache_and_retrieve_data
       services_by_cache_key = count_services.index_by(&:cache_key)

       results = Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
+        Rails.cache.read_multi('a', 'b', 'c', 'd')
         Rails.cache.fetch_multi(*services_by_cache_key.keys) do |key|
           service = services_by_cache_key[key]

diff --git a/config/initializers/zz_metrics.rb b/config/initializers/zz_metrics.rb
index ff3ae9a24676..222ef5bbf954 100644
--- a/config/initializers/zz_metrics.rb
+++ b/config/initializers/zz_metrics.rb
@@ -14,7 +14,7 @@
 # In development mode, we turn off eager loading when we're running
 # `rails generate migration` because eager loading short-circuits the
 # loading of our custom migration templates.
-if Gitlab::Metrics.enabled? && !Rails.env.test? && !(Rails.env.development? && defined?(Rails::Generators))
+if Gitlab::Metrics.enabled? #&& !Rails.env.test? && !(Rails.env.development? && defined?(Rails::Generators))
   require 'pathname'
   require 'connection_pool'

1. Enable web_exporter metrics in config/gitlab.yml

  monitoring:
    web_exporter:
      enabled: true

2. Open up localhost:3000 and open projects (these should trigger refresh_cache_and_retrieve_data function)

3. Grep metrics to observe read_multi stats

curl localhost:8083/metrics 2> /dev/null  | grep gitlab_cache_read_multikey_count_bucket

# HELP gitlab_cache_read_multikey_count Multiprocess metric
# TYPE gitlab_cache_read_multikey_count histogram
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="+Inf"} 16
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="1"} 0
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="10"} 16
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="100"} 16
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="1000"} 16
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="5"} 8
gitlab_cache_read_multikey_count_bucket{action="index",controller="RootController",feature_category="projects",le="50"} 16


➜  gitlab git:(schin1-rails-cache-multikey-metrics) ✗ curl localhost:8083/metrics 2> /dev/null  | grep gitlab_cache_operation_duration_seconds_bucket | grep multi
gitlab_cache_operation_duration_seconds_bucket{le="+Inf",operation="read_multi"} 8
gitlab_cache_operation_duration_seconds_bucket{le="0.0001",operation="read_multi"} 0
gitlab_cache_operation_duration_seconds_bucket{le="0.001",operation="read_multi"} 0
gitlab_cache_operation_duration_seconds_bucket{le="0.01",operation="read_multi"} 8
gitlab_cache_operation_duration_seconds_bucket{le="0.1",operation="read_multi"} 8
gitlab_cache_operation_duration_seconds_bucket{le="1.0",operation="read_multi"} 8
gitlab_cache_operation_duration_seconds_bucket{le="1.0e-05",operation="read_multi"} 0

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.