Add Elasticsearch request metrics to rack middleware

What does this MR do?

This will export metrics to Prometheus to allow us to track number of requests and time for requests for different controllers. A follow up will involve adding this to Grafana rails controller details.

This will allow us to get a detailed break down of time spent in Elasticsearch requests by controller and action. This is following the same pattern used in lib/gitlab/metrics/redis_rack_middleware.rb.

While doing this I also noticed an opportunity to make a small refactoring to both of these middleware.

Firstly I moved the initialization of the metrics into initialize so that they aren't invoked multiple times.

Secondly I removed the conditional logic which is avoiding adding metrics when metrics are disabled. This is not necessary because these rack middleware are only inserted when metrics are enabled and there will always be a transaction present as it is unconditionally created in Gitlab::Metrics::RackMiddleware. Additionally we don't need to worry about this setting changing since it already requires a restart if you disable prometheus for it to take affect and I've manually tested this works as expected when enabled and disabled.

Screenshots

Perform a few searches the look at the metrics:

$ curl --silent http://localhost:3000/-/metrics |grep elasticsearch | grep SearchController

http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="+Inf"} 28
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="0.1"} 26
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="0.25"} 27
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="0.5"} 28
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="1"} 28
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="10"} 28
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="2.5"} 28
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="300"} 28
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="5"} 28
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="60"} 28
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="count.json",le="600"} 28
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="+Inf"} 4
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="0.1"} 3
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="0.25"} 3
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="0.5"} 4
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="1"} 4
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="10"} 4
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="2.5"} 4
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="300"} 4
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="5"} 4
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="60"} 4
http_elasticsearch_requests_duration_seconds_bucket{controller="SearchController",action="show",le="600"} 4
http_elasticsearch_requests_duration_seconds_count{controller="SearchController",action="count.json"} 28
http_elasticsearch_requests_duration_seconds_count{controller="SearchController",action="show"} 4
http_elasticsearch_requests_duration_seconds_sum{controller="SearchController",action="count.json"} 0.81737800000000005
http_elasticsearch_requests_duration_seconds_sum{controller="SearchController",action="show"} 0.69054599999999999
http_elasticsearch_requests_total{controller="SearchController",action="count.json"} 20
http_elasticsearch_requests_total{controller="SearchController",action="show"} 8

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team

#214279 (closed)