Exclude health endpoints from HTTP request metrics
Problem to solve
Eliminate pollution of application http request metrics.
Intended users
Personas are described at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/ -->
Further details
We currently include health check endpoints in the requests rack middleware metrics. This causes the application metrics to be polluted with healtcheck traffic. For example, when draining we return 503
on /-/readiness
in order cause the load balancer to remove a node in production.
Proposal
Filter the health check and metrics endpoints from being observed by the requests rack middleware.
Optionally, record those metrics separately in a new http_health_requests_total
metric. I don't think this needs to be a histogram like http_request_duration_seconds
.
Permissions and Security
Documentation
Availability & Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Links / references
See the infrastructure discussion issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8924