Inject migration code path in HTTP metrics
This MR adds the required changes to embed a new migration_path
label in all registry_http_*
Prometheus metrics so that we can partition them by code path (old, new, or none). The value for this label cannot be set at the handler level, it has to be set at the request level, because it varies per request.
So far we've been using gitlab.com/gitlab-org/labkit/metrics
to automatically build/expose these HTTP metrics, which internally uses github.com/prometheus/client_golang/prometheus/promhttp
(here). Unfortunatelly, promhttp
only exposes code
and method
labels (plus any fixed labels, which we use for the route
label here, defined once, when setting up each handler's instrumentation), it does not support dynamic/per-request labels. Because of this, it's not possible to achieve this without custom modifications.
promhttp
is meant to be a generic/simple implementation for the most common scenario, so there is no desire to support dynamic/per-request labels like we're trying to achieve here (and the same was confirmed with the library authors in their mailing list here). Therefore, there is no point in changing Labkit to support something like this either, as there is no support for it in the underlying promhttp
package.
Considering this, and because this is a specialized need, which we only need to keep around for the duration of the Phase 1 migration, the path of least resistance is to copy the source of the corresponding promhttp
and labkit/metrics
instrumentation methods and modify it so that it adds the label to each metric as we want to. This is far from ideal but it's the easiest way and the benefit from having this label during the migration is invaluable.
Related to #391 (closed)