Add RoundTripper that records external HTTP requests
We already have tracing (Jaeger) and request correlation roundtrippers that applications can use to enrich HTTP requests with timing data.
What we don't have are simple request metrics tracked into Prometheus and/or the application logs. One use case where the lack of visibility around this has come up several times before is serving data from GCS downstream, such as uploads (artifacts, images, etc.) Workhorse uses custom metrics to track upload request latencies: https://gitlab.com/gitlab-org/gitlab/blob/107768e4096ae4ada25901ad3be2508e03da80be/workhorse/internal/objectstore/prometheus.go
However, this solution is:
- specific to Workhorse
- only tracks one particular type of request (uploads i.e. writes)
We should add a new RoundTripper
that tracks generic HTTP request metrics into Prometheus. Workhorse could then use this in its HttpTransport
: https://gitlab.com/gitlab-org/gitlab/blob/c8a45af37ee4fb99ee84241c78fa3a5331d1c09b/workhorse/internal/helper/httptransport/http_transport.go#L30, as would other apps.
This could simply be a latency histogram for now, and a status code counter.
We can also use these metrics to create SLIs as we did here for Workhorse uploads: https://gitlab.com/gitlab-com/runbooks/blob/kube-metrics/metrics-catalog/services/google-cloud-storage.jsonnet#L41-68. These components are surfaced in this dashboard.