Workhorse HTTP Metrics should distinguish when Workhorse has fetched a resource from object storage on behalf of the client
Currently the GitLab Workhorse metrics don't distinguish between requests which make onward object storage requests and those that don't.
This also cannot be determined based on the route.
For example in https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/incident-management/-/issues/353, the Dedicated Team are receiving Apdex alerts for some customers who have disabled HTTP Redirect functionality when Workhorse is returning a object storage reference.
For various reasons, for these customers, the configuration has been set that Workhorse streams the object storage resource and then returns it directly to the requestor, without using a signed URL redirect.
This leads to some very slow requests when large blobs (multi-gigabyte potentially) are loaded from object storage.
Workaround
When Workhorse fetches from object storage resources on behalf of users, the requests should be able to excluded from the SLIs.
One potential option would be to add a object_storage="true" (naming tbd) label to gitlab_workhorse_http_request_*. This could be use to segment requests which are taking an excessively long time due to object storage requests compared to other reasons, such as a slow backend.
Solution
Workhorse: set metrics labels for external URLs (!187163 - merged) adds a label ato Workhorse HTTP metrics fetched_external_url that can be used to filter/count these metrics in a different way.