Instrument blob downloads
Problem
We currently don't have any fine-grain metrics for blob downloads. We only have high-level storage metrics, such as the count of operations (Get, Delete, etc.). This makes it difficult to have visibility over the number, frequency, and size of downloaded blobs.
Related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13644 (internal).
Solution
Expose a new Prometheus counter and histogram for blob downloads. This should include the size of downloaded* blobs, as well as whether these downloads have been pre-signed and hand over to clients (if supported by the storage backends and enabled in the registry config) or served directly.
We can emit these metrics after stating the blob (which gives us its size) and before redirecting clients to the pre-signed URL (link) or serving it directly (link).
* Caveat: If the registry pre-signs download URLs and redirects clients to them, we can't be sure whether the download that we have reported is going to be done and succeed. This is true because once we hand over the download URL to clients, we have no way to determine whether it'll be used and how many times. These downloads are served directly by the storage backend.