Improved HTTP metrics for the Container Registry
Context
The Container Registry has been using custom Prometheus metric collectors (from Docker). As part of gitlab-org/container-registry#31 (closed), we've moved away from these custom collectors and are now using LabKit for the HTTP metrics.
The development is now complete and ready to be merged and released. This issue is to discuss and coordinate the deployment of these changes and any changes that might be needed for Prometheus and Grafana dashboards.
Changes
Before
Name | Type | Cardinality | Help |
---|---|---|---|
promhttp_metric_handler_requests_in_flight | gauge | 1 | Current number of scrapes being served. |
promhttp_metric_handler_requests_total | counter | 3 | Total number of scrapes by HTTP status code. |
registry_http_in_flight_requests | gauge | 8 | The in-flight HTTP requests |
registry_http_request_duration_seconds | histogram | 0 | The HTTP request latencies in seconds. |
registry_http_request_size_bytes | histogram | 0 | The HTTP request sizes in bytes. |
registry_http_requests_total | counter | 2 | Total number of HTTP requests made. |
registry_http_response_size_bytes | histogram | 0 | The HTTP response sizes in bytes. |
registry_notifications_pending_total | gauge | 1 | The gauge of pending events in queue |
Metrics are grouped by the handler
label, whose value is the name of the Go HTTP handler used internally in the registry source code (e.g. base
, manifest
). This makes it difficult for someone unfamiliar with the registry codebase to identify the corresponding route path (dashboard).
After
Name | Type | Cardinality | Help |
---|---|---|---|
promhttp_metric_handler_requests_in_flight | gauge | 1 | Current number of scrapes being served. |
promhttp_metric_handler_requests_total | counter | 3 | Total number of scrapes by HTTP status code. |
registry_http_in_flight_requests | gauge | 1 | A gauge of requests currently being served by the http server. |
registry_http_request_duration_seconds | histogram | 0 | A histogram of latencies for requests to the http server. |
registry_http_request_size_bytes | histogram | 0 | A histogram of sizes of requests to the http server. |
registry_http_requests_total | counter | 3 | A counter for requests to the http server. |
registry_http_response_size_bytes | histogram | 0 | A histogram of response sizes for requests to the http server. |
registry_http_time_to_write_header_seconds | histogram | 0 | A histogram of request durations until the response headers are written. |
registry_notifications_pending_total | gauge | 1 | The gauge of pending events in queue |
We're already providing all metrics except registry_http_time_to_write_header_seconds
. The difference is that we'll now be using the buckets and descriptions from LabKit for all these metrics.
Additionally, we've renamed the label handler
to route,
and for its value, we're now using the route path (e.g. /v2
, /v2/{name}/manifests/{reference}
) instead of the handler name. This makes the container registry metrics more similar to the GitLab API metrics (dashboard).
Example
Before
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP registry_http_in_flight_requests The in-flight HTTP requests
# TYPE registry_http_in_flight_requests gauge
registry_http_in_flight_requests{handler="base"} 0
registry_http_in_flight_requests{handler="blob"} 0
registry_http_in_flight_requests{handler="blob_upload"} 0
registry_http_in_flight_requests{handler="blob_upload_chunk"} 0
registry_http_in_flight_requests{handler="catalog"} 0
registry_http_in_flight_requests{handler="manifest"} 0
registry_http_in_flight_requests{handler="tag"} 0
registry_http_in_flight_requests{handler="tags"} 0
# HELP registry_http_request_duration_seconds The HTTP request latencies in seconds.
# TYPE registry_http_request_duration_seconds histogram
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="0.005"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="0.01"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="0.025"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="0.05"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="0.1"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="0.25"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="0.5"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="1"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="2.5"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="5"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="10"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="25"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="60"} 1
registry_http_request_duration_seconds_bucket{handler="catalog",method="get",le="+Inf"} 1
registry_http_request_duration_seconds_sum{handler="catalog",method="get"} 0.003966673
registry_http_request_duration_seconds_count{handler="catalog",method="get"} 1
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="0.005"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="0.01"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="0.025"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="0.05"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="0.1"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="0.25"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="0.5"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="1"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="2.5"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="5"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="10"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="25"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="60"} 2
registry_http_request_duration_seconds_bucket{handler="tags",method="get",le="+Inf"} 2
registry_http_request_duration_seconds_sum{handler="tags",method="get"} 0.003753272
registry_http_request_duration_seconds_count{handler="tags",method="get"} 2
# HELP registry_http_request_size_bytes The HTTP request sizes in bytes.
# TYPE registry_http_request_size_bytes histogram
registry_http_request_size_bytes_bucket{handler="catalog",le="1024"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="2048"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="4096"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="8192"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="16384"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="32768"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="65536"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="131072"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="262144"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="524288"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="1.048576e+06"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="2.097152e+06"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="4.194304e+06"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="8.388608e+06"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="1.6777216e+07"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="3.3554432e+07"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="6.7108864e+07"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="1.34217728e+08"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="2.68435456e+08"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="5.36870912e+08"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="1.073741824e+09"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="2.147483648e+09"} 1
registry_http_request_size_bytes_bucket{handler="catalog",le="+Inf"} 1
registry_http_request_size_bytes_sum{handler="catalog"} 117
registry_http_request_size_bytes_count{handler="catalog"} 1
registry_http_request_size_bytes_bucket{handler="tags",le="1024"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="2048"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="4096"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="8192"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="16384"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="32768"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="65536"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="131072"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="262144"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="524288"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="1.048576e+06"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="2.097152e+06"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="4.194304e+06"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="8.388608e+06"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="1.6777216e+07"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="3.3554432e+07"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="6.7108864e+07"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="1.34217728e+08"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="2.68435456e+08"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="5.36870912e+08"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="1.073741824e+09"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="2.147483648e+09"} 2
registry_http_request_size_bytes_bucket{handler="tags",le="+Inf"} 2
registry_http_request_size_bytes_sum{handler="tags"} 253
registry_http_request_size_bytes_count{handler="tags"} 2
# HELP registry_http_requests_total Total number of HTTP requests made.
# TYPE registry_http_requests_total counter
registry_http_requests_total{code="200",handler="catalog",method="get"} 1
registry_http_requests_total{code="200",handler="tags",method="get"} 1
registry_http_requests_total{code="404",handler="tags",method="get"} 1
# HELP registry_http_response_size_bytes The HTTP response sizes in bytes.
# TYPE registry_http_response_size_bytes histogram
registry_http_response_size_bytes_bucket{handler="catalog",le="1024"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="2048"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="4096"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="8192"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="16384"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="32768"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="65536"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="131072"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="262144"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="524288"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="1.048576e+06"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="2.097152e+06"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="4.194304e+06"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="8.388608e+06"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="1.6777216e+07"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="3.3554432e+07"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="6.7108864e+07"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="1.34217728e+08"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="2.68435456e+08"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="5.36870912e+08"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="1.073741824e+09"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="2.147483648e+09"} 1
registry_http_response_size_bytes_bucket{handler="catalog",le="+Inf"} 1
registry_http_response_size_bytes_sum{handler="catalog"} 29
registry_http_response_size_bytes_count{handler="catalog"} 1
registry_http_response_size_bytes_bucket{handler="tags",le="1024"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="2048"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="4096"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="8192"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="16384"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="32768"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="65536"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="131072"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="262144"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="524288"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="1.048576e+06"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="2.097152e+06"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="4.194304e+06"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="8.388608e+06"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="1.6777216e+07"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="3.3554432e+07"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="6.7108864e+07"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="1.34217728e+08"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="2.68435456e+08"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="5.36870912e+08"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="1.073741824e+09"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="2.147483648e+09"} 2
registry_http_response_size_bytes_bucket{handler="tags",le="+Inf"} 2
registry_http_response_size_bytes_sum{handler="tags"} 147
registry_http_response_size_bytes_count{handler="tags"} 2
After
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 2
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP registry_http_in_flight_requests A gauge of requests currently being served by the http server.
# TYPE registry_http_in_flight_requests gauge
registry_http_in_flight_requests 0
# HELP registry_http_request_duration_seconds A histogram of latencies for requests to the http server.
# TYPE registry_http_request_duration_seconds histogram
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="0.005"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="0.025"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="0.1"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="0.5"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="1"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="10"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="30"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="60"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="300"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="+Inf"} 1
registry_http_request_duration_seconds_sum{code="200",method="get",route="/v2/_catalog"} 0.003994198
registry_http_request_duration_seconds_count{code="200",method="get",route="/v2/_catalog"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="0.005"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="0.025"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="0.1"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="0.5"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="1"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="10"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="30"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="60"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="300"} 1
registry_http_request_duration_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="+Inf"} 1
registry_http_request_duration_seconds_sum{code="200",method="get",route="/v2/{name}/tags/list"} 0.003148036
registry_http_request_duration_seconds_count{code="200",method="get",route="/v2/{name}/tags/list"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="0.005"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="0.025"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="0.1"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="0.5"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="1"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="10"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="30"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="60"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="300"} 1
registry_http_request_duration_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="+Inf"} 1
registry_http_request_duration_seconds_sum{code="404",method="get",route="/v2/{name}/tags/list"} 0.001440414
registry_http_request_duration_seconds_count{code="404",method="get",route="/v2/{name}/tags/list"} 1
# HELP registry_http_request_size_bytes A histogram of sizes of requests to the http server.
# TYPE registry_http_request_size_bytes histogram
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="10"} 0
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="64"} 0
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="256"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="1024"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="65536"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="262144"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="1.048576e+06"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="6.7108864e+07"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="+Inf"} 1
registry_http_request_size_bytes_sum{code="200",method="get",route="/v2/_catalog"} 117
registry_http_request_size_bytes_count{code="200",method="get",route="/v2/_catalog"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="10"} 0
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="64"} 0
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="256"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="1024"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="65536"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="262144"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="1.048576e+06"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="6.7108864e+07"} 1
registry_http_request_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="+Inf"} 1
registry_http_request_size_bytes_sum{code="200",method="get",route="/v2/{name}/tags/list"} 126
registry_http_request_size_bytes_count{code="200",method="get",route="/v2/{name}/tags/list"} 1
registry_http_request_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="10"} 0
registry_http_request_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="64"} 0
registry_http_request_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="256"} 1
registry_http_request_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="1024"} 1
registry_http_request_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="65536"} 1
registry_http_request_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="262144"} 1
registry_http_request_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="1.048576e+06"} 1
registry_http_request_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="6.7108864e+07"} 1
registry_http_request_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="+Inf"} 1
registry_http_request_size_bytes_sum{code="404",method="get",route="/v2/{name}/tags/list"} 127
registry_http_request_size_bytes_count{code="404",method="get",route="/v2/{name}/tags/list"} 1
# HELP registry_http_requests_total A counter for requests to the http server.
# TYPE registry_http_requests_total counter
registry_http_requests_total{code="200",method="get",route="/v2/_catalog"} 1
registry_http_requests_total{code="200",method="get",route="/v2/{name}/tags/list"} 1
registry_http_requests_total{code="404",method="get",route="/v2/{name}/tags/list"} 1
# HELP registry_http_response_size_bytes A histogram of response sizes for requests to the http server.
# TYPE registry_http_response_size_bytes histogram
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="10"} 0
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="64"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="256"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="1024"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="65536"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="262144"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="1.048576e+06"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="6.7108864e+07"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/_catalog",le="+Inf"} 1
registry_http_response_size_bytes_sum{code="200",method="get",route="/v2/_catalog"} 29
registry_http_response_size_bytes_count{code="200",method="get",route="/v2/_catalog"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="10"} 0
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="64"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="256"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="1024"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="65536"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="262144"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="1.048576e+06"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="6.7108864e+07"} 1
registry_http_response_size_bytes_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="+Inf"} 1
registry_http_response_size_bytes_sum{code="200",method="get",route="/v2/{name}/tags/list"} 31
registry_http_response_size_bytes_count{code="200",method="get",route="/v2/{name}/tags/list"} 1
registry_http_response_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="10"} 0
registry_http_response_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="64"} 0
registry_http_response_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="256"} 1
registry_http_response_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="1024"} 1
registry_http_response_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="65536"} 1
registry_http_response_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="262144"} 1
registry_http_response_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="1.048576e+06"} 1
registry_http_response_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="6.7108864e+07"} 1
registry_http_response_size_bytes_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="+Inf"} 1
registry_http_response_size_bytes_sum{code="404",method="get",route="/v2/{name}/tags/list"} 116
registry_http_response_size_bytes_count{code="404",method="get",route="/v2/{name}/tags/list"} 1
# HELP registry_http_time_to_write_header_seconds A histogram of request durations until the response headers are written.
# TYPE registry_http_time_to_write_header_seconds histogram
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="0.005"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="0.025"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="0.1"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="0.5"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="1"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="10"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="30"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/_catalog",le="+Inf"} 1
registry_http_time_to_write_header_seconds_sum{code="200",method="get",route="/v2/_catalog"} 0.003977299
registry_http_time_to_write_header_seconds_count{code="200",method="get",route="/v2/_catalog"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="0.005"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="0.025"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="0.1"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="0.5"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="1"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="10"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="30"} 1
registry_http_time_to_write_header_seconds_bucket{code="200",method="get",route="/v2/{name}/tags/list",le="+Inf"} 1
registry_http_time_to_write_header_seconds_sum{code="200",method="get",route="/v2/{name}/tags/list"} 0.003145559
registry_http_time_to_write_header_seconds_count{code="200",method="get",route="/v2/{name}/tags/list"} 1
registry_http_time_to_write_header_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="0.005"} 1
registry_http_time_to_write_header_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="0.025"} 1
registry_http_time_to_write_header_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="0.1"} 1
registry_http_time_to_write_header_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="0.5"} 1
registry_http_time_to_write_header_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="1"} 1
registry_http_time_to_write_header_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="10"} 1
registry_http_time_to_write_header_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="30"} 1
registry_http_time_to_write_header_seconds_bucket{code="404",method="get",route="/v2/{name}/tags/list",le="+Inf"} 1
registry_http_time_to_write_header_seconds_sum{code="404",method="get",route="/v2/{name}/tags/list"} 0.00127936
registry_http_time_to_write_header_seconds_count{code="404",method="get",route="/v2/{name}/tags/list"} 1
# HELP registry_notifications_pending_total The gauge of pending events in queue
# TYPE registry_notifications_pending_total gauge
registry_notifications_pending_total 0
Considerations
It seems to me that the only necessary change is to update the registry Grafana dashboard (link) so that we group requests by route
instead of handler
? If so, I can make that change, just want to confirm that anything else is needed.
Related to gitlab-org/container-registry#31 (closed).