Skip to content

Prometheus instrumentation

Andrew Newdigate requested to merge prometheus-instrumentation into master

Extends !278 (merged) (merge that MR first and rebase)

Note until !278 (merged) is merged: use https://gitlab.com/gitlab-org/gitlab-workhorse/compare/an%2Fprometheus...prometheus-instrumentation to diff

This change adds the "standard" promhttp metrics to workhorse, implemented by github.com/prometheus/client_golang/prometheus/promhttp as http handler middlewares.

  • InstrumentHandlerCounter
  • InstrumentHandlerDuration
  • InstrumentHandlerInFlight
  • InstrumentHandlerRequestSize
  • InstrumentHandlerResponseSize
  • InstrumentHandlerTimeToWriteHeader

For each of these metrics, we'll have code, method and route labels.

For example:

# HELP gitlab_workhorse_http_in_flight_requests A gauge of requests currently being served by the registry.
# TYPE gitlab_workhorse_http_in_flight_requests gauge
gitlab_workhorse_http_in_flight_requests 0

# HELP gitlab_workhorse_http_request_duration_seconds A histogram of latencies for requests to the registry.
# TYPE gitlab_workhorse_http_request_duration_seconds histogram
gitlab_workhorse_http_request_duration_seconds_bucket{code="502",method="get",route="",le="0.01"} 5
gitlab_workhorse_http_request_duration_seconds_sum{code="502",method="get",route=""} 0.009138354999999999
gitlab_workhorse_http_request_duration_seconds_count{code="502",method="get",route=""} 5

# HELP gitlab_workhorse_http_request_size_bytes A histogram of sizes of requests to the registry.
# TYPE gitlab_workhorse_http_request_size_bytes histogram
gitlab_workhorse_http_request_size_bytes_bucket{code="502",method="get",route="",le="10"} 0
gitlab_workhorse_http_request_size_bytes_sum{code="502",method="get",route=""} 405
gitlab_workhorse_http_request_size_bytes_count{code="502",method="get",route=""} 5


# HELP gitlab_workhorse_http_requests_total A counter for requests to the registry.
# TYPE gitlab_workhorse_http_requests_total counter
gitlab_workhorse_http_requests_total{code="502",method="get",route=""} 5

# HELP gitlab_workhorse_http_time_to_write_header_seconds A histogram of request durations until the response headers are written.
# TYPE gitlab_workhorse_http_time_to_write_header_seconds histogram
gitlab_workhorse_http_time_to_write_header_seconds_bucket{code="502",method="get",route="",le="0.01"} 5
gitlab_workhorse_http_time_to_write_header_seconds_sum{code="502",method="get",route=""} 0.009059493
gitlab_workhorse_http_time_to_write_header_seconds_count{code="502",method="get",route=""} 5

Bucket sizes still need to be tuned.

Edited by Andrew Newdigate

Merge request reports