Duplicate metrics from application prometheus exporter
Hitting the metrics endpoint on the application (curl localhost:8080/-/metrics
) yields duplicate time series with identical labels, but with different values:
http_request_duration_seconds_count{method="get",status="200"} 0
http_request_duration_seconds_count{method="get",status="204"} 0
http_request_duration_seconds_count{method="get",status="301"} 0
http_request_duration_seconds_count{method="get",status="302"} 0
http_request_duration_seconds_count{method="get",status="303"} 0
http_request_duration_seconds_count{method="get",status="304"} 0
http_request_duration_seconds_count{method="get",status="307"} 0
http_request_duration_seconds_count{method="get",status="400"} 0
http_request_duration_seconds_count{method="get",status="401"} 0
http_request_duration_seconds_count{method="get",status="403"} 0
http_request_duration_seconds_count{method="get",status="404"} 0
http_request_duration_seconds_count{method="get",status="410"} 0
http_request_duration_seconds_count{method="get",status="412"} 0
http_request_duration_seconds_count{method="get",status="422"} 0
http_request_duration_seconds_count{method="get",status="429"} 0
http_request_duration_seconds_count{method="get",status="500"} 0
http_request_duration_seconds_count{method="get",status="503"} 0
http_request_duration_seconds_count{method="get",status="200"} 5470994
http_request_duration_seconds_count{method="get",status="204"} 1977
http_request_duration_seconds_count{method="get",status="301"} 180959
http_request_duration_seconds_count{method="get",status="302"} 401805
http_request_duration_seconds_count{method="get",status="303"} 67
http_request_duration_seconds_count{method="get",status="304"} 6666464
http_request_duration_seconds_count{method="get",status="307"} 1
http_request_duration_seconds_count{method="get",status="400"} 570
http_request_duration_seconds_count{method="get",status="401"} 58161
http_request_duration_seconds_count{method="get",status="403"} 133958
http_request_duration_seconds_count{method="get",status="404"} 26841
http_request_duration_seconds_count{method="get",status="405"} 6
http_request_duration_seconds_count{method="get",status="422"} 309
http_request_duration_seconds_count{method="get",status="429"} 36261
http_request_duration_seconds_count{method="get",status="500"} 18813
http_request_duration_seconds_count{method="get",status="503"} 303
On gitlab.com, we suspect this is causing many application metrics to be falsely read as 0. I'm not sure how long this has been going on.
cc @bjk-gitlab