http_request_duration_seconds_count metrics should be initialized to zero at startup
The http_request_duration_seconds_count
has a label for status.
Each time a request returns a HTTP 500
status, the http_request_duration_seconds_count{status="500"}
is incremented etc.
Until the first instance of a particular status code is recorded, after process startup, the series for that status code does not exist.
This is fine, except that it means we can't tell when the metric is missing because of a configuration error, and when the metric is missing because there have not been any errors. This leads to us generating alerts about missing metrics, such as:
Another example:
Instead of lazy initializing these counters, we should eager initialize them at startup time, for a set of common HTTP status codes (200, 301, 304, 400, 401, 403, 404, 500 at a minimum).
cc @bjk