Add more request_concurrency related metrics
What does this MR do?
Adds metrics providing details about the adaptive request concurrency mechanism introduced in Add adaptive request concurrency (!5546 - merged).
This MR adds three new gauge metrics:
# HELP gitlab_runner_request_concurrency_hard_limit Configured request_concurrency limit
# TYPE gitlab_runner_request_concurrency_hard_limit gauge
gitlab_runner_request_concurrency_hard_limit{runner="9_F4bzrV3",system_id="r_HKflXh8gaYx3"} 10
# HELP gitlab_runner_request_concurrency_adaptive_limit Computed adaptive request concurrency limit
# TYPE gitlab_runner_request_concurrency_adaptive_limit gauge
gitlab_runner_request_concurrency_adaptive_limit{runner="9_F4bzrV3",system_id="r_HKflXh8gaYx3"} 1
# HELP gitlab_runner_request_concurrency_used_limit Used request concurrency limit
# TYPE gitlab_runner_request_concurrency_used_limit gauge
gitlab_runner_request_concurrency_used_limit{runner="9_F4bzrV3",system_id="r_HKflXh8gaYx3"} 1gitlab_runner_request_concurrency_hard_limit shows the hard limit, defined by the requests_concurrency setting in config.toml. Request concurrency may never exceed this.
gitlab_runner_request_concurrency_adaptive_limit shows the currently calculated value of the adaptive limit. It's calculated as something between 1 and the value shown by the hard limit. If new jobs are incoming, the value is increased from 1 by the factor of 10%, each time a new job was properly taken. Until it reaches the hard limit. If there are request errors or new jobs are not being detected, the value is decreased by the factor of 5%, until it reaches 1.
gitlab_runner_request_concurrency_used_limit represents the value of limit chosen to be used when making the decision. It's either the adaptive limit (rounded up, as requests number is an integer) or the hard limit - depending which one is smaller at the moment.
Together with existing metrics: gitlab_runner_request_concurrency (describing current number of concurrent requests for jobs being made) and gitlab_runner_request_concurrency_exceeded_total (a counter increased each time Runner detected that a request would exceed the limit and prevented it) they can show the image of what's happening with requesting new jobs.
All these metrics are partitioned by the runner_id and system_id values.
Why was this MR needed?
To make sure that GitLab Runner operators are able to fully understand how Runner's requests for jobs are being handled and how to adapt the configuration (if needed).