Draw 409 and 5xx responses for /api/v4/jobs/request calls on a dedicated panel
In our main CI dashboard we have this panel: https://dashboards.gitlab.net/d/000000159/ci?viewPanel=91&orgId=1. It shows all the responses that our Runner Managers are getting when sending POST /api/v4/jobs/request
requests.
As noted in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12752, a high amount of 409 responses may be a sign that something wrong is happening and any appearance of 5xx errors is a sign that something is wrong. Another issue tracks the idea of adding alerting on the 5xx. But we should extract the same information as in the panel listed above, but limited to only 409 and 5xx errors, and print it in some visible place. So that in case of another Runners incident it will be easier to recognize that some problems with jobs scheduling are happening on the GitLab side and this is the place where the root cause of the incident is.