API requests are currently queuing for up to 5s
Looks like we're currently experiencing unicorn worker saturation on the API fleet.
Requests are taking up to 3s to get a unicorn.
From ELK: https://log.gitlab.net/goto/00fc02c4687dd647370664ad6f83c1f4
From Grafana: https://dashboards.gitlab.net/d/api-main/api-overview?orgId=1&from=now-7d&to=now&fullscreen&panelId=34
What's interesting is that single_node_unicorn_workers
is being saturated, see below:
However, the workers
metric is far from saturated.
This might indicated that requests are not being spread evenly through the API fleet, with some nodes queuing up while others remain free.
Edited by Andrew Newdigate