Increase runner concurrency limit
Overview
Saturation point: ci-runners/shared_runners saturation
We are hitting the saturation limit on our shared runner managers due to exceeding the current concurrency limit of 800
.
This issue is to discuss whether we can raise that limit, or whether we should add at more VMs to the shared runner fleet.
corrective action for production#4473 (closed)
Timeline
Wednesday (2021-05-05)
-
Resize srm7
instance to match the CPU as other runner managers👉 production#4349 (closed) / https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12574 -
Upgrade docker-machine
to usewait
👉 production#4484 (closed) -
Increase srm7
concurrent to900
👉 production#4491 (closed)
Thursday (2021-05-06)
-
Divert srm6
togitlab-ci-plan-free-6
(also migrate towait
by upgradingdocker-machine
)👉 production#4432 (closed) -
Increase concurrent up to 900
by end of the day (we need to slowly increase the concurrent given we migrated to a new project)
Monday (2021-05-10)
-
Divert srm5
togitlab-ci-plan-free-5
(also migrate towait
by upgradingdocker-machine
)👉 production#4545 (closed) -
Increase concurrent up to 1100
forsrm5
by end of the day (we need to slowly increase the concurrent given we migrated to a new project)👉 production#4545 (closed) -
Divert srm4
togitlab-ci-plan-free-4
(also migrate towait
by upgradingdocker-machine
)👉 production#4546 (closed) -
Increase concurrent up to 1100
forsrm4
by end of the day (we need to slowly increase the concurrent given we migrated to a new project)👉 production#4546 (closed) -
Increase concurrent
andlimit
to1100
for srm6👉 production#4547 (closed) -
Increase concurrent
andlimit
to1100
for srm7👉 production#4547 (closed)
Tuesday (2021-05-11)
-
Divert srm3
togitlab-ci-plan-free-3
(also migrate towait
by upgradingdocker-machine
)👉 production#4548 (closed) -
Increase concurrent up to 1100
forsrm3
by end of the day (we need to slowly increase the concurrent given we migrated to a new project)👉 production#4548 (closed)
Wednesday (2021-05-12)
-
Increase concurrent
to1200
👉 production#4581 (closed) -
Increase limit
to a higher number thanconcurrent
to have Idle machines at concurrent. We should have100
Idle machines when at capacity. This depends don't reach quota limits during capacity👉 production#4570 (closed)
Edited by Darren Eastman