Scaling GKE cluster used by CI
The current amount of active, parallel development happening has caused us to start hitting the limits of schedulable resources on our CI's GKE cluster again. This increased load, combined with the sane resource defaults we've put into place has resulted in failed deployments with the state of "unscheduable".
@WarheadsSE did manual cleanup within helm
and kubectl
to remove stale items. This cleared up much of the contention, but we remain unable to deploy all current WIP branches and MRs.
Do we want to manually scale this as we did in the past, or should we instead rely on the GKE autoscaler, but keep an eye out for excessive growth?
Edited by Jason Plum