Discussion: Standardise GOMAXPROCS across all Go services
Background
There is a mismatch between cgroup quota and Go's default GOMAXPROCS selection. GOMAXPROCS does not take into account the limits, it instead looks at the number of cores available on the box.
Problem
This can result in it overcommitting, which makes any individual process more susceptible to being throttled.
Workaround
We can work around this by manually setting GOMAXPROCS
to match the cgroup quota. Examples:
- gitlab-com/gl-infra/k8s-workloads/gitlab-com!2623 (merged)
- gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles!341 (merged)
However, there are several issues with this:
- It's inconsistent: Only a small subset of services have it set. We need a safer default.
- It's prone to drift: When quota changes, it is likely that we will forget to adjust the GOMAXPROCS value.
Proposal
I think we should consider standardising on an automated approach.
One such approach is the uber-go/automaxprocs library.