Adjust Unicorn worker formula

Summary

Adjust the Unicorn worker count formula to increase request handling

Proposal

In larger deployments we are limited by the number of available unicorn workers to process parallel requests. This leads to request queuing.

In gitlab.com production we these ratios[0]:

• Web workers: 36 for 16 CPUs. (2.25:1)
• API workers: 20 for 16 CPUs. (1.25:1)
• Git workers: 25 for 16 CPUs. (1.56:1)

Looking at the total cores in gitlab.com production[1], the over-subscription ratio is ~1.8:1.

With this higher ratio, we still see only about 60% CPU utilization at peak times.

I propose we change the current formula, CPUs + 1, to be int(CPUs * 1.5 + 1). This will keep the current value of 2 workers on a single CPU instance, and give a 16 CPU instance a bump from 17 workers to 25 workers. This should help improve the capacity for users running the default formula.

References

[0]: Unicorn Worker dashboard: https://dashboards.gitlab.com/d/RUOkovumz/unicorn-workers [1]: Thanos query: sum(max without (pid) (unicorn_workers{env="gprd",type=~"web|api|git",stage="main"})) / sum(instance:node_cpus:count{env="gprd",type=~"web|api|git",stage="main"})