Document scaling guidelines on per-chart basis
We should add a section to the documentation we provide for each of our component charts, which provide reasoning and guidance on scaling each component to align to the official reference architectures provided by Gitlab.
Items to include:
- General guidance for converting cpu and memory recommendations from VM to Pod(s).
- How those numbers were reached (such as Puma's process/thread model as opposed to Sidekiq Cluster), and how each component behaves for scaling.
Sidekiq at 10k is 4 VMs with 4vCPU / 15 GB each. This translates to 16 vCPU / 60 GB.
Our current understanding of Sidekiq Pods are somewhat vague, and that is related to the fact that the workload can greatly vary, but the general understand is that should it be broken down to approximately 1 vCPU / 2 GB per process, per pod. This results in a 1C:2G ratio, and a direct translation of 16 Pods. These can be vertically scaled, but aligning to 1:2 ratio. Thus, a 2 vCPU / 4 GB Pod would translate to only 8 Pods.
Webservice (Rails) at 10k is 3 VMs with 32 vCPU / 30 GB each. This translates to 96 vCPU and 90 GB.
Our current understanding of webservice pods, using Puma, should be allocated at < 0.5 vCPU / ~1 GB per process (not thread). By default, our pods are 2 processes, with 4 threads each so, 1 vCPU / 2 GB / pod. That is a direct translation of 32 Pods. These can be vertically scaled, but should never have be less than 1:2. Thus, these could be set to have 4 processes and 8 GB. That would reduce to 16 Pods, but the kuebrnetes nodes in must be able to handle pods of that size.