Explore switching nginx to being a daemonset on Webservice nodes for Cloud Native Hybrids
In a recent test against a scaled 1000 RPS / 50k user Cloud Native Hybrid RA, the standard 2 nginx pods became a bottleneck.
In heavy circumstances the 2 pods deployed in the Supporting node pool could indeed become a bottleneck compared to Linux package deployments where there is an nginx process deployed on each Rails node. There's also the challenge that, depending on the cloud provider, network bandwidth can be limited at times for the smaller machines more common for the supporting pool such as AWS and it's network credits so increasing the pod count alone may not be sufficient long term.
Switching to be closer to the Linux package approach then could be a good solution to all of the above by deploying nginx pods on each Webservice node via a daemonset
. Each webservice node pool as recommended has just under 4 vCPU free at minimum to more than accommodate the pods due to the spread of webservice pods and they typically will not fall victim to network limiting due to them being bigger machines. Scaling will naturally increase nicely with up the RA sizings as well since the Webservice nodes are increased.
Task is to explore, test and proceed if viable.