Add NGINX as an optional sidecar to the webservice pod
Enabling the NGINX ingress controller brings with it some complexity and additional cross-zone ingress/egress that we would be able to avoid by setting up NGINX as a sidecar in the webservice pod. This would ensure that traffic between nginx and workhorse does not cross availability zones, which can be (in the case of GitLab.com) a large source of cloud spend for egress/ingress.
While adding the sidecar might be an easy addition, there are some issues we will need to sort out first:
- we should have an SSOT for nginx config that is shared between the sidecar and ingress controller
- one issue that came up during our readiness review is that the connection between nginx and workhorse is not encrypted, and on VMs we use unix socket. Would it make sense to have a shared socket between the containers?
- are there any downsides to removing the nginx ingress controller?