Remove ingress-nginx from in front of API

As part of issue #1924 (closed) we were able to determine that the web/api components of Gitlab.com don't actually need ingress-nginx in front of them. This is because the only thing nginx gives us is proxy-buffering for users with slow connections. However, we have determined that cloudflare already buffers requests for us, so there is no value to keeping nginx.

As api service is now the only service that uses ingress-nginx, and it's no longer needed, this issue is to track removing it from in front of api in Gitlab.com

Tasks

• reserve static LB IPs in terraform (https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/3011)
• configure K8s to explicitly make use of the reserved IPs for loadbalancing (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1274 (merged))
• Configure gitlab-com and haproxy to have api not use nginx in pre
• Configure gitlab-com and haproxy to have api not use nginx in gstg
• Configure gitlab-com and haproxy to have api not use nginx in gprd-cny
• Configure gitlab-com and haproxy to have api not use nginx in gprd
• Remove ingress-nginx from being deployed in all environments: #2098 (closed)
• Cleanup runbooks, metrics, and any other references to nginx: #2100 (closed)

Update 2021-11-11

We needed to bring brought back in place because of seeing too many errors for long-running requests.

Investigation issue: #2114 (closed)