Reevaluate NGINX Ingress service upstream annotation

Summary

The Webservice chart configures it's Ingress with the nginx.ingress.kubernetes.io/service-upstream=true by default. This was introduced in Set nginx.ingress.kubernetes.io/service-upstrea... (#2406 - closed) expecting to help with load balancing and to avoid reloads when the list of active Pod changes:

# Hands off traffic directly to the Kubernetes Service. Helps with
# balancing traffic more easily, and does not require NGINX to keep
# a list of active hosts and reload config each time a pod goes up/down.

https://gitlab.com/gitlab-org/charts/gitlab/-/blob/e8f661b356e22caf8bfe1b77bf92803e3b8add36/examples/ref/10k.yaml#L38

NGINX Ingress now dynamically maintains the list of active endpoints with lua.

https://github.com/kubernetes/ingress-nginx/blob/f820c51858ac98314ca535e091211ed13f1d5b62/rootfs/etc/nginx/template/nginx.tmpl#L519

Relates https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1773+

Questions to answer

  • Do we still need the nginx.ingress.kubernetes.io/service-upstream annotation for Webservice deployments?
  • Should we toggle the default.?