Evaluate whether we still need to set `nginx.ingress.kubernetes.io/service-upstream: "true"` for NGINX
For the nginx ingress we set this due to a bug in the controller that would cause problems with lua balancing where traffic would be sent to a subset of pods on reloads, causing them to be overloaded. At the time, we didn't see any reason not to connect directly to the service but now that we have the new version of the ingress, we may want to revisit this for API.
Reference: #1358 (comment 459061429)
Investigate/Test
- Currently this option is enabled - meaning we'll send traffic to the service endpoint
- We should investigate if the behavior has change with the latest nginx controller that we are running
- Send traffic through the ingress during a deployment
- Validate we do/do not see HTTP50{2|3}'s
- Consider changing/removing this configuration and retest
cc @skarbek
Edited by John Skarbek