Investigation: Consider changes to NGINX Ingress

During #1922 (closed) we discovered that when new nodes come online to allow for scheduling new NGINX Ingress Controller Pods, the traffic was balanced between the nodes, not the Pods. This had a negative side effect that Pods were seeing a large imbalance of traffic.

Consider/Investigate the following changes to determine if we want/need to adjust this:

1. Consider changing the externalTrafficPolicy: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
2. Changing the HPA to the beta version which supports scaling policies, and consider ensuring we scale up a certain number of Pods: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#scaling-policies - though this is not fool proof, because we won't have the ability to say, put at least X number of Pods on a given node
3. Consider changing to the GCE Ingress: https://github.com/kubernetes/ingress-gce

Things to keep in mind:

1. We may rid of the nginx ingress entirely: #1924 (closed) - if so, this issue is not useful and can be closed
2. We need to ensure we do not lose our remote_ip in our logging and applications - these are used for auditing and application rate limiting practices
3. Consider the performance impact - we may be adding an extra network hop which might be detrimental depending on the performance of the node receiving and needing to forward said traffic to an appropriate Pod (instance types can influence available network bandwidth)
4. ...