Real client IP address is lost in EKS
Summary
The nginx ingress controller is capturing the load balancer IP address rather than the real client IP in EKS.
I started investigating this as part of gitlab-org/gitlab#218457 (closed), and landed here when running in to a number of issues.
I doubt this issue is explicitly in the helm chart, but this seems to be the best spot to start with it (the fix may be sample configs for each cloud, correcting my mistake).
Steps to reproduce
Install the helm chart using the default setup on AWS. The log file for nginx displays the load balancer IP address rather than the actual client.
I tried adding the following to the chart install, and that fixed http/s connections from outside the VPC, but SSH stopped working.
nginx-ingress:
controller:
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
config:
proxy-real-ip-cidr: 10.2.0.0/16 # The VPC CIDR
use-proxy-protocol: "true"
I've also tried without the PROXY protocol annotation and without the use-proxy-protocol line, and while services worked, the logs still contained the incorrect IP address because the ELB defaults to forwarding over TCP, so the X-Forwarded-For headers aren't added.
Current behavior
All log lines show 10.2.0.0/16 addresses (mostly the load balancer, but some internal clients like runners are in there)
Expected behavior
The real client IP address to be logged and reported to components that use it.
Versions
- Chart: 4.0.4
- Platform:
- Cloud: EKS
- Kubernetes:
- Client: 1.18..3
- Server: 1.16.8-eks-e16311
- Helm:
- Client: 3.2.2
- Server: N/A
Relevant logs
10.2.1.17 - [10.2.1.17] - - [07/Jun/2020:07:14:37 +0000] "GET /api/v4/application/statistics HTTP/2.0" 200 181 "https://code.wm.edu/admin" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36" 2153 0.096 [gitlab-gitlab-prod-webservice-8181] 10.2.9.69:8181 181 0.096 200 0898eccb06e0e337fde424f1b852186e