Draft: Updates to correct AWS NLB usage
Summary
There seems to be some issues with the example config here: https://gitlab.com/gitlab-org/charts/gitlab/-/blob/v4.10.0/examples/aws/elb-layer4-loadbalancer.yaml
- This config creates a classic load balancer, not NLB as stated.
- If we enable proxy protocol using the annotation:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
, it applies to all the elb backends, not just SSH, causing the HTTP traffic (TCP 80, 443) to fail with400 bad request
errors. Doc: AWS Loadbalancers
Workaround:
To serve both HTTP(s) and SSH through the classic ELB,
- Remove all annotations from
nginx-ingress.controller.service
. It creates all listeners with proxy protocol disabled. - Create a proxy protocol policy just for SSH service at AWS following the instructions: Configure proxy protocol support for your Classic Load Balancer. Example commands:
$ aws elb create-load-balancer-policy --load-balancer-name <lb-name> --policy-name proxy-protocol --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
$ aws elb set-load-balancer-policies-for-backend-server --load-balancer-name <lb-name> --instance-port <instance-port> --policy-names proxy-protocol
Please note that the <instance-port>
above is not 22, it's the Nodeport(Instance port) mapped to the SSH service on the Loadbalancer. You'll get this from the Loadbalancer listener page.
Potential permanent fixes:
- Find the reason why enabling proxy protocol(
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
) on port 80 and 443 breaks web services (404 bad request
) and fix it from the application side. - Wait until
aws-load-balancer-proxy-protocol
supports customisation of backend services. Then turn on proxy protocol for just SSH via annotations. - Use separate service/ingress for SSH service and enable proxy protocol just for that through
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
.
Steps to reproduce
Install GitLab using Helm, using the following config. Only custom values are given.
Configuration used
(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))
global:
shell:
# Configure GitLab Shell's NGINX TCP to expect ProxyProtcol headers (and strip them)
tcp:
proxyProtocol: true
nginx-ingress:
controller:
service:
annotations:
# Configure NLB, Layer 4 with 'tcp'
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
# Configure NLB to use ProxyProtocol
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
config:
proxy-real-ip-cidr: 10.2.0.0/16 # The VPC CIDR, to be trusted
use-proxy-protocol: "true" # Tell NGINX to Expect ProxyProtocol headers
I have also tried flipping TLS off and on for global.ingress
with various settings such as cert-manager, own self-signed certificates etc. But, nothing changed the error behaviour. It appeared as if proxy protocol was creating issues with web services.
Current behavior
If we use the recommended settings for Layer 4 load balancing, web services won't work.
Expected behavior
Web services + SSH should work when Layer 4 settings are used.
Versions
- Chart: 4.10.3
- Platform:
- Cloud: EKS
- Kubernetes: (
kubectl version
)- Client: v1.16.6-beta.0
- Server: v1.18.9-eks-d1db3c
- Helm: (
helm version
)- Client: v3.1.2
- Server:
Relevant logs
Was getting 400 bad request
error on UI. Ingress logs showed 400 return code for PROXY TCP4
requests.
What does this MR do?
Related issues
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion.
Required
-
Merge Request Title and Description are up to date, accurate, and descriptive -
MR targeting the appropriate branch -
MR has a green pipeline on GitLab.com -
Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks -
trigger-package has a green pipeline running against latest commit
Expected (please provide an explanation if not completing)
-
Test plan indicating conditions for success has been posted and passes -
Documentation created/updated -
Tests added -
Integration tests added to GitLab QA -
Equivalent MR/issue for omnibus-gitlab opened
Closes #2700 (closed)