Skip to content

Draft: Updates to correct AWS NLB usage

Gerard Hickey requested to merge 2700-fix-nlb into master

Summary

There seems to be some issues with the example config here: https://gitlab.com/gitlab-org/charts/gitlab/-/blob/v4.10.0/examples/aws/elb-layer4-loadbalancer.yaml

  1. This config creates a classic load balancer, not NLB as stated.
  2. If we enable proxy protocol using the annotation: service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" , it applies to all the elb backends, not just SSH, causing the HTTP traffic (TCP 80, 443) to fail with 400 bad request errors. Doc: AWS Loadbalancers

Workaround:

To serve both HTTP(s) and SSH through the classic ELB,

$ aws elb create-load-balancer-policy --load-balancer-name <lb-name> --policy-name proxy-protocol --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
$ aws elb set-load-balancer-policies-for-backend-server --load-balancer-name <lb-name> --instance-port <instance-port> --policy-names proxy-protocol

Please note that the <instance-port> above is not 22, it's the Nodeport(Instance port) mapped to the SSH service on the Loadbalancer. You'll get this from the Loadbalancer listener page.

Potential permanent fixes:

  • Find the reason why enabling proxy protocol(service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" ) on port 80 and 443 breaks web services (404 bad request) and fix it from the application side.
  • Wait until aws-load-balancer-proxy-protocol supports customisation of backend services. Then turn on proxy protocol for just SSH via annotations.
  • Use separate service/ingress for SSH service and enable proxy protocol just for that through service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" .

Steps to reproduce

Install GitLab using Helm, using the following config. Only custom values are given.

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

global:
  shell:
    # Configure GitLab Shell's NGINX TCP to expect ProxyProtcol headers (and strip them)
    tcp:
      proxyProtocol: true
nginx-ingress:
  controller:
    service:
      annotations:
        # Configure NLB, Layer 4 with 'tcp'
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
        # Configure NLB to use ProxyProtocol
        service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" 
      config:
        proxy-real-ip-cidr: 10.2.0.0/16  # The VPC CIDR, to be trusted
        use-proxy-protocol: "true"       # Tell NGINX to Expect ProxyProtocol headers

I have also tried flipping TLS off and on for global.ingress with various settings such as cert-manager, own self-signed certificates etc. But, nothing changed the error behaviour. It appeared as if proxy protocol was creating issues with web services.

Current behavior

If we use the recommended settings for Layer 4 load balancing, web services won't work.

Expected behavior

Web services + SSH should work when Layer 4 settings are used.

Versions

  • Chart: 4.10.3
  • Platform:
    • Cloud: EKS
  • Kubernetes: (kubectl version)
    • Client: v1.16.6-beta.0
    • Server: v1.18.9-eks-d1db3c
  • Helm: (helm version)
    • Client: v3.1.2
    • Server:

Relevant logs

Was getting 400 bad request error on UI. Ingress logs showed 400 return code for PROXY TCP4 requests.

What does this MR do?

Related issues

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • Merge Request Title and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
  • trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Tests added
  • Integration tests added to GitLab QA
  • Equivalent MR/issue for omnibus-gitlab opened

Closes #2700 (closed)

Merge request reports