Skip to content

Update NLB example for AWS Load Balancer Controller

Jason Young requested to merge 2700-l4-lb-recommended-settings into master

What does this MR do?

This updates the AWS NLB example with a re-tested configuration and makes use of the recommended AWS Load Balancer Controller

Implementation Notes

I'm setting the service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "80" service annotation intentionally. When left to the default (target port) - health checks will use the port (443 or 80 depending on TLS termination, 80 for http, and 22 for ssh) for the target groups. Unfortunately this results in a cascade of

broken header: "" while reading PROXY protocol, client: [Load Balancer IP], server: 0.0.0.0:22

errors for every health check due to the fact that:

After you enable proxy protocol, the proxy protocol header is also included in health check connections from the load balancer. However, with health check connections, the client connection information is not sent in the proxy protocol header.

via: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#proxy-protocol

The rationale to changing them all to 80 is that the healthcheck is for the gitlab-nginx-ingress-controller (which itself will check upstream pods) - if one port is healthy on the gitlab-nginx-ingress-controller pod, they realistically all should be - and this prevents the logs from being flooded with broken header errors.

Related issues

Closes #2700 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • Merge Request Title and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • [ ] MR has a green pipeline on GitLab.com
  • When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • [ ] Tests added
  • [ ] Integration tests added to GitLab QA
  • [ ] Equivalent MR/issue for omnibus-gitlab opened
  • Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.

Test Plan

  1. Deploy an EKS Cluster - I used our bootstrap script
  2. Deploy the AWS Load Balancer Controller: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/deploy/installation/
  3. Deploy with the recommended values (the default items assume certmanager and Let's Encrypt, with TLS passthrough)
  4. The NLB can be deployed with a TLS listener and terminate TLS at the LB (optionally using an AWS ACM certificate) - see the comments in the example

I tested SSH, GitLab API, GitLab UI, and GitLab KAS at length.

Note: As far as I can tell, the NLB does not support TLS termination at the LB and then using TLS between it and the gitlab-nginx-ingress-controller - there's not a definitive guide to that (that I could find) - but it's implied from https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/use_cases/nlb_tls_termination/ [Edit: Likely inaccurate, pending additional testing]

Edited by Jason Young

Merge request reports