Can't configure AWS NLB with EKS / helm chart and load UI

Summary

I'm attempting to get gitlab installed on EKS (eks.2 / kubernetes 1.23) using the helm chart (6.4.3 / 15.4.3), aws-load-balancer-controller (1.4.5 / 2.4.4). Load balancer gets created using an ACM certificate but continue to get varying degrees of 400 Bad Request The plain HTTP request was sent to HTTPS port, nginx or just 404 Not Found nginx when attempting to access the load balancer hostname, or even setting up my hosts file using the load balancer ip addresses to point to the eventual permanent hostname until I can verify it works and then update the DNS records accordingly.

Have tried various things listed below in addition to looking at the following issues and attempting suggested settings:

#2700 (closed) #2343 (closed)

Steps to reproduce

helm chart install command:

helm upgrade --install gitlab gitlab/gitlab -n gitlab \
  --set global.edition=ce \
  --set global.hosts.https=false \
  --set global.ingress.configureCertmanager=false \
  --set global.ingress.enabled=false \
  --set global.ingress.tls.enabled=false \
  --set global.hosts.domain=eks.xxx.xxx.xxx.net \
  --set certmanager-issuer.email=xxx@xxx.com

Current behavior

Getting 404 not found, or 400 Bad Request The plain HTTP request was sent to HTTPS port

Expected behavior

Load https://gitlab.xxx.xxx.xxx.xxx.net (setting hosts file to load balancer ip's, pointed to gitlab.xxx.xxx.xxx.xxx.net)

Versions

Chart: 6.4.3 / 15.4.3
Platform:
- Cloud: eks.2 / kubernetes 1.23
Kubernetes: (kubectl version) Client Version: v1.25.0 Kustomize Version: v4.5.7 Server Version: v1.23.10-eks-15b7512
Helm: (helm version) version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.19.2"}

Relevant logs

Service logs just show the same 400 errors I get in the browser, all other activity (presumably health checks) show 200's.

Various attempted annotations added to svc/gitlab-nginx-ingress-controller

    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-name: gitlab-ingress
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
    service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip

404 not found

helm upgrade --install gitlab gitlab/gitlab -n gitlab \
  --set global.edition=ce \
  --set global.hosts.https=false \
  --set global.ingress.configureCertmanager=false \
  --set global.ingress.enabled=false \
  --set global.ingress.tls.enabled=false \
  --set global.hosts.domain=eks.xxx.xxx.xxx.net \
  --set certmanager-issuer.email=x@xxx.com \
  --set certmanager.install=false -f ./elb-layer7-loadbalancer.yaml

# Configure the use of AWS ELB LoadBalancer, in Layer 7 mode
#
# !! ONLY for 'aws-load-balancer-backend-protocol: http' !!
#
# - Configures ELB to be layer 7, terminating SSL with ACM
# - Configures NGINX to trust X-Forwarded-* headers from ELB
# - Route incoming HTTP traffic from ELB to port 80 (http) of NGINX
#
# - See https://kubernetes.io/docs/concepts/services-networking/service/#ssl-support-on-aws
# - See https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/guide/service/annotations/
# - See https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers

global:
  ingress:
    # Disable TLS termination on Ingress objects, by NGINX
    tls:
      enabled: false

nginx-ingress:
  controller:
    config:
      # pass the X-Forwarded-* headers directly from the upstream
      use-forwarded-headers: "true"
    service:
      annotations:
        # Layer 7, injects X-Forwarded-* headers
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
        # Configure ACM certifiates
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        service.beta.kubernetes.io/aws-load-balancer-type: external
        # Configure which ports are to terminate SSL.
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
      targetPorts:
        https: http # the ELB will send HTTP to 443

All the things I've tried:

grep -irH "service.beta" *

svc.yaml:    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
svc.yaml:    service.beta.kubernetes.io/aws-load-balancer-type: external
svc.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx

svc10.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
svc10.yaml:    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
svc10.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc10.yaml:    service.beta.kubernetes.io/aws-load-balancer-type: external

svc11.yaml:    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
svc11.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc11.yaml:    service.beta.kubernetes.io/aws-load-balancer-type: external
svc11.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
svc11.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https

svc2.yaml:    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
svc2.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc2.yaml:    service.beta.kubernetes.io/aws-load-balancer-type: external
svc2.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
svc2.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

svc3.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
svc3.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc3.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

svc4.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
svc4.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc4.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
svc4.yaml:    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

svc5.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
svc5.yaml:    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
svc5.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc5.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
svc5.yaml:    service.beta.kubernetes.io/aws-load-balancer-type: external

svc6.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
svc6.yaml:    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
svc6.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc6.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
svc6.yaml:    service.beta.kubernetes.io/aws-load-balancer-type: external
svc6.yaml:    service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
svc7.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
svc7.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc7.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
svc7.yaml:    service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"

svc8.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
svc8.yaml:    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
svc8.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc8.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

svc9.yaml:    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
svc9.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:xxx
svc9.yaml:    service.beta.kubernetes.io/aws-load-balancer-type: external
svc9.yaml:    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
svc9.yaml:    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https