Gitlab not working with External NLB

Summary

I am unable to use a dedicated ingress-nginx to setup AWS NLB as the LB for gitlab with SSL re-direction from NGINX.

I was however able to use the shared nginx-ingress by simply having this config for global.ingress.annotations set to

external-dns.alpha.kubernetes.io/target: ingress-class.domain.com

However, I intend to use a dedicated NLB for gitlab for the following reasons:

• I need SSH port exposed for git alone
• git.domain.com cannot be host-forwarded via ingress for SSH based connections with a shared ingress provider

Steps to reproduce

• helm install gitlab/gitlab -f ./values.yaml

Configuration used

values.yaml

global:
  hosts:
    https: false
    domain: domain.com
    gitlab:
      name: git.domain.com
    registry:
      name: registry.domain.com
  ingress:
    configureCertmanager: false
    annotations:
      external-dns.alpha.kubernetes.io/target: git.domain.com
    tls:
      enabled: false
  psql:
    host: infra-gitlab-rds.domain.com
    password:
      key: gitlabPassword
      secret: gitlab-rds-secret
  minio:
    enabled: false
  grafana:
    enabled: false
  appConfig:
    artifacts:
      bucket: gitlab-artifacts
    lfs:
      bucket: gitlab-lfs
    packages:
      bucket: gitlab-packages
    uploads:
      bucket: gitlab-uploads
    externalDiffs:
      bucket: gitlab-external-diffs
    terraformState:
      bucket: gitlab-terraform-state
    dependencyProxy:
      bucket: gitlab-dependency-proxy
  object_store:
    enabled: true
    proxy_download: true
    connection:
      secret: gitlab-object-store
      key: connection.yaml
    storage_options:
      server_side_encryption: aws:kms
      server_side_encryption_kms_key_id: "XXXXXXXXXXXXXXXXXX"
  serviceAccount:
    enabled: true
    create: true
    annotations:
      eks.amazonaws.com/role-arn: XXXXXXXXXXXXXXXXXX

certmanager:
  createCustomResource: false
  install: false

nginx-ingress:
  controller:
    scope:
      enabled: true
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - gitlab-nginx
              - key: app.kubernetes.io/instance
                operator: In
                values:
                - gitlab-nginx
              - key: app.kubernetes.io/component
                operator: In
                values:
                - controller
            topologyKey: kubernetes.io/hostname
    priorityClassName: "infra-apps"
    service:
      enableHttps: true
      targetPorts:
        https: http
      annotations:
        external-dns.alpha.kubernetes.io/hostname: git.domain.com
        service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Name=gitlab-nginx,class=nginx
        service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
        service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: XXXXXXXXXXXXXXXXXX
        service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: XXXXXXXXXXXXXXXXXX
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    config:
      use-forwarded-headers: "true"
      client-header-timeout: "420"
      proxy-stream-timeout: "200s"
      client-body-timeout: "420"
    headers:
      X-Forwarded-Ssl: "on"
    stats:
      enabled: true
    metrics:
      enabled: true
  podSecurityPolicy:
    enabled: true
  serviceAccount:
    create: true

prometheus:
  install: false

postgresql:
  install: false

gitlab-runner:
  install: false

Current behavior

all requests to http://git.domain.com as well as https://git.domain.com are getting following reply:

curl: (52) Empty reply from server

this seems to be the case when an http request is being offered to the server expecting https request

Expected behavior

• http redirection to https://

curl -IL http://git.domain.com
HTTP/1.1 308 Permanent Redirect
Server: nginx/1.19.2
Date: Thu, 01 Oct 2020 06:53:30 GMT
Content-Type: text/html
Content-Length: 171
Connection: keep-alive
Location: https://git.domain.com/

HTTP/1.1 302 Found
Server: nginx/1.19.2
Date: Thu, 01 Oct 2020 06:53:30 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Cache-Control: no-cache
Content-Security-Policy-Report-Only: default-src 'self'; frame-src 'self' https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com; img-src * data: blob:; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://www.recaptcha.net https://apis.google.com 'nonce-qLVrKVNr+2apfr4GKB5DSg=='; style-src 'self' 'unsafe-inline'
Location: http://git.domain.com/users/sign_in
Referrer-Policy: strict-origin-when-cross-origin
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: DENY
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: PNRS3f83Lc2
X-Runtime: 0.066135
X-Ua-Compatible: IE=edge
X-Xss-Protection: 1; mode=block

HTTP/1.1 308 Permanent Redirect
Server: nginx/1.19.2
Date: Thu, 01 Oct 2020 06:53:30 GMT
Content-Type: text/html
Content-Length: 171
Connection: keep-alive
Location: https://git.domain.com/users/sign_in

HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Thu, 01 Oct 2020 06:53:31 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Vary: Accept-Encoding
Cache-Control: max-age=0, private, must-revalidate
Content-Security-Policy-Report-Only: default-src 'self'; frame-src 'self' https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com; img-src * data: blob:; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://www.recaptcha.net https://apis.google.com 'nonce-YmuidOJcCPaWQjrl3t0TSQ=='; style-src 'self' 'unsafe-inline'
Etag: W/"677139820c392ced1b52beb703903448"
Referrer-Policy: strict-origin-when-cross-origin
Set-Cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IkltUm1OelExTkdZMkxXVmtaVGd0TkRRMFlpMWlNR0V6TFdZd01qRTVNekUwWlRNeE1TST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--d7d168a99b84a564898af05fd50ec894aea007d1; path=/; expires=Mon, 01 Oct 2040 06:53:31 -0000; HttpOnly
Set-Cookie: _gitlab_session=e9bcaf212e959de36470c8f6b4b24847; path=/; expires=Thu, 01 Oct 2020 08:53:31 -0000; HttpOnly
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: DENY
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: SbNxo86mpk6
X-Runtime: 0.324777
X-Ua-Compatible: IE=edge
X-Xss-Protection: 1; mode=block

Versions

• Chart: gitlab/gitlab@4.4.1
• Platform:
  • Cloud: EKS
• Kubernetes: (kubectl version)
  • Client: v1.18.4
  • Server: v1.16.13-eks-2ba888
• Helm: (helm version)
  • Client: v3.2.4
  • Server: N/A

Relevant logs

curl -IL https://git.domain.com

curl: (52) Empty reply from server