Skip to content

Wrong ingress rules order when using AWS ALB

Summary

When using the GitLab Helm chart on AWS EKS with an ALB (Application Load Balancer), the rules are being set in the wrong order.
AWS ALB rules are checked one-by-one and when found a match it will stop processing other rules down the line.
The more specific the rule is, the higher it should be in the list.

The currently generated ingress for webservice-default results is always hitting the first backend, also if a /admin/sidekiq/* URL is requested.
This results into stylesheets/javascript not being loaded on the Admin -> Background Jobs page: OWRVdHH

Steps to reproduce

  • Setup aws-load-balancer-controller on AWS EKS
  • Install GitLab on AWS EKS via Helm chart
  • Most important Helm values for this (also see config below):
    • Disable nginx-ingress
    • Configure ingress to use alb class (for aws-load-balancer-controller), set path to /*, configure necessary annotations for aws-load-balancer-controller

Configuration used

    certmanager:
      install: false
    gitlab:
      gitaly:
        persistence:
          accessMode: ReadWriteMany
          size: 50Gi
          storageClass: efs
      task-runner:
        backups:
          objectStorage:
            config:
              key: config
              secret: gitlab-backup-s3cmd-config
    gitlab-runner:
      install: false
    global:
      appConfig:
        artifacts:
          bucket: ***
          enabled: true
          proxy_download: true
        backups:
          bucket: ***
          tmpBucket: ***
        lfs:
          bucket: ***
          enabled: true
          proxy_download: true
        object_store:
          connection:
            key: connection
            secret: gitlab-object-storage-s3-connection
          enabled: true
          proxy_download: true
          storage_options:
            server_side_encryption: AES256
        packages:
          bucket: ***
          enabled: true
          proxy_download: true
        pseudonymizer:
          bucket: ***
          connection:
            key: connection
            secret: gitlab-object-storage-s3-connection
        uploads:
          bucket: ***
          enabled: true
          proxy_download: true
      email:
        from: ***
        reply_to: ***
      gitlab:
        license:
          key: license
          secret: gitlab-gitlab-license
      hosts:
        domain: ***
      ingress:
        annotations:
          alb.ingress.kubernetes.io/group.name: gitlab
          alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
          alb.ingress.kubernetes.io/scheme: internet-facing
          alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06
          alb.ingress.kubernetes.io/success-codes: 200-399
          alb.ingress.kubernetes.io/target-type: ip
          alb.ingress.kubernetes.io/wafv2-acl-arn: ***
        class: alb
        configureCertmanager: false
        enabled: true
        path: /*
        tls:
          enabled: false
      initialRootPassword:
        key: password
        secret: gitlab-gitlab-initial-root-password
      minio:
        enabled: false
      platform:
        eksRoleArn: ***
      psql:
        connectTimeout: 30
        database: gitlabhq_production
        host: ***
        password:
          key: password
          secret: gitlab-postgresql-password
          useSecret: true
        port: 5432
        username: gitlab
      redis:
        host: ***
        password:
          enabled: true
          key: password
          secret: gitlab-redis-password
        port: 6379
        scheme: rediss
      registry:
        bucket: ***
      serviceAccount:
        create: false
        enabled: true
        name: gitlab-aws
      shell:
        tcp:
          proxyProtocol: true
      smtp:
        address: ***
        authentication: login
        domain: ***
        enabled: true
        openssl_verify_mode: peer
        password:
          key: password
          secret: gitlab-smtp-password
        port: 587
        starttls_auto: true
        user_name: ***
    nginx-ingress:
      enabled: false
    postgresql:
      install: false
    redis:
      install: false

Current behavior

The current Helm chart generates this ingress spec for webservice-default (path is set to /* by me in Helm values):

spec:
  rules:
  - host: ***
    http:
      paths:
      - backend:
          serviceName: gitlab-webservice-default
          servicePort: 8181
        path: /*
        pathType: ImplementationSpecific
      - backend:
          serviceName: gitlab-webservice-default
          servicePort: 8080
        path: /admin/sidekiq/*
        pathType: ImplementationSpecific

AWS ALB rules: current-rules

Admin - Background Jobs page: OWRVdHH

Expected behavior

The webservice-default ingress spec should actually be like this:

spec:
  rules:
  - host: ***
    http:
      paths:
      - backend:
          serviceName: gitlab-webservice-default
          servicePort: 8080
        path: /admin/sidekiq/*
        pathType: ImplementationSpecific
      - backend:
          serviceName: gitlab-webservice-default
          servicePort: 8181
        path: /*
        pathType: ImplementationSpecific

AWS ALB rules: expected-rules

Admin - Background Jobs page: KKjiaZEGC

Versions

  • Chart: 4.11.3
  • Platform:
    • Cloud: EKS
  • Kubernetes:
    • Client: v1.19.3
    • Server: v1.19.8-eks-96780e
  • Helm: (helm version)
    • Client: v3.5.4

Relevant logs

N/A