Gitlab-Pages returns a "domain does not exist"

Summary

Gitlab-pages pod on the cluster responds to calls with "domain does not exist"

Steps to reproduce

  • I have a network load balancer handling TLS termination, forwarding to the built in NGINX ingress for traffic
  • I enable gitlab-pages, and configure the service to be ClusterIP (I cannot add annotations to this service as I am using chart 15.11, and I do not want a classic load balancer)
  • I add an ingress pointing to the service in step 2, port 80, with DNS entries pointing the additional pages domains to my gitlab domain
  • I visit the pages section of the project, go to the url defined there: https://gitlab-qa.{domain}/general-documentation
  • I receive a 500 Internal Server Error from gitlab
  • The log inside the pages pod states: "error":"domain does not exist","host":"gitlab-qa.{domain}"

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

gitlab:
  gitlab:
    gitlab-pages:
      service:
        customDomains:
          type: ClusterIP
    toolbox:
      persistence:
        accessMode: ReadWriteOnce
        enabled: true
        size: 300Gi
      resources:
        requests:
          cpu: 2000m
          memory: 6G
      serviceAccount:
        annotations:
          eks.amazonaws.com/role-arn: <snip>
        create: true
        enabled: true
        name: gitlab-qa-s3-service-account
  global:
    pages:
      enabled: true
      externalHttp: 
        - <snip>.0.0/16
    appConfig:
      artifacts:
        bucket: <snip>
        enabled: true
      ciSecureFiles:
        connection: {}
        enabled: false
      dependencyProxy:
        enabled: false
      externalDiffs:
        enabled: false
        when: null
      lfs:
        enabled: false
      object_store:
        connection:
          key: connection
          provider: AWS
          region: eu-west-1
          secret: gitlab-secrets
        enabled: true
        proxy_download: true
        storage_options:
          server_side_encryption: AES256
      packages:
        enabled: false
      terraformState:
        connection: {}
        enabled: false
      uploads:
        bucket: <snip>
        enabled: true
    email:
      display_name: GitLab
      from: gitlab-qa@<snip>
      reply_to: noreply@<snip>
    hosts:
      gitlab:
        name: git-qa.<snip>
      https: true
      ssh: git-qa.<snip>
      pages:
        name: <snip>
        https: true
    ingress:
      configureCertmanager: false
      enabled: true
      tls:
        enabled: false
    psql:
      database: gitlabhq_production
      host: <snip>
      password:
        key: gitlab-postgres-password
        secret: gitlab-secrets
        useSecret: true
      port: 5432
      username: gitlab
    redis:
      host: <snip>
      password:
        enabled: false
        key: gitlab-redis-password
        secret: gitlab-secrets
      scheme: redis
    smtp:
      address: email-smtp.eu-west-1.amazonaws.com
      authentication: login
      domain: <snip>
      enabled: true
      password:
        key: smtp-password
        secret: gitlab-secrets
      port: 587
      starttls_auto: true
      user_name: <snip>
  nginx-ingress:
    controller:
      service:
        externalTrafficPolicy: Cluster
        enableHttp: false
        targetPorts:
          https: http
        annotations:
          external-dns.alpha.kubernetes.io/hostname: git-qa.<snip>
          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <snip>
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
          service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60'
          service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
          service.beta.kubernetes.io/aws-load-balancer-internal: "true"
          service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false
          service.beta.kubernetes.io/load-balancer-source-ranges: <snip>
  postgresql:
    install: false
  redis:
    install: false

We overlay these values on top of the chart defaults

Current behavior

Calls to gitlab-pages pod fail with "domain does not exist"

Expected behavior

Gitlab-pages to load, same as the linux package

Versions

  • Chart: 6.11.12
  • Platform:
    • Cloud:EKS
  • Kubernetes: (kubectl version)
    • Client: v1.24
    • Server: v1.24
  • Helm: (helm version)
    • Client: v3.13
    • Server: v3.13

Relevant logs

pages logs 
[
  {
    "correlation_id": "4ae500031b47a5276ce0d0252d1142b0",
    "error": "httprange: new resource 403: \"403 Forbidden\"",
    "host": "gitlab-qa.<snip>",
    "level": "error",
    "msg": "vfs.Root",
    "path": "/general-documentation",
    "time": "2023-10-25T12:26:56Z"
  },
  {
    "content_type": "text/html; charset=utf-8",
    "correlation_id": "4ae500031b47a5276ce0d0252d1142b0",
    "duration_ms": 324,
    "host": "gitlab-qa.<snip>",
    "level": "info",
    "method": "GET",
    "msg": "access",
    "pages_https": false,
    "proto": "HTTP/1.1",
    "referrer": "",
    "remote_addr": "<snip>.18.23:42690",
    "remote_ip": "<snip>.18.23",
    "status": 500,
    "system": "http",
    "time": "2023-10-25T12:26:56Z",
    "ttfb_ms": 324,
    "uri": "/general-documentation",
    "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    "written_bytes": 2905
  },
  {
    "correlation_id": "c16e9c18aebe6fece5ed14f6e03226f2",
    "error": "domain does not exist",
    "host": "gitlab-qa.<snip>",
    "level": "error",
    "lookup_paths": [
      {
        "project_id": 3380,
        "prefix": "/general-documentation/",
        "source": {
          "type": "zip",
          "path": "https://gitlab-pages.s3.eu-west-1.amazonaws.com/37/5f/375fc19c529cbd4f1644a417df69451f2dccde778ee3e2ee4214ddd86fddf1cb/pages_deployments/15188/artifacts.zip?X-Amz-Expires=86400&X-Amz-Date=20231025T122656Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUIGCXQURWNZWPYYP%2F20231025%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=f3b93b1f9758f7aff477e60d77cccf9080ab94bbe5b12465995d7d1cd0d08088",
          "sha256": "c5cda48e3926b49d1d034b8a5749e4401c3f6b71536ff699702df57d65157ed2",
          "file_count": 2417,
          "file_size": 48977820
        }
      }
    ],
    "lookup_paths_count": 1,
    "msg": "could not find project lookup path",
    "path": "/favicon.ico",
    "time": "2023-10-25T12:26:57Z"
  },
  {
    "correlation_id": "c16e9c18aebe6fece5ed14f6e03226f2",
    "error": "domain does not exist",
    "host": "gitlab-qa.<snip>",
    "level": "error",
    "lookup_paths": [
      {
        "project_id": 3380,
        "prefix": "/general-documentation/",
        "source": {
          "type": "zip",
          "path": "https://gitlab-pages.s3.eu-west-1.amazonaws.com/37/5f/375fc19c529cbd4f1644a417df69451f2dccde778ee3e2ee4214ddd86fddf1cb/pages_deployments/15188/artifacts.zip?X-Amz-Expires=86400&X-Amz-Date=20231025T122656Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUIGCXQURWNZWPYYP%2F20231025%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=f3b93b1f9758f7aff477e60d77cccf9080ab94bbe5b12465995d7d1cd0d08088",
          "sha256": "c5cda48e3926b49d1d034b8a5749e4401c3f6b71536ff699702df57d65157ed2",
          "file_count": 2417,
          "file_size": 48977820
        }
      }
    ],
    "lookup_paths_count": 1,
    "msg": "could not find project lookup path",
    "path": "/favicon.ico",
    "time": "2023-10-25T12:26:57Z"
  },
  {
    "content_type": "text/html; charset=utf-8",
    "correlation_id": "c16e9c18aebe6fece5ed14f6e03226f2",
    "duration_ms": 0,
    "host": "gitlab-qa.<snip>",
    "level": "info",
    "method": "GET",
    "msg": "access",
    "pages_https": false,
    "proto": "HTTP/1.1",
    "referrer": "https://gitlab-qa.<snip>/general-documentation",
    "remote_addr": "<snip>.18.23:42690",
    "remote_ip": "<snip>.18.23",
    "status": 404,
    "system": "http",
    "time": "2023-10-25T12:26:57Z",
    "ttfb_ms": 0,
    "uri": "/favicon.ico",
    "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    "written_bytes": 3067
  },
  {
    "content_type": "",
    "correlation_id": "01HDKCTVF3J68ENF9G6GY8BK48",
    "duration_ms": 0,
    "host": "<snip>.16.165:8090",
    "level": "info",
    "method": "GET",
    "msg": "access",
    "pages_https": false,
    "proto": "HTTP/1.1",
    "referrer": "",
    "remote_addr": "<snip>.19.149:46452",
    "remote_ip": "<snip>.19.149",
    "status": 200,
    "system": "http",
    "time": "2023-10-25T12:27:00Z",
    "ttfb_ms": 0,
    "uri": "/-/readiness",
    "user_agent": "kube-probe/1.27+",
    "written_bytes": 8
  },
  {
    "correlation_id": "13f784cf8be048ccf156e674d6f87521",
    "error": "httprange: new resource 403: \"403 Forbidden\"",
    "host": "gitlab-qa.<snip>",
    "level": "error",
    "msg": "vfs.Root",
    "path": "/general-documentation",
    "time": "2023-10-25T12:27:03Z"
  },
  {
    "content_type": "text/html; charset=utf-8",
    "correlation_id": "13f784cf8be048ccf156e674d6f87521",
    "duration_ms": 2,
    "host": "gitlab-qa.<snip>",
    "level": "info",
    "method": "GET",
    "msg": "access",
    "pages_https": false,
    "proto": "HTTP/1.1",
    "referrer": "",
    "remote_addr": "<snip>.18.23:44224",
    "remote_ip": "<snip>.18.23",
    "status": 500,
    "system": "http",
    "time": "2023-10-25T12:27:03Z",
    "ttfb_ms": 2,
    "uri": "/general-documentation",
    "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    "written_bytes": 2905
  },
  {
    "correlation_id": "74f9af7aa865e1511f3c5f4599bbb450",
    "error": "domain does not exist",
    "host": "gitlab-qa.<snip>",
    "level": "error",
    "lookup_paths": [
      {
        "project_id": 3380,
        "prefix": "/general-documentation/",
        "source": {
          "type": "zip",
          "path": "https://gitlab-pages.s3.eu-west-1.amazonaws.com/37/5f/375fc19c529cbd4f1644a417df69451f2dccde778ee3e2ee4214ddd86fddf1cb/pages_deployments/15188/artifacts.zip?X-Amz-Expires=86400&X-Amz-Date=20231025T122656Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUIGCXQURWNZWPYYP%2F20231025%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=f3b93b1f9758f7aff477e60d77cccf9080ab94bbe5b12465995d7d1cd0d08088",
          "sha256": "c5cda48e3926b49d1d034b8a5749e4401c3f6b71536ff699702df57d65157ed2",
          "file_count": 2417,
          "file_size": 48977820
        }
      }
    ],
    "lookup_paths_count": 1,
    "msg": "could not find project lookup path",
    "path": "/favicon.ico",
    "time": "2023-10-25T12:27:03Z"
  },
  {
    "correlation_id": "74f9af7aa865e1511f3c5f4599bbb450",
    "error": "domain does not exist",
    "host": "gitlab-qa.<snip>",
    "level": "error",
    "lookup_paths": [
      {
        "project_id": 3380,
        "prefix": "/general-documentation/",
        "source": {
          "type": "zip",
          "path": "https://gitlab-pages.s3.eu-west-1.amazonaws.com/37/5f/375fc19c529cbd4f1644a417df69451f2dccde778ee3e2ee4214ddd86fddf1cb/pages_deployments/15188/artifacts.zip?X-Amz-Expires=86400&X-Amz-Date=20231025T122656Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUIGCXQURWNZWPYYP%2F20231025%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=f3b93b1f9758f7aff477e60d77cccf9080ab94bbe5b12465995d7d1cd0d08088",
          "sha256": "c5cda48e3926b49d1d034b8a5749e4401c3f6b71536ff699702df57d65157ed2",
          "file_count": 2417,
          "file_size": 48977820
        }
      }
    ],
    "lookup_paths_count": 1,
    "msg": "could not find project lookup path",
    "path": "/favicon.ico",
    "time": "2023-10-25T12:27:03Z"
  },
  {
    "content_type": "text/html; charset=utf-8",
    "correlation_id": "74f9af7aa865e1511f3c5f4599bbb450",
    "duration_ms": 0,
    "host": "gitlab-qa.<snip>",
    "level": "info",
    "method": "GET",
    "msg": "access",
    "pages_https": false,
    "proto": "HTTP/1.1",
    "referrer": "https://gitlab-qa.<snip>/general-documentation",
    "remote_addr": "<snip>.18.23:44224",
    "remote_ip": "<snip>.18.23",
    "status": 404,
    "system": "http",
    "time": "2023-10-25T12:27:03Z",
    "ttfb_ms": 0,
    "uri": "/favicon.ico",
    "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    "written_bytes": 3067
  },
  {
    "correlation_id": "0e30a2e94dfe419df9e32f55691dc1d8",
    "error": "httprange: new resource 403: \"403 Forbidden\"",
    "host": "gitlab-qa.<snip>",
    "level": "error",
    "msg": "vfs.Root",
    "path": "/general-documentation",
    "time": "2023-10-25T12:27:09Z"
  },
  {
    "content_type": "text/html; charset=utf-8",
    "correlation_id": "0e30a2e94dfe419df9e32f55691dc1d8",
    "duration_ms": 0,
    "host": "gitlab-qa.<snip>",
    "level": "info",
    "method": "GET",
    "msg": "access",
    "pages_https": false,
    "proto": "HTTP/1.1",
    "referrer": "",
    "remote_addr": "<snip>.18.23:44226",
    "remote_ip": "<snip>.18.23",
    "status": 500,
    "system": "http",
    "time": "2023-10-25T12:27:09Z",
    "ttfb_ms": 0,
    "uri": "/general-documentation",
    "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    "written_bytes": 2905
  },
  {
    "correlation_id": "343089b3c17320ea51a200ba53c5a408",
    "error": "domain does not exist",
    "host": "gitlab-qa.<snip>",
    "level": "error",
    "lookup_paths": [
      {
        "project_id": 3380,
        "prefix": "/general-documentation/",
        "source": {
          "type": "zip",
          "path": "https://gitlab-pages.s3.eu-west-1.amazonaws.com/37/5f/375fc19c529cbd4f1644a417df69451f2dccde778ee3e2ee4214ddd86fddf1cb/pages_deployments/15188/artifacts.zip?X-Amz-Expires=86400&X-Amz-Date=20231025T122656Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUIGCXQURWNZWPYYP%2F20231025%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=f3b93b1f9758f7aff477e60d77cccf9080ab94bbe5b12465995d7d1cd0d08088",
          "sha256": "c5cda48e3926b49d1d034b8a5749e4401c3f6b71536ff699702df57d65157ed2",
          "file_count": 2417,
          "file_size": 48977820
        }
      }
    ],
    "lookup_paths_count": 1,
    "msg": "could not find project lookup path",
    "path": "/favicon.ico",
    "time": "2023-10-25T12:27:09Z"
  },
  {
    "correlation_id": "343089b3c17320ea51a200ba53c5a408",
    "error": "domain does not exist",
    "host": "gitlab-qa.<snip>",
    "level": "error",
    "lookup_paths": [
      {
        "project_id": 3380,
        "prefix": "/general-documentation/",
        "source": {
          "type": "zip",
          "path": "https://gitlab-pages.s3.eu-west-1.amazonaws.com/37/5f/375fc19c529cbd4f1644a417df69451f2dccde778ee3e2ee4214ddd86fddf1cb/pages_deployments/15188/artifacts.zip?X-Amz-Expires=86400&X-Amz-Date=20231025T122656Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAUIGCXQURWNZWPYYP%2F20231025%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=f3b93b1f9758f7aff477e60d77cccf9080ab94bbe5b12465995d7d1cd0d08088",
          "sha256": "c5cda48e3926b49d1d034b8a5749e4401c3f6b71536ff699702df57d65157ed2",
          "file_count": 2417,
          "file_size": 48977820
        }
      }
    ],
    "lookup_paths_count": 1,
    "msg": "could not find project lookup path",
    "path": "/favicon.ico",
    "time": "2023-10-25T12:27:09Z"
  },
  {
    "content_type": "text/html; charset=utf-8",
    "correlation_id": "343089b3c17320ea51a200ba53c5a408",
    "duration_ms": 0,
    "host": "gitlab-qa.<snip>",
    "level": "info",
    "method": "GET",
    "msg": "access",
    "pages_https": false,
    "proto": "HTTP/1.1",
    "referrer": "https://gitlab-qa.<snip>/general-documentation",
    "remote_addr": "<snip>.18.23:44226",
    "remote_ip": "<snip>.18.23",
    "status": 404,
    "system": "http",
    "time": "2023-10-25T12:27:09Z",
    "ttfb_ms": 0,
    "uri": "/favicon.ico",
    "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36",
    "written_bytes": 3067
  },
  {
    "content_type": "",
    "correlation_id": "01HDKCV57KSC2WX979YG3F28VN",
    "duration_ms": 0,
    "host": "<snip>.16.165:8090",
    "level": "info",
    "method": "GET",
    "msg": "access",
    "pages_https": false,
    "proto": "HTTP/1.1",
    "referrer": "",
    "remote_addr": "<snip>.19.149:36838",
    "remote_ip": "<snip>.19.149",
    "status": 200,
    "system": "http",
    "time": "2023-10-25T12:27:10Z",
    "ttfb_ms": 0,
    "uri": "/-/readiness",
    "user_agent": "kube-probe/1.27+",
    "written_bytes": 8
  }
]
Edited by Jason Plum