gitlab-workhorse 502 badgateway: failed to receive response: EOF

Summary

After deploying the last GitLab Helm chart release (v6.11.5) on my Kubernetes cluster, all pods are running and working but I get sometimes 502 error "502 Whoops, GitLab is taking too much time to respond." while navigating through the GitLab UI. A simple browser refresh (F5) and the page is loading correctly. The error happens about one time every 30 page loads, it seems quite random.

gitlab-workhorse container has the "bad gateway" log as described bellow.

Here are some related issues but the error message is not the same :

“502 Whoops, GitLab is taking too much time to respond.” after 13.x update

502-Whoops, GitLab is taking too much time to respond

Steps to reproduce

Navigate through the deployed GitLab web service.

Configuration used

certmanager:
  install: false
  installCRDs: false
gitlab:
  toolbox:
    enabled: false
  kas:
    ingress:
      tls:
        secretName: gitlab-kas-tls
  migrations:
    enabled: true
  sidekiq:
    minReplicas: 1
    registry:
      enabled: false
    resources:
      limits:
        memory: 1.5G
      requests:
        cpu: 50m
        memory: 625M
  unicorn:
    registry:
      enabled: false
  webservice:
    enabled: true
    minReplicas: 1
    workhorse:
      resources:
        limits:
          memory: 100M
        requests:
          cpu: 10m
          memory: 10M
    resources:
      limits:
        memory: 1.5G
      requests:
        cpu: 100m
        memory: 900M
    ingress:
      tls:
        enabled: true
        secretName: gitlab-gitlab-tls
    registry:
      enabled: false
  gitlab-shell:
    minReplicas: 1
gitlab-runner:
  runners:
    config: |
      [[runners]]
        [runners.kubernetes]
        image = "ubuntu:22.04"
        {{- if .Values.global.minio.enabled }}
        [runners.cache]
          Type = "s3"
          Path = "gitlab-runner"
          Shared = true
          [runners.cache.s3]
            ServerAddress = {{ include "gitlab-runner.cache-tpl.s3ServerAddress" . }}
            BucketName = "runner-cache"
            BucketLocation = "us-east-1"
            Insecure = false
        {{ end }}
        [[runners.kubernetes.volumes.empty_dir]]
        name = "docker-certs"
        mount_path = "/certs/client"
        medium = "Memory"
    privileged: true
global:
  rails:
    bootsnap:
      enabled: false
  edition: ce
  registry:
    enabled: false
  appConfig:
    defaultProjectsFeatures:
      containerRegistry: false
  email:
    display_name: 'GitLab'
    from: 'gitlab@example.com'
    reply_to: 'no-reply@example.com'
  hosts:
    domain: example.com
    https: true
  ingress:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt
      kubernetes.io/tls-acme: true
    class: nginx
    configureCertmanager: false
    enabled: true
    provider: nginx
    tls:
      enabled: true
      secretName: gitlab-wildcard-tls
  kas:
    enabled: true
  smtp:
    address: 'ssl0.ovh.net'
    enabled: true
    tls: true
    password:
      key: password
      secret: smtp-password
    port: 465
    authentication: "login"
    starttls_auto: false
    user_name: example.com
    openssl_verify_mode: "none"
  psql:
    host: postgresql.postgresql.svc.cluster.local
    password:
      secret: postgresql-password
      key: password
  redis:
    password:
      enabled: true
      secret: redis-password
      key: password
    host: redis-master.redis.svc.cluster.local
grafana:
  enabled: false
nginx-ingress:
  enabled: false
prometheus:
  enabled: false
  install: false
registry:
  enabled: false
postgresql:
  install: false
redis:
  install: false
minio:
  resources:
    requests:
      cpu: 10m
      memory: 64Mi

Versions

• Chart: 6.11.5
• Platform:
  • Cloud: OVH Public Cloud Managed Kubernetes Service
• Kubernetes: (kubectl version)
  • Client: Major:"1", Minor:"25", GitVersion:"v1.25.9", GoVersion:"go1.19.8", Compiler:"gc", Platform:"darwin/amd64"
  • Server: Major:"1", Minor:"25", GitVersion:"v1.25.9", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"
• Helm: (helm version)
  • Client: v3.11.0
  • Server: v3.11.0

Relevant logs

{
  "correlation_id": "01H11KCBBZ19JX2RGRBF9XMQCP",
  "duration_ms": 108,
  "error": "badgateway: failed to receive response: EOF",
  "level": "error",
  "method": "GET",
  "msg": "",
  "time": "2023-05-22T11:26:44Z",
  "uri": "/api/v4/internal/kubernetes/modules/starboard_vulnerability/policies_configuration"
}

{
  "content_type": "text/plain",
  "correlation_id": "01H11KCBBZ19JX2RGRBF9XMQCP",
  "duration_ms": 109,
  "host": "gitlab-webservice-default.gitlab.svc:8181",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "",
  "remote_addr": "10.2.0.229:51654",
  "remote_ip": "10.2.0.229",
  "route": "^/api/",
  "status": 502,
  "system": "http",
  "time": "2023-05-22T11:26:44Z",
  "ttfb_ms": 109,
  "uri": "/api/v4/internal/kubernetes/modules/starboard_vulnerability/policies_configuration",
  "user_agent": "gitlab-kas/v15.11.0/v15.11.0",
  "written_bytes": 24
}