Skip to content

"wait-for-deps" script never completes, doesn't log any error messages

Summary

Our GitLab deployment is currently stuck on being unable to perform the /scripts/wait-for-deps scripts. We became aware of it this morning when our GitLab Webservice went down and did not start back up. The dependencies initContainer never completes, but it also never prints any logs except for this:

Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Copying smtp_settings.rb into /srv/gitlab/config

As a test we then removed the dependencies initContainer to see if the service will start up regardlesss. The webservice then logs the following, but never becomes live:

Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Copying smtp_settings.rb into /srv/gitlab/config
Starting Puma
{"timestamp":"2023-05-23T12:15:24.664Z","pid":1,"message":"Puma starting in cluster mode..."}
{"timestamp":"2023-05-23T12:15:24.664Z","pid":1,"message":"* Puma version: 5.6.5 (ruby 3.0.6-p216) (\"Birdie's Version\")"}
{"timestamp":"2023-05-23T12:15:24.664Z","pid":1,"message":"*  Min threads: 4"}
{"timestamp":"2023-05-23T12:15:24.664Z","pid":1,"message":"*  Max threads: 4"}
{"timestamp":"2023-05-23T12:15:24.664Z","pid":1,"message":"*  Environment: production"}
{"timestamp":"2023-05-23T12:15:24.664Z","pid":1,"message":"*   Master PID: 1"}
{"timestamp":"2023-05-23T12:15:24.664Z","pid":1,"message":"*      Workers: 2"}
{"timestamp":"2023-05-23T12:15:24.664Z","pid":1,"message":"*     Restarts: (✔) hot (✖) phased"}
{"timestamp":"2023-05-23T12:15:24.664Z","pid":1,"message":"* Preloading application"}

As a further test, we bumped up from chart 6.10.0 to 6.11.5. This time, also on the migrations Job we can see it getting stuck on /scripts/wait-for-deps:

Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Attempting to run '/scripts/wait-for-deps /scripts/db-migrate' as a main process

It would seem that clearly some depedency is suddenly missing, but we have no idea what it is and are not able to discover it.

Steps to reproduce

(Please provide the steps to reproduce the issue)

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

shared-secrets:
  enabled: false
upgradeCheck:
  enabled: false
ingress:
  enabled: false
postgresql:
  install: false
prometheus:
  install: false
global:
  hpa:
    apiVersion: autoscaling/v2
  pdb:
    apiVersion: policy/v1
  certificates:
    customCAs:
      - secret: gitlab
        keys:
          - ca.crt
  gitaly:
    internal:
      names: # To scale the Gitaly StatefulSet, add additional node names here
        - default
        - additional1
  psql:
    password:
      secret: gitlab
      key: gitlab-psql-password
    host: "{{PSQL_HOST}}"
    port: 5432
    username: "{{PSQL_USER}}"
    database: "{{PSQL_DATABASE}}"

  hosts:
    domain: "gitlab.{{PRIVATE_DNS_ZONE_NAME}}"
    gitlab:
      name: "gitlab.{{PRIVATE_DNS_ZONE_NAME}}"
  ingress:
    enabled: false
    configureCertmanager: false
  minio:
    enabled: false
  appConfig:
    omniauth:
      enabled: true
      #autoSignInWithProvider: azure_activedirectory_v2
      syncProfileFromProvider: [azure_activedirectory_v2]
      syncProfileAttributes: [email]
      allowSingleSignOn: [azure_activedirectory_v2]
      blockAutoCreatedUsers: false
      autoLinkLdapUser: false
      autoLinkSamlUser: false
      autoLinkUser: true
      externalProviders: []
      allowBypassTwoFactor: true
      providers:
        - secret: gitlab
          key: gitlab-aad-oauth2-provider
    object_store:
      enabled: false
      proxy_download: true
      storage_options: {}
      connection:
        secret: gitlab
        key: gitlab-storage-account-key
    lfs:
      enabled: true
      proxy_download: true
      bucket: git-lfs
      connection:
        secret: gitlab
        key: gitlab-storage-account-key
    artifacts:
      enabled: true
      proxy_download: true
      bucket: gitlab-artifacts
      connection:
        secret: gitlab
        key: gitlab-storage-account-key
    uploads:
      enabled: true
      proxy_download: true
      bucket: gitlab-uploads
      connection:
        secret: gitlab
        key: gitlab-storage-account-key
    packages:
      enabled: true
      proxy_download: true
      bucket: gitlab-packages
      connection:
        secret: gitlab
        key: gitlab-storage-account-key
    externalDiffs:
      enabled:
      when:
      proxy_download: true
      bucket: gitlab-mr-diffs
      connection:
        secret: gitlab
        key: gitlab-storage-account-key
    terraformState:
      enabled: false
      bucket: gitlab-terraform-state
      connection:
        secret: gitlab
        key: gitlab-storage-account-key
    ciSecureFiles:
      enabled: false
      bucket: gitlab-ci-secure-files
      connection:
        secret: gitlab
        key: gitlab-storage-account-key
    dependencyProxy:
      enabled: false
      bucket: gitlab-dependency-proxy
      connection:
        secret: gitlab
        key: gitlab-storage-account-key
    backups:
      bucket: gitlab-backups
      tmpBucket: gitlab-backups-tmp

certmanager:
  install: false
nginx-ingress:
  enabled: false

gitlab:
  gitaly:
    securityContext:
      fsGroupChangePolicy: "OnRootMismatch"
    persistence:
      volumeName: "{{GITALY_VOLUME_NAME}}"
    nodeSelector:
      topology.kubernetes.io/zone: "{{GITALY_VOLUME_ZONE}}" 
  toolbox:
    enabled: true
    backups:
      objectStorage:
        config:
          secret: gitlab
          key: gitlab-storage-account-key
        backend: azure
gitlab-runner:
  install: true
  runners:
    config: |
      [[runners]]
        [runners.kubernetes]
          namespace = "gitlab"
          image = "ubuntu:22.04"
          image_pull_secrets = ["image-pull-secret"]
          [[runners.kubernetes.volumes.secret]]
            name = "gitlab"
            mount_path = "/usr/local/share/ca-certificates"
            read_only = true
            [runners.kubernetes.volumes.secret.items]
              "ca.crt" = "ca.crt"

Current behavior

"wait-for-deps" getting stuck, not telling us why

Expected behavior

"wait-for-deps" getting stuck should tell us why!

Versions

  • Chart: 6.11.5
  • Platform:
    • Cloud: AKS
  • Kubernetes: (kubectl version)
    • Client: v1.25.5
    • Server: v1.25.5
  • Helm: (helm version)
    • Client: v3.11.1
    • Server:

Relevant logs

(Please provide any relevate log snippets you have collected, using code blocks (```) to format)