Missing privilege separation directory: /run/sshd

Summary

While upgrading helm chart from v4.6.3 to v4.7.4, gitlab-shell goes in CrashLoopBackoff State with the error:

==> /var/log/gitlab-shell/ssh.log <==
Missing privilege separation directory: /run/sshd

/run/ is a symlink /var/run, the image used in the gitlab shell pod (chart 4.6.3) registry.gitlab.com/gitlab-org/build/cng/gitlab-shell:v13.13.0 and the container does not contain /run/sshd directory nor with char v4.7.4 image registry.gitlab.com/gitlab-org/build/cng/gitlab-shell:v13.14.0.

I could not reproduce the error anyway I tried, Although I have been testing with GCP deployment and Kubernetnes version 1.17.15

Customer environment: cluster is hybrid cluster with 3 kube masters déployed as a vm, and 5 worker on bare metal. and Cluster is running kubernetes 1.18.14.

I have also tried to mimic the deployment as close to customers values.yaml (attached) that I have but still could not reproduce the error.

The issue seem to go away if we set in the gitab-shell sshd configmap :

UsePrivilegesSeparation: No

But its strange that we even have to do it. Any ideas what might be causing this problem?

Steps to reproduce

None

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

certmanager:
  install: false
  rbac:
    create: true
  webhook:
    enabled: false
gitlab:
  registry:
    enabled: false
  sidekiq:
    registry:
      enabled: false
  task-runner:
    backups:
      objectStorage:
        config:
          key: key
          secret: secret
  webservice:
    registry:
      enabled: false
gitlab-runner:
  certsSecretName: gitlab-runner-cert
  install: false
  rbac:
    create: true
  runners:
    cache:
      cacheShared: true
      cacheType: s3
      s3BucketLocation: us-east-1
      s3BucketName: runner-cache
      s3CacheInsecure: false
      s3CachePath: gitlab-runner
      s3ServerAddress: server-address
      secret: secret
    privileged: true
global:
  antiAffinity: soft
  appConfig:
    artifacts:
      bucket: gitlab-artifacts-storage
      connection:
        key: connection
        secret: secret
    backups:
      bucket: gitlab-backup-storage
      enable: true
      tmpBucket: gitlab-tmp-storage
    cron_jobs:
      pipeline_schedule_worker:
        cron: /15 * * * *
    defaultCanCreateGroup: false
    defaultProjectsFeatures:
      builds: true
      issues: true
      mergeRequests: true
      snippets: true
      wiki: true
    defaultTheme: null
    enableUsagePing: true
    externalDiffs:
      bucket: gitlab-mr-diffs
      connection:
        key: connection
        secret: secret
      when: null
    extra:
      googleAnalyticsId: null
      piwikSiteId: null
      piwikUrl: null
    gravatar:
      plainUrl: null
      sslUrl: null
    incomingEmail:
      address: ""
      enabled: false
      expungeDeleted: false
      host: hostname
      idleTimeout: 60
      logger:
        logPath: /dev/stdout
      mailbox: inbox
      password:
        key: password
        secret: ""
      port: 993
      ssl: true
      startTls: false
      user: ""
    issueClosingPattern: null
    ldap:
      servers:
        main:
          active_directory: false
          admin_group: ""
          allow_username_or_email_login: true
          attributes:
            email:
            - mail
            - email
            - userPrincipalName
            first_name: givenName
            last_name: xx
            name: xx
            username:
            - uid
            - userid
            - sAMAccountName
          base: dc=xx,dc=xx
          bind_dn: uid=xx,ou=xx,dc=xx,dc=xx
          block_auto_created_users: false
          group_base: ""
          host: hostname
          label: LDAP
          method: plain
          password:
            key: password
            secret: passwordl
          port: 389
          sync_ssh_keys: false
          uid: uid
          user_filter: ""
    lfs:
      bucket: gitlab-lfs-storage
      connection:
        key: connection
        secret: secret
      enable: true
    omniauth:
      allowSingleSignOn:
      - saml
      autoLinkLdapUser: false
      autoLinkSamlUser: false
      autoSignInWithProvider: null
      blockAutoCreatedUsers: true
      enabled: false
      externalProviders: []
      providers: []
      syncProfileAttributes:
      - email
      syncProfileFromProvider: []
    packages:
      bucket: gitlab-packages-storage
      connection:
        key: connection
        secret: secret
    pseudonymizer:
      bucket: gitlab-ps
      configMap: null
      connection: {}
    uploads:
      bucket: gitlab-uploads-storage
      connection:
        key: connection
        secret: secret
    usernameChangingEnabled: false
    webhookTimeout: null
  application:
    allowClusterRoles: false
    create: false
    links: []
  busybox:
    image:
      repository: busybox
      tag: latest
  certificates:
    customCAs:
    - secret: secret
    - secret: secret
    image:
      repository: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates
      tag: tag
  email:
    display_name: GitLab
    from: ""
    reply_to: ""
    subject_suffix: ""
  gitaly:
    authToken:
      key: token
      secret: secret
    enabled: false
    external:
    - hostname: hostname
      name: default
    shell:
      authToken:
        key: secret
        secret: shell-secret
      port: 2223
  grafana:
    enabled: false
  hosts:
    domain: hostname
    gitlab:
      https: true
      name: hostname
    https: true
    ssh: null
  ingress:
    annotations:
      ingress.kubernetes.io/config-backend: http-request set-header X-Forwarded-Proto
        https
    configureCertmanager: false
    enabled: true
    tls:
      enabled: false
  initialRootPassword: {}
  kubectl:
    image:
      pullSecrets: []
      repository: registry.gitlab.com/gitlab-org/build/cng/kubectl
      tag: 1.13.12
    securityContext:
      fsGroup: 65534
      runAsUser: 65534
  minio:
    credentials: {}
    enabled: false
  operator:
    enabled: false
    rollout:
      autoPause: true
  pages:
    accessControl: false
    apiSecret: {}
    artifactsServer: false
    enabled: false
    externalHttp: false
    externalHttps: false
    host: null
    https: false
    objectStore:
      bucket: pages
      connection: {}
      enabled: false
    path: null
    port: null
  praefect:
    authToken: {}
    autoMigrate: true
    dbSecret: {}
    enabled: false
    gitalyReplicas: 3
    psql:
      sslMode: disable
  psql:
    database: gitlabhq_production
    host: hostname
    password:
      key: postgres-password
      secret: secret
    passwordHard: password
    port: 1520
    username: gitlab
  rails:
    bootsnap:
      enabled: true
  railsSecrets: {}
  redis:
    host: hostname
    password:
      key: secret
      secret: gitlab-redis-secret
  registry:
    bucket: registry
    certificate: {}
    httpSecret: {}
  runner:
    registrationToken: {}
  service:
    annotations: {}
  shell:
    authToken: {}
    hostKeys: {}
    port: 2223
  smtp:
    address: hostname
    authentication: ""
    enabled: false
    openssl_verify_mode: peer
    port: 25
    starttls_auto: false
    user_name: ""
  time_zone: UTC
  webservice:
    workerTimeout: 60
  workhorse:
    serviceName: webservice
nginx-ingress:
  controller:
    config:
      enable-vts-status: "true"
      hsts-include-subdomains: "false"
      server-name-hash-bucket-size: "256"
      server-tokens: "false"
      ssl-ciphers: ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4
      ssl-protocols: TLSv1.1 TLSv1.2
      use-http2: "false"
    metrics:
      enabled: true
      service:
        annotations:
          prometheus.io/port: "10254"
          prometheus.io/scrape: "true"
    minAvailable: 1
    publishService:
      enabled: true
    replicaCount: 2
    resources:
      requests:
        cpu: 100m
        memory: 100Mi
    scope:
      enabled: true
    service:
      externalTrafficPolicy: Local
    stats:
      enabled: true
  defaultBackend:
    minAvailable: 1
    replicaCount: 1
    resources:
      requests:
        cpu: 5m
        memory: 5Mi
  enabled: false
  rbac:
    create: true
  serviceAccount:
    create: true
  tcpExternalConfig: "true"
postgresql:
  existingSecret: secret
  imageTag: 9.6.8
  install: false
  metrics:
    enabled: true
  postgresDatabase: gitlabhq_production
  postgresUser: gitlab
  usePasswordFile: true
prometheus:
  alertmanager:
    enabled: false
  alertmanagerFiles:
    alertmanager.yml: {}
  install: false
  kubeStateMetrics:
    enabled: false
  nodeExporter:
    enabled: false
  pushgateway:
    enabled: false
  rbac:
    create: true
registry:
  enabled: false
shared-secrets:
  enabled: true
  rbac:
    create: true

Versions

  • Chart: (4.6.3)
  • Platform:
    • Cloud: (GKE)
    • Self-hosted: ( VM | bare metal )
  • Kubernetes: (1.18.14)

ZD

Edited by Julius Kvedaras