Sidekiq MemoryKiller causes `configure` initContainer failure

Summary

When the Sidekiq MemoryKiller triggers a normal worker shutdown in the context of a Helm-managed Gitlab deployment, this leads to the restart of the sidekiq-all-in-one-v2 pod containers and initContainers by the liveliness probe. However, because the sidekiq-init-secrets volume is a projected volume with defaultMode: 400. Additionally, the sidekiq-secrets volume is an emptyDir volume, meaning that because the pod itself is not restarted by the failed liveliness probe, the files moved from sidekiq-init-secrets to sidekiq-secrets still exist in memory and are read-only. This results in the configure initContainer failing to complete successfully with:

'/init-config/gitaly/./gitaly_token' -> '/init-secrets/gitaly/./gitaly_token'
cp: cannot create regular file '/init-secrets/gitaly/./gitaly_token': Permission denied

Steps to reproduce

This has happened organically in a few different environments, but the following steps should reproduce:

  1. Deploy Gitlab using the gitlab helm charts
  2. Trigger graceful shutdown of sidekiq workers by setting a low SIDEKIQ_MEMORY_KILLER_MAX_RSS
  3. Wait for liveliness probes to trigger restart of pod containers
  4. This should result in Init: CrashLoopBackOff for the configure initContainer with logs matching those pasted above

Current behavior

Expected behavior

(What you're expecting to happen)

Versions

  • Chart: sidekiq-5.8.2
  • Platform:
    • Cloud: AKS
    • Self-hosted: Konvoy
  • Kubernetes: (kubectl version)
    • Client: v1.21.0
    • Server: v1.20.13
  • Helm: (helm version)
    • Client: Using the helm-controller from Flux
    • Server: helm-controller version 0.14.0

Relevant logs

I modified the ConfigMap containing the /config/configure script to output file permissions for the folders where init-sidekiq-secret and sidekiq-secrets are mounted to to show the problem more clearly.

Updated script is just:

  configure: |
    set -e
    config_dir="/init-config"
    secret_dir="/init-secrets"

    for secret in gitaly registry rails-secrets ; do
      mkdir -p "${secret_dir}/${secret}"
      ls -al "${config_dir}/${secret}/" || true
      ls -al "${secret_dir}/${secret}/" || true
      cp -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
    done
    for secret in redis minio objectstorage postgres ldap omniauth smtp kas pages oauth-secrets mailroom ; do
      if [ -e "${config_dir}/${secret}" ]; then
        mkdir -p "${secret_dir}/${secret}"
        cp -v -r -L "${config_dir}/${secret}/." "${secret_dir}/${secret}/"
      fi
    done

These are the logs after that change.

> k logs -n gitlab gitlab-sidekiq-all-in-1-v2-b97f49679-thshv -c configure
total 4
drwxr-sr-x.  2 root 1000  60 Mar 31 00:24 .
drwxr-sr-x. 10 root 1000 200 Mar 31 00:24 ..
-r--r-----.  1 root 1000  64 Mar 31 00:24 gitaly_token
total 4
drwxr-sr-x.  2 1000 1000  60 Mar 31 00:24 .
drwxrwsrwt. 10 root 1000 200 Mar 31 00:24 ..
-r--r-----.  1 1000 1000  64 Mar 31 00:24 gitaly_token
'/init-config/gitaly/./gitaly_token' -> '/init-secrets/gitaly/./gitaly_token'
cp: cannot create regular file '/init-secrets/gitaly/./gitaly_token': Permission denied
Edited by Tim Seagren