SideKiq Crash after docker prune

Summary

After running a docker system prune -a -f to clean up old docker images and ultimately free up a lot of resources, the sidekiq (and sometimes unicorn) pod crashes. A simple delete of the failing pods will restart new pods which are working properly.

Apologies if that should be posted somewhere else, I'm seeing this issue only with Sidekiq, although from the source of the error it could be nothing to do with gitlab and purely how kubernetes mounts the secrets (if so can someone point me to a proper way to clean old docker image without removing docker secrets?)

The output of the kubectl describe pod sidekiq is in "Relevant Logs" below

Steps to reproduce

While gitlab is running do a docker system prune -a -f and then do a kubectl get pods and sidekiq (and potentially unicorn) will have crashed and will get stuck on a CrashLoopBackOff

Current behavior

Sidekiq get stuck in a CrashLoopBackOff

Expected behavior

Sidekiq should still be up and running even after cleaning the docker volumes and images

Versions

• Chart: 3.0
• Platform:
  • Self-hosted: bare metal
• Kubernetes: (kubectl version)
  • Client: 1.17.2
  • Server: 1.17.2
• Helm: (helm version)
  • Client: 3.1.0

Relevant logs

(Please provide any relevate log snippets you have collected, using code blocks (```) to format)

  dependencies:
    Container ID:  docker://5a9add50d555681c0041ce3c0bd03f4f40a9a7cd208c8347126f485b75993025
    Image:         registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v12.7.2
    Image ID:      docker-pullable://registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce@sha256:c0bfab00f848708f55c6dec531494861e7e189fb5588d6ca9151e901a5a652f1
    Port:          <none>
    Host Port:     <none>
    Args:
      /scripts/wait-for-deps
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/fbec14d0-1946-44ab-b619-bad7927c6350/volume-subpaths/sidekiq-secrets/dependencies/2\\\" to rootfs \\\"/var/lib/docker/overlay28a554ed92c6a3d1125d62c1aed72a2969146e8b8045a472b838d1e13ca47f681/merged\\\" at \\\"/cryptfs/docker/overlay2/8a554ed92c6a3d1125d62c1aed72a2969146e8b8045a472b838d1e13ca47f681/merged/srv/gitlab/config/secrets.yml\\\" caused \\\"no such file or directory\\\"\"": unknown
      Exit Code:    128
      Started:      Mon, 17 Feb 2020 08:53:25 +0000
      Finished:     Mon, 17 Feb 2020 08:53:25 +0000