Sidekiq and unicorn pods fails when deploying gitlab helm chart 1.6.1
Summary
Sidekiq and unicorn pods fail to start because there's no psql-password. I guess the reason for this behavior lies in the configure script executed by the init-containers. The init-containers have all needed secrets mounted as a projected volume. This leads to the fact, that inside the volume the secret-file is a symlink:
$ ls -la /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~projected/init-sidekiq-secrets/postgres/
total 0
drwxr-sr-x 2 root sysadmin 60 Mar 12 13:24 .
drwxrwsrwt 9 root sysadmin 200 Mar 12 13:24 ..
lrwxrwxrwx 1 root root 32 Mar 12 13:24 psql-password -> ../..data/postgres/psql-password
$ cat /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~projected/init-sidekiq-secrets/postgres/psql-password
REDACTED
But since the configure script uses cp -r
that symlink is copied as-is (as a symlink) instead of copying its content:
$ ls -la /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~empty-dir/sidekiq-secrets/postgres/
total 0
drwxr-sr-x 2 sysadmin sysadmin 60 Mar 12 13:24 .
drwxrwsrwt 8 root sysadmin 160 Mar 12 13:24 ..
lrwxrwxrwx 1 sysadmin sysadmin 32 Mar 12 13:24 psql-password -> ../..data/postgres/psql-password
$ cat /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~empty-dir/sidekiq-secrets/postgres/psql-password
cat: /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~empty-dir/sidekiq-secrets/postgres/psql-password: No such file or directory
Steps to reproduce
Install gitlab chart 1.6.1
Configuration used
certmanager:
install: false
certmanager-issuer:
email: REDACTED
gitlab:
migrations:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-rails-ce
sidekiq:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce
task-runner:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-task-runner-ce
unicorn:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ce
workhorse:
image: registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce
global:
hosts:
domain: REDACTED
externalIP: REDACTED
https: false
ingress:
class: ""
configureCertmanager: false
tls:
secretName: REDACTED
nginx-ingress:
enabled: false
registry:
enabled: false
Current behavior
Pods fail to start
Expected behavior
Pods don't fail to start
Versions
- Chart: 1.6.1
- Platform:
- Self-hosted: bare metal
- Kubernetes: (
kubectl version
)- Client: v1.10.0
- Server: v1.8.4
- Helm: (
helm version
)- Client: v2.13.0
- Server: v2.13.0
Relevant logs
from both containers (sidekiq and unicorn)
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
(erb):7:in `read': No such file or directory @ rb_sysopen - /etc/gitlab/postgres/psql-password (Errno::ENOENT)
from (erb):7:in `<main>'
from /usr/lib/ruby/2.5.0/erb.rb:876:in `eval'
from /usr/lib/ruby/2.5.0/erb.rb:876:in `result'
from /scripts/set-config:22:in `block in <main>'
from /scripts/set-config:18:in `each'
from /scripts/set-config:18:in `<main>'
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/database.yml