Some unicorn pods can't initialize
Summary
Trying to deploy the latest chart version (2.0.3) and some (not all) of the unicorn pods are failing to start
Steps to reproduce
Deploy 2.0.3 charts with gitlab version 11.7.12
Configuration used
gitlab:
unicorn:
helmTests:
enabled: false
redis:
serviceName: stunnel
enabled: true
metrics:
enabled: false
nodeSelector:
gpfs-install: 'true'
tolerations:
- key: "gpfs"
operator: "Equal"
value: "true"
effect: "NoSchedule"
- key: "gpfs"
operator: "Equal"
value: "true"
effect: "NoExecute"
minReplicas: 5
registry:
enabled: false
workerProcesses: 20
workerTimeout: 60
workhorse:
image: us.icr.io/opentoolchain/gitlab-workhorse
tag: v11.7.12
image:
repository: us.icr.io/opentoolchain/gitlab-unicorn
tag: v11.7.12
pullPolicy: 'Always'
pullSecrets:
- name: otc-registry-secret
Current behavior
Gitlab deploys fine, everything starts up, except for half of the unicorn pods??? The pods that fail all have the same exception in their logs:
mrennie$ kubectl logs gitlab-unicorn-5998f6b7b7-m4thw -c dependencies
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
/usr/lib/ruby/2.4.0/fileutils.rb:1290:in `initialize': Permission denied @ rb_sysopen - /srv/gitlab/config/unicorn.rb (Errno::EACCES)
from /usr/lib/ruby/2.4.0/fileutils.rb:1290:in `open'
from /usr/lib/ruby/2.4.0/fileutils.rb:1290:in `block in copy_file'
from /usr/lib/ruby/2.4.0/fileutils.rb:1289:in `open'
from /usr/lib/ruby/2.4.0/fileutils.rb:1289:in `copy_file'
from /usr/lib/ruby/2.4.0/fileutils.rb:430:in `copy_file'
from /usr/lib/ruby/2.4.0/fileutils.rb:357:in `block in cp'
from /usr/lib/ruby/2.4.0/fileutils.rb:1461:in `block in fu_each_src_dest'
from /usr/lib/ruby/2.4.0/fileutils.rb:1475:in `fu_each_src_dest0'
from /usr/lib/ruby/2.4.0/fileutils.rb:1459:in `fu_each_src_dest'
from /usr/lib/ruby/2.4.0/fileutils.rb:356:in `cp'
from /scripts/set-config:30:in `block in <main>'
from /scripts/set-config:28:in `each'
from /scripts/set-config:28:in `<main>'
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/resque.yml
Copying other config files found in /var/opt/gitlab/templates
Copying unicorn.rb into /srv/gitlab/config
Even with half of the unicorn pods failing, gitlab will operate and take traffic. None of the other pods fail.
Expected behavior
That all of the unicorn pods would start fine
Versions
- Chart: 2.0.3
- Platform:
- Cloud: IBM Cloud
- Kubernetes: (
kubectl version
)- Client: v1.14.2
- Server: v1.12.9+IKS
- Helm: (
helm version
)- Client: v2.13.1
- Server: v2.14.1