gitlab-unicorn `dependencies` init container fails to start
Summary
New summary: See this comment and thread for details.
TL;DR: When trying to force the Gitlab chart to use an external CertManager instance, the chart spectacularly fails to deploy in that the helm upgrade
command itself will time out, and even if you hack away at the things that Gitlab thinks it needs (but doesn't) like the self-sign init containers, you still end up in a bad state.
Solution: Don't try to force Gitlab to use an external cert manager until that's tested.
Old summary.
The dependencies
init container appears to fail in reading/creating the resque.yml
file.
Steps to reproduce
No idea, my chart deployment is in a bad state and has been for some unknown amount of time (ugh, a move & job change lead to me not checking the deployment in a while). See the commit hashes below to bracket the time range. Some time between 85 days ago (last deployment) and today.
Configuration used
I suspect this isn't actually necessary. It looks like the issue is the init container has a race condition and a migration was expected to be run, but was not.
Relevant config items: I'm running my own separate cert-manager instance, not part of the chart. So I followed the TLS instructions to not install cert manager, reference the secret names, etc.
Current behavior
The init container for the gitlab-unicorn deployments fails consistently (across upgrades and killing of pods to create fresh instances) with an error:
$ kubectl logs -n gitlab gitlab-unicorn-674f9d46c5-m786z -c dependencies
+ /scripts/set-config /var/opt/gitlab/templates /var/opt/gitlab/config/gitlab/
/scripts/set-config:22:in `write': No such file or directory @ rb_sysopen - /var/opt/gitlab/config/gitlab/resque.yml (Errno::ENOENT)
from /scripts/set-config:22:in `block in <main>'
from /scripts/set-config:18:in `each'
from /scripts/set-config:18:in `<main>'
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /var/opt/gitlab/config/gitlab/resque.yml
Expected behavior
The init container should not exit with an error status code.
Versions
- Chart: Multiple versions of the chart starting with e09c5792 to 8f3e2b13
- Platform:
- Cloud: AKS
- Self-hosted: (OpenShift | Minikube | Rancher RKE | ?)
- Kubernetes: Multiple versions from 1.10.6 to 1.11.4
- Client: 1.11.4
- Server: 1.11.4
- Helm: (
helm version
) Multiple versions from 2.9.1 to 2.11.0- Client: 2.11.0
- Server: 2.11.0
Relevant logs
$ kubectl logs -n gitlab gitlab-unicorn-674f9d46c5-m786z -c dependencies
+ /scripts/set-config /var/opt/gitlab/templates /var/opt/gitlab/config/gitlab/
/scripts/set-config:22:in `write': No such file or directory @ rb_sysopen - /var/opt/gitlab/config/gitlab/resque.yml (Errno::ENOENT)
from /scripts/set-config:22:in `block in <main>'
from /scripts/set-config:18:in `each'
from /scripts/set-config:18:in `<main>'
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /var/opt/gitlab/config/gitlab/resque.yml