gitlab-rails .gitlab_workhorse_secret ownershop causing EACCS
Summary
gitlab-ctl reconfigure
resets the ownership of /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret
in such a way that gitlab-rails can no longer boot.
Steps to reproduce
I have no idea how we got into this state - I assume it is something to do with our gitlab.rb. I think I introduced the behaviour when I added gitlab_workhorse['secret_token'] = 'secrets-goes-here'
What is the current bug behavior?
Whenever we run gitlab-ctl reconfigure the gitlab-rails application can no longer boot because the initialiser that reads the secret during boot fails. This causes gitlab-ctl to fail also, as it boots rails in order run migration steps.
What is the expected correct behavior?
gitlab-ctl runs without failure, and the application can be started after completion.
Relevant logs
sudo gitlab-ctl reconfigure
.. stuff ..
Recipe: gitlab::gitlab-workhorse
* service[gitlab-workhorse] action restart
- restart service service[gitlab-workhorse]
Running handlers:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20181012-16459-8hmebb" ----
STDOUT: rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:180:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:180:in `open'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:180:in `write_secret'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/gitlab_workhorse_secret.rb:4:in `rescue in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/gitlab_workhorse_secret.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:11:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
/opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret does not contain 32 bytes
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/workhorse.rb:172:in `secret'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/gitlab_workhorse_secret.rb:2:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:11:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20181012-16459-8hmebb" ----
Ran "bash" "/tmp/chef-script20181012-16459-8hmebb" returned 1
Running handlers complete
Chef Client failed. 7 resources updated in 19 seconds
Details of package version
benanderson@flux-prod-git-syd4-1:~$ dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===========================-==================-==================-============================================================
ii gitlab-ce 11.3.1-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis)
un gitlab-ee <none> <none> (no description available)
Environment details
- Operating System:
benanderson@flux-prod-git-syd4-1:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
- Installation Target, remove incorrect values:
- VM: AWS
- Installation Type, remove incorrect values:
- Upgrade from (current upgrade is 11.3.11 -> 11.3.14, but we've had this instance since ~v7.
- Is there any other software running on the machine: Nope
- Is this a single or multiple node installation? Multiple node
- Resources
- CPU:
Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
- Memory total:
- CPU:
benanderson@flux-prod-git-syd4-1:~$ free -mh
total used free shared buff/cache available
Mem: 7.8G 2.1G 1.4G 72M 4.2G 5.3G
Swap: 2.0G 50M 2.0G
Configuration details
There are a lot of secrets in gitlab.rb - sanitisation review in progress. Will edit when I've got it. I'm pretty sure we introduced the problem around 11.X.Y, but it could also be due to the addition of
gitlab_workhorse['secret_token'] = 'secret-secret-secret'