Runners exhibit 500 internal server error when new build is posted
After migrating from a source-installed GitLab to an omnibus-installed GitLab (I used GitLab's documentation on the migration process: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/backup_restore.md and https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/mysql_to_postgresql.md) and I found that my CI runners were no longer working. I got "Checking for builds... failed" in system.log, and the build page remained at status "running" forever.
Additionally, if I ran gitlab-ci-multi-runner --debug --log-level "debug" run
I would see a 500 internal server error
any time I tried to run a build. Checking production.log showed OpenSSL::Cipher::CipherError (bad decrypt) gitlab
, which lead me to the gitlab-secrets.json file.
Fixing the problem involved setting the value for gitlab_ci -> db_key_base in (omnibus installation) /var/opt/gitlab/gitlab-secrets.json to the value of db_key_base in (from-source installation) ./gitlab/config/secrets.yml, then running gitlab-ctl reconfigure
.
Current issues talk about the secrets file causing problems with 2FA but there's nothing about it affecting CI (ex: https://gitlab.com/gitlab-org/gitlab-ce/issues/1960#note_1606998)
Given that I used GitLab's documentation on the migration process and presumably didn't miss any steps that would cause this issue, I suspect others will run into the same problem that I did.