FATAL error on upgrade to GitLab 15 when missing arctifact, lfs or uploads object storage provider
Summary
The new global object store configuration parsing for gitlab.rb
in GitLab 15 has this problem for the edge-case where you declare gitlab_rails['xxxxx_object_store_enabled'] = true
, but then don’t specify the object storage. So that is, if you have any such declaration but don’t specify an object storage for it, the error will be about a missing one for Terraform:
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:189:in `missing_key’:
Missing setting ‘provider’ in ‘connection’ section in ‘terraform_state’ section
in /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml (Settingslogic::MissingSetting)
In one known instance it was the LFS which had object storage enabled, but no storage provider supplied, but still the error was about Terraform provider missing. In another case, there was a provider specified for the Terraform storage, but a different storage was missing a provider, and still the error was about a missing Terraform provider.
Steps to reproduce
- Enable artifact, lfs, or uploads object storage in
gitlab.rb
(e.g.gitlab_rails['artifacts_object_store_enabled'] = true
) - Do not specify a storage provider for it
Example Project
(This cannot be reproduced in a GitLab.com project: it is a system setting for self-managed GitLab instances)
What is the current bug behavior?
For most storages where object store is enabled, but system configuration is missing the storage provider:
- Bug: a
gitlab-ctl reconfigure
will fail with an error that the Terraform storage provider is missing, but not report which storage is actually missing. - If the Terraform State object store is enabled, but the provider is missing, this actually reconfigures without error
- Regression: The reconfigure will fail, whereas previously this would succeed, with storage falling back to filesystem storage
What is the expected correct behavior?
This should be resolved either by
- ignoring missing providers and failing back to file storage (as it did in 14.10 and earlier), or
- by correctly reporting which provider is actually missing, not always saying that it’s Terraform
Relevant logs and/or screenshots
Output for missing artifacts object storage
Running handlers:
[2022-07-13T15:55:20+10:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:
execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 476) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
Settingslogic::MissingSetting: Missing setting 'provider' in 'connection' section in 'terraform_state' section in /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:16:in `verify!'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:45:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:44:in `each'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:44:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:37:in `tap'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:37:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1
Warnings:
Environment variable LANG specifies a non-UTF-8 locale. GitLab requires UTF-8 encoding to function properly. Please check your locale settings.
Running handlers complete
[2022-07-13T15:55:20+10:00] ERROR: Exception handlers complete
Infra Phase failed. 3 resources updated in 26 seconds
Warnings:
Environment variable LANG specifies a non-UTF-8 locale. GitLab requires UTF-8 encoding to function properly. Please check your locale settings.
[2022-07-13T15:55:20+10:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2022-07-13T15:55:20+10:00] FATAL: ---------------------------------------------------------------------------------------
[2022-07-13T15:55:20+10:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2022-07-13T15:55:20+10:00] FATAL: ---------------------------------------------------------------------------------------
[2022-07-13T15:55:20+10:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 476) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
Settingslogic::MissingSetting: Missing setting 'provider' in 'connection' section in 'terraform_state' section in /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:16:in `verify!'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:45:in `block (2 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:44:in `each'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:44:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:37:in `tap'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/direct_upload_support.rb:37:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
theseus:/home/mjl # gitlab-rake gitab:env:info rake aborted! Don't know how to build task 'gitab:env:info' (See the list of available tasks with `rake --tasks`) Did you mean? gitlab:env:info /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `' (See full trace by running task with --trace) theseus:/home/mjl # gitlab-rake gitlab:env:info System information System: openSUSE 15.3 Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.5p203 Gem Version: 3.1.4 Bundler Version:2.3.15 Rake Version: 13.0.6 Redis Version: 6.2.7 Sidekiq Version:6.4.0 Go Version: unknown GitLab information Version: 15.1.2-ee Revision: 324ae02f89f Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.10 URL: http://theseus HTTP Clone URL: http://theseus/some-group/some-project.git SSH Clone URL: git@theseus:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.7.4 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
theseus:/home/mjl # gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.7.4 ? ... OK (14.7.4) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/1 ... yes 4/2 ... yes 3/3 ... yes 10/4 ... yes 10/5 ... yes 13/6 ... yes 3/7 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.5) Git user has default SSH configuration? ... yes Active users: ... 3 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished