Documentation regarding secret generation is not correct
Summary
Recently GitLab.com attempt to deploy a change that required a new secret to be automatically generated. Our documentation states that for any secret not already known to the chart, will be generated: https://docs.gitlab.com/charts/installation/secrets.html
This was attempted during a recently deploy, but this failed on GitLab.com:
Checking database connection and schema version
Missing Rails.application.secrets.ci_jwt_signing_key for production environment. The secret will be generated and stored in config/secrets.yml.
rake aborted!
Errno::EROFS: Read-only file system @ rb_sysopen - /srv/gitlab/config/secrets.yml
/srv/gitlab/config/initializers/01_secret_token.rb:106:in `write'
/srv/gitlab/config/initializers/01_secret_token.rb:106:in `write_secrets_yml'
/srv/gitlab/config/initializers/01_secret_token.rb:47:in `create_tokens'
/srv/gitlab/config/initializers/01_secret_token.rb:109:in `<main>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:55:in `load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:55:in `load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:318:in `block in load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:291:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:318:in `load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/engine.rb:666:in `block in load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:182:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/engine.rb:665:in `load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/engine.rb:625:in `block (2 levels) in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/engine.rb:624:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/engine.rb:624:in `block in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:32:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:32:in `run'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:61:in `block in run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:50:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:50:in `tsort_each_child'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/initializable.rb:60:in `run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/application.rb:363:in `initialize!'
/srv/gitlab/config/environment.rb:5:in `<main>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:324:in `block in require'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:291:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.1/lib/active_support/dependencies.rb:324:in `require'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/application.rb:339:in `require_environment!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.1/lib/rails/application.rb:523:in `block in run_tasks_blocks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'
Tasks: TOP => db:version => db:load_config => environment
(See full trace by running task with --trace)
WARNING: Waiting for all services to be operational, and data migrations to complete.
If this container continues to fail / restart, please see:
https://docs.gitlab.com/charts/troubleshooting/index.html#application-containers-constantly-initializing
As these secrets are centrally managed not by our helm chart but instead by manual creation, this documentation is misleading. This failed as the secret generation had occurred inside of the rails application and attempted to write to a location that is read-only as it's sucking in secrets that helm is responsible for creating.
Steps to reproduce
- Self manage secrets, but purposely forget to include one
- Attempt to install the chart
- The dependencies container will fail to write a generated certificate and get stuck in a crash loop
Configuration used
Configurations used by GitLab.com and smaller environments are stored in https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com
Current behavior
The dependencies container crashes.
Regardless, our documentation should be update to include information such as this.
Expected behavior
This is a good question. Ideally the secret that is missing would be reported such that an administrator can begin the process of handling said secret, but we'd need to know about this prior to the upgrade
Versions
- Chart:
80f38d5dfc0a235ab49414b41e9d04081a77a15b
- Platform:
- Cloud: GKE
- Kubernetes:
- Client:
1.15.11
- Server:
1.15.11-gke.1
- Client:
- Helm:
- Client:
2.16.5
- Server:
n/a
(Tillerless plugin)
- Client: