Gitlab-CE docker container refuse to start: Error executing action `restart` on resource 'runit_service[prometheus]'
Summary
Our GitLab-CE docker instance refuse to start, it seems to be related to Prometheus which we don't use
Steps to reproduce
It's hard to tell since we have 3 instances and they all get upgrade at the same time, it's the only one that fails since the last upgrade and it used to work after it.
What is the current bug behavior?
Gitlab-CE crash on startup
What is the expected correct behavior?
Gitlab-CE should start
Relevant logs and/or screenshots
Recipe: monitoring::prometheus
* runit_service[prometheus] action restart
================================================================================
Error executing action `restart` on resource 'runit_service[prometheus]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/prometheus ----
STDOUT: timeout: run: /opt/gitlab/service/prometheus: (pid 719) 138s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/prometheus ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/prometheus returned 1
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `tap'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:136:in `safe_sv_shellout!'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/helpers.rb:164:in `restart_service'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb:357:in `block in <class:RunitService>'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/prometheus.rb
87: runit_service 'prometheus' do
88: options({
89: log_directory: prometheus_log_dir,
90: flags: runtime_flags,
91: env_dir: prometheus_static_etc_dir
92: }.merge(params))
93: log_options node['gitlab']['logging'].to_hash.merge(
94: { log_directory: prometheus_log_dir }
95: )
96: end
97:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/prometheus.rb:87:in `from_file'
runit_service("prometheus") do
action [:enable]
updated true
default_guard_interpreter :default
declared_type :runit_service
cookbook_name "monitoring"
recipe_name "prometheus"
service_name "prometheus"
options {:env_dir=>"/opt/gitlab/etc/prometheus/env", :log_directory=>"/var/log/gitlab/prometheus", :flags=>"--web.listen-address=localhost:9090 --storage.tsdb.path=/var/opt/gitlab/prometheus/data --config.file=/var/opt/gitlab/prometheus/prometheus.yml"}
log_options {"svlogd_size"=>209715200, "svlogd_num"=>30, "svlogd_timeout"=>86400, "svlogd_filter"=>"gzip", "svlogd_udp"=>nil, "svlogd_prefix"=>nil, "udp_log_shipping_host"=>nil, "udp_log_shipping_hostname"=>nil, "udp_log_shipping_port"=>514, "logrotate_frequency"=>"daily", "logrotate_maxsize"=>nil, "logrotate_size"=>nil, "logrotate_rotate"=>30, "logrotate_compress"=>"compress", "logrotate_method"=>"copytruncate", "logrotate_postrotate"=>nil, "logrotate_dateformat"=>nil, :log_directory=>"/var/log/gitlab/prometheus"}
run_template_name "prometheus"
log_template_name "prometheus"
check_script_template_name "prometheus"
finish_script_template_name "prometheus"
control_template_names {}
end
System Info:
------------
chef_version=15.12.22
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Recipe: monitoring::alertmanager
* runit_service[alertmanager] action restart (up to date)
Recipe: monitoring::postgres-exporter
* runit_service[postgres-exporter] action restart (up to date)
Recipe: monitoring::grafana
* runit_service[grafana] action restart (up to date)
Running handlers:
There was an error running gitlab-ctl reconfigure:
runit_service[prometheus] (monitoring::prometheus line 87) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/prometheus ----
STDOUT: timeout: run: /opt/gitlab/service/prometheus: (pid 719) 138s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/prometheus ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/prometheus returned 1
Running handlers complete
Chef Infra Client failed. 373 resources updated in 03 minutes 57 seconds
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
``` System information System: Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.28.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.4.3 Revision: 6689552c479 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.9 URL: https://module.xxxxx.com HTTP Clone URL: https://module.xxxxx.com/some-group/some-project.git SSH Clone URL: ssh://git@module.xxxxx.com:20024/some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.7.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git ```
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 13.7.0 ? ... OK (13.7.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: FAILED - Internal API unreachable gitlab-shell self-check failed Try fixing it: Make sure GitLab is running; Check the gitlab-shell configuration file: sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml Please fix the error above and rerun the checks. Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks. Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Checking Reply by email ... IMAP server credentials are correct? ... Checking git-module@kopaxgroup.com yes Init.d configured correctly? ... skipped MailRoom running? ... skipped Checking Reply by email ... Finished Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 6 users of 100 limit. Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/1 ... yes 5/2 ... yes 13/3 ... yes 7/5 ... yes 6/6 ... yes 2/7 ... yes 9/12 ... yes 9/14 ... yes 9/15 ... yes 9/16 ... yes 6/52 ... yes 9/53 ... yes 6/54 ... yes 9/56 ... yes 12/59 ... yes 6/62 ... yes 6/63 ... yes 7/64 ... yes 9/66 ... yes 13/68 ... yes 6/69 ... yes 12/70 ... yes 14/71 ... yes 12/72 ... yes 6/73 ... yes 13/74 ... yes 13/75 ... yes 17/76 ... yes 13/77 ... yes 18/78 ... yes 4/80 ... yes 19/81 ... yes 13/84 ... yes 20/86 ... yes 20/87 ... yes 20/88 ... yes 13/90 ... yes 17/91 ... yes 13/92 ... yes 22/93 ... yes 3/94 ... yes 13/95 ... yes 14/97 ... yes 13/99 ... yes 9/101 ... yes 9/102 ... yes 23/103 ... yes 9/104 ... yes 13/109 ... yes 24/110 ... yes 13/111 ... yes 13/112 ... yes 13/113 ... yes 9/114 ... yes 9/116 ... yes 13/122 ... yes 9/127 ... yes 9/128 ... yes 3/129 ... yes 13/131 ... yes 23/138 ... yes 3/139 ... yes 4/141 ... yes 4/142 ... yes 9/145 ... yes 29/147 ... yes 29/148 ... yes 29/149 ... yes 29/150 ... yes 29/151 ... yes 29/152 ... yes 29/153 ... yes 26/161 ... yes 23/162 ... yes 29/165 ... yes 30/167 ... yes 3/169 ... yes 26/171 ... yes 9/173 ... yes 26/174 ... yes 30/176 ... yes 23/177 ... yes 26/178 ... yes 13/179 ... yes 13/180 ... yes 13/181 ... yes 13/182 ... yes 22/183 ... yes 22/184 ... yes 13/185 ... yes 13/186 ... yes 13/187 ... yes 13/188 ... yes 3/190 ... yes 26/192 ... yes 29/193 ... yes 29/194 ... yes 13/195 ... yes 29/196 ... yes 13/197 ... yes 26/198 ... yes 13/199 ... yes 26/201 ... yes 26/202 ... yes 13/203 ... yes 26/204 ... yes 26/206 ... yes 3/207 ... yes 26/208 ... yes 26/209 ... yes 36/211 ... yes 23/214 ... yes 36/215 ... yes 36/217 ... yes 13/218 ... yes 26/219 ... yes 26/220 ... yes 26/221 ... yes 29/222 ... yes 36/223 ... yes 36/224 ... yes 3/225 ... yes 36/226 ... yes 13/228 ... yes 36/231 ... yes 36/232 ... yes 36/233 ... yes 36/234 ... yes 36/235 ... yes 36/236 ... yes 36/237 ... yes 36/238 ... yes 36/239 ... yes 36/240 ... yes 26/241 ... yes 36/242 ... yes 29/244 ... yes 29/245 ... yes 37/246 ... yes 29/248 ... yes 29/250 ... yes 29/251 ... yes 40/253 ... yes 40/254 ... yes 13/256 ... yes 23/257 ... yes 23/258 ... yes 3/260 ... yes 26/261 ... yes 26/264 ... yes 36/265 ... yes 36/266 ... yes 26/267 ... yes 36/268 ... yes 36/270 ... yes 36/271 ... yes 36/272 ... yes 36/273 ... yes 36/274 ... yes 36/275 ... yes 36/276 ... yes 36/277 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.24.0 ? ... yes (2.28.0) Git user has default SSH configuration? ... yes Active users: ... 10 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Possible fixes
No idea