gitlab-ctl reconfigure fails because /var/opt/gitlab/gitlab-monitor is missing
Summary
Running gitlab-ctl reconfigure fails on Running handlers when the following gitlab.rb config options are used.
gitlab_exporter['enable'] = true
gitlab_exporter['home'] = "/var/opt/gitlab/gitlab-monitor"
Steps to reproduce
Set config options as below and run gitlab-ctl reconfigure.
gitlab_exporter['enable'] = true
gitlab_exporter['home'] = "/var/opt/gitlab/gitlab-monitor"
What is the current bug behavior?
files/gitlab-cookbooks/monitoring/recipes/gitlab-exporter.rb deletes gitlab_monitor home directory without checking gitlab.rb config.
This causes the gitlab-ctl reconfigure process to fail since the home directory no longer exits. The log_directory is not affected.
What is the expected correct behavior?
Cleanup task should only delete /var/opt/gitlab/gitlab-monitor
dir if that path is not explicitly defined in config.
Relevant logs and/or screenshots
Recipe: monitoring::gitlab-exporter
* directory[/var/opt/gitlab/gitlab-monitor] action create
- create new directory /var/opt/gitlab/gitlab-monitor
- change mode from '' to '0755'
- change owner from '' to 'git'
Recipe: <Dynamically Defined Resource>
* service[gitlab-monitor] action nothing (skipped due to action :nothing)
Recipe: monitoring::gitlab-exporter
* runit_service[gitlab-monitor] action disable
* ruby_block[disable gitlab-monitor] action run (skipped due to only_if)
(up to date)
* directory[/var/opt/gitlab/gitlab-monitor] action delete
- delete existing directory /var/opt/gitlab/gitlab-monitor
* directory[/var/log/gitlab/gitlab-exporter] action create (up to date)
* template[/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml] action create
* Parent directory /var/opt/gitlab/gitlab-monitor does not exist.
================================================================================
Error executing action `create` on resource 'template[/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml]'
================================================================================
Chef::Exceptions::EnclosingDirectoryDoesNotExist
------------------------------------------------
Parent directory /var/opt/gitlab/gitlab-monitor does not exist.
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/gitlab-exporter.rb
57: template "#{gitlab_exporter_dir}/gitlab-exporter.yml" do
58: source "gitlab-exporter.yml.erb"
59: owner gitlab_user
60: mode "0600"
61: notifies :restart, "service[gitlab-exporter]"
62: variables(
63: probe_sidekiq: node['monitoring']['gitlab-exporter']['probe_sidekiq'],
64: redis_url: redis_url,
65: connection_string: connection_string,
66: redis_enable_client: node['gitlab']['gitlab-rails']['redis_enable_client']
67: )
68: end
69:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/gitlab-exporter.rb:57:in `from_file'
template("/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml") do
action [:create]
default_guard_interpreter :default
source "gitlab-exporter.yml.erb"
declared_type :template
cookbook_name "monitoring"
recipe_name "gitlab-exporter"
mode "0600"
variables {:probe_sidekiq=>true, :redis_url=>#<URI::Generic unix:/var/opt/gitlab/redis/redis.socket>, :connection_string=>"dbname=gitlabhq_production user=gitlab host=/var/opt/gitlab/postgresql port=5432 password=", :redis_enable_client=>true}
owner "git"
group nil
path "/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml"
verifications []
end
System Info:
------------
chef_version=14.13.11
platform=ubuntu
platform_version=18.04
ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
There was an error running gitlab-ctl reconfigure:
template[/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml] (monitoring::gitlab-exporter line 57) had an error: Chef::Exceptions::EnclosingDirectoryDoesNotExist: Parent directory /var/opt/gitlab/gitlab-monitor does not exist.
Running handlers complete
Chef Client failed. 3 resources updated in 08 seconds
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 18.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.3.1-ee Revision: 699a4525856 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://gitlab.hds.colostate.edu HTTP Clone URL: https://gitlab.hds.colostate.edu/some-group/some-project.git SSH Clone URL: git@gitlab.hds.colostate.edu:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 10.0.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.0.0 ? ... OK (10.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/2 ... yes 4/3 ... yes 5/4 ... yes 3/5 ... yes 3/12 ... yes 6/14 ... yes 3/17 ... yes 3/18 ... yes 3/19 ... yes 3/20 ... yes 3/21 ... yes 3/24 ... yes 3/25 ... yes 3/26 ... yes 3/27 ... yes 3/30 ... yes 3/31 ... yes 3/32 ... yes 3/33 ... yes 3/34 ... yes 3/35 ... yes 3/36 ... yes 3/37 ... yes 3/38 ... yes 3/39 ... yes 3/41 ... yes 3/42 ... yes 3/43 ... yes 3/44 ... yes 3/45 ... yes 3/46 ... yes 3/48 ... yes 3/49 ... yes 3/50 ... yes 3/51 ... yes 3/52 ... yes 3/53 ... yes 3/54 ... yes 3/55 ... yes 3/56 ... yes 3/57 ... yes 3/58 ... yes 3/59 ... yes 3/60 ... yes 3/61 ... yes 3/62 ... yes 3/63 ... yes 3/64 ... yes 3/65 ... yes 3/66 ... yes 3/67 ... yes 3/68 ... yes 3/69 ... yes 3/71 ... yes 3/72 ... yes 3/73 ... yes 3/75 ... yes 3/76 ... yes 3/77 ... yes 3/78 ... yes 3/79 ... yes 3/80 ... yes 3/81 ... yes 3/82 ... yes 3/83 ... yes 3/84 ... yes 3/86 ... yes 3/87 ... yes 3/88 ... yes 3/89 ... yes 3/90 ... yes 3/91 ... yes 3/92 ... yes 3/93 ... yes 3/94 ... yes 3/95 ... yes 3/96 ... yes 3/97 ... yes 3/98 ... yes 3/99 ... yes 3/100 ... yes 3/101 ... yes 3/102 ... yes 3/103 ... yes 3/105 ... yes 3/107 ... yes 3/109 ... yes 3/110 ... yes 10/111 ... yes 3/112 ... yes 24/113 ... yes 3/114 ... yes 3/115 ... yes 5/117 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 16 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... yes (6.8.3)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
files/gitlab-cookbooks/monitoring/recipes/gitlab-exporter.rb
Current
directory '/var/opt/gitlab/gitlab-monitor' do
action :delete
recursive true
end
Possible Solution
if gitlab_exporter_dir != '/var/opt/gitlab/gitlab-monitor' then
directory '/var/opt/gitlab/gitlab-monitor' do
action :delete
recursive true
end
end