gitlab-ctl reconfigure fails because /var/opt/gitlab/gitlab-monitor is missing

Summary

Running gitlab-ctl reconfigure fails on Running handlers when the following gitlab.rb config options are used.

gitlab_exporter['enable'] = true
gitlab_exporter['home'] = "/var/opt/gitlab/gitlab-monitor"

Steps to reproduce

Set config options as below and run gitlab-ctl reconfigure.

gitlab_exporter['enable'] = true
gitlab_exporter['home'] = "/var/opt/gitlab/gitlab-monitor"

What is the current bug behavior?

files/gitlab-cookbooks/monitoring/recipes/gitlab-exporter.rb deletes gitlab_monitor home directory without checking gitlab.rb config.

This causes the gitlab-ctl reconfigure process to fail since the home directory no longer exits. The log_directory is not affected.

What is the expected correct behavior?

Cleanup task should only delete /var/opt/gitlab/gitlab-monitor dir if that path is not explicitly defined in config.

Relevant logs and/or screenshots

Recipe: monitoring::gitlab-exporter
  * directory[/var/opt/gitlab/gitlab-monitor] action create
    - create new directory /var/opt/gitlab/gitlab-monitor
    - change mode from '' to '0755'
    - change owner from '' to 'git'
Recipe: <Dynamically Defined Resource>
  * service[gitlab-monitor] action nothing (skipped due to action :nothing)
Recipe: monitoring::gitlab-exporter
  * runit_service[gitlab-monitor] action disable
    * ruby_block[disable gitlab-monitor] action run (skipped due to only_if)
     (up to date)
  * directory[/var/opt/gitlab/gitlab-monitor] action delete
    - delete existing directory /var/opt/gitlab/gitlab-monitor
  * directory[/var/log/gitlab/gitlab-exporter] action create (up to date)
  * template[/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml] action create
    * Parent directory /var/opt/gitlab/gitlab-monitor does not exist.
    ================================================================================
    Error executing action `create` on resource 'template[/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml]'
    ================================================================================
    
    Chef::Exceptions::EnclosingDirectoryDoesNotExist
    ------------------------------------------------
    Parent directory /var/opt/gitlab/gitlab-monitor does not exist.
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/gitlab-exporter.rb
    
     57: template "#{gitlab_exporter_dir}/gitlab-exporter.yml" do
     58:   source "gitlab-exporter.yml.erb"
     59:   owner gitlab_user
     60:   mode "0600"
     61:   notifies :restart, "service[gitlab-exporter]"
     62:   variables(
     63:     probe_sidekiq: node['monitoring']['gitlab-exporter']['probe_sidekiq'],
     64:     redis_url: redis_url,
     65:     connection_string: connection_string,
     66:     redis_enable_client: node['gitlab']['gitlab-rails']['redis_enable_client']
     67:   )
     68: end
     69: 
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/gitlab-exporter.rb:57:in `from_file'
    
    template("/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml") do
      action [:create]
      default_guard_interpreter :default
      source "gitlab-exporter.yml.erb"
      declared_type :template
      cookbook_name "monitoring"
      recipe_name "gitlab-exporter"
      mode "0600"
      variables {:probe_sidekiq=>true, :redis_url=>#<URI::Generic unix:/var/opt/gitlab/redis/redis.socket>, :connection_string=>"dbname=gitlabhq_production user=gitlab host=/var/opt/gitlab/postgresql port=5432 password=", :redis_enable_client=>true}
      owner "git"
      group nil
      path "/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml"
      verifications []
    end
    
    System Info:
    ------------
    chef_version=14.13.11
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

template[/var/opt/gitlab/gitlab-monitor/gitlab-exporter.yml] (monitoring::gitlab-exporter line 57) had an error: Chef::Exceptions::EnclosingDirectoryDoesNotExist: Parent directory /var/opt/gitlab/gitlab-monitor does not exist.

Running handlers complete
Chef Client failed. 3 resources updated in 08 seconds

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		Ubuntu 18.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.22.0
Sidekiq Version:5.2.7
Go Version:	unknown


GitLab information
Version:	12.3.1-ee
Revision:	699a4525856
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	10.9
URL:		https://gitlab.hds.colostate.edu
HTTP Clone URL:	https://gitlab.hds.colostate.edu/some-group/some-project.git
SSH Clone URL:	git@gitlab.hds.colostate.edu:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	10.0.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 10.0.0 ? ... OK (10.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/2 ... yes
4/3 ... yes
5/4 ... yes
3/5 ... yes
3/12 ... yes
6/14 ... yes
3/17 ... yes
3/18 ... yes
3/19 ... yes
3/20 ... yes
3/21 ... yes
3/24 ... yes
3/25 ... yes
3/26 ... yes
3/27 ... yes
3/30 ... yes
3/31 ... yes
3/32 ... yes
3/33 ... yes
3/34 ... yes
3/35 ... yes
3/36 ... yes
3/37 ... yes
3/38 ... yes
3/39 ... yes
3/41 ... yes
3/42 ... yes
3/43 ... yes
3/44 ... yes
3/45 ... yes
3/46 ... yes
3/48 ... yes
3/49 ... yes
3/50 ... yes
3/51 ... yes
3/52 ... yes
3/53 ... yes
3/54 ... yes
3/55 ... yes
3/56 ... yes
3/57 ... yes
3/58 ... yes
3/59 ... yes
3/60 ... yes
3/61 ... yes
3/62 ... yes
3/63 ... yes
3/64 ... yes
3/65 ... yes
3/66 ... yes
3/67 ... yes
3/68 ... yes
3/69 ... yes
3/71 ... yes
3/72 ... yes
3/73 ... yes
3/75 ... yes
3/76 ... yes
3/77 ... yes
3/78 ... yes
3/79 ... yes
3/80 ... yes
3/81 ... yes
3/82 ... yes
3/83 ... yes
3/84 ... yes
3/86 ... yes
3/87 ... yes
3/88 ... yes
3/89 ... yes
3/90 ... yes
3/91 ... yes
3/92 ... yes
3/93 ... yes
3/94 ... yes
3/95 ... yes
3/96 ... yes
3/97 ... yes
3/98 ... yes
3/99 ... yes
3/100 ... yes
3/101 ... yes
3/102 ... yes
3/103 ... yes
3/105 ... yes
3/107 ... yes
3/109 ... yes
3/110 ... yes
10/111 ... yes
3/112 ... yes
24/113 ... yes
3/114 ... yes
3/115 ... yes
5/117 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... yes (2.22.0)
Git user has default SSH configuration? ... yes
Active users: ... 16
Is authorized keys file accessible? ... yes
Elasticsearch version 5.6 - 6.x? ... yes (6.8.3)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

files/gitlab-cookbooks/monitoring/recipes/gitlab-exporter.rb

Current

directory '/var/opt/gitlab/gitlab-monitor' do
  action :delete
  recursive true
end

Possible Solution

if gitlab_exporter_dir != '/var/opt/gitlab/gitlab-monitor' then
  directory '/var/opt/gitlab/gitlab-monitor' do
    action :delete
    recursive true
  end
end
Edited by Joshua Eli Swick