Skip to content

[gitlab-provisioner] Nightly pipeline fails due to SSH connectivity problems

Triage has shown that SSH connectivity fails to provisioned cluster nodes causing Ansible jobs to fail.

Originally logged as a problem with gitlab-ctl reconfigure with this noted failure:

As seen in https://gitlab.com/gitlab-org/distribution/gitlab-provisioner/-/jobs/517904488

Starting Chef Client, version 14.14.29
resolving cookbooks for run list: [gitlab-ee]
Synchronizing Cookbooks:
  - gitlab-ee (0.0.1)
  - package (0.1.0)
  - gitlab (0.0.1)
  - consul (0.1.0)
  - repmgr (0.1.0)
  - runit (4.3.0)
  - postgresql (0.1.0)
  - redis (0.1.0)
  - monitoring (0.1.0)
  - registry (0.1.0)
  - mattermost (0.1.0)
  - praefect (0.1.0)
  - letsencrypt (0.1.0)
  - nginx (0.1.0)
  - gitaly (0.1.0)
  - acme (4.1.1)
  - crond (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...

================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb
================================================================================

NameError
---------
uninitialized constant #<Class:Gitlab>::Undefined

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:105:in `block in from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:29:in `auto_vivify'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:105:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:22:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:20:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:

 98:  
 99:    def from_file(_file_path)
100:      # Throw errors for unrecognized top level calls (usually spelling mistakes)
101:      config_strict_mode true
102:      # Turn on node deprecation messages
103:      Gitlab::Deprecations::NodeAttribute.log_deprecations = true
104:      # Allow auto mash creation during from_file call
105>>     Gitlab::ConfigMash.auto_vivify { super }
106:    ensure
107:      config_strict_mode false
108:      Gitlab::Deprecations::NodeAttribute.log_deprecations = false
109:    end
110:  
111:    # Enhance set so strict mode errors aren't thrown as long as the setting is witin our defined config
112:    def internal_set(symbol, value)
113:      if configuration.key?(symbol)
114:        configuration[symbol] = value

System Info:
------------
chef_version=14.14.29
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
Running handlers complete
Edited by Robert Marshall