[gitlab-provisioner] Nightly pipeline fails due to SSH connectivity problems
Triage has shown that SSH connectivity fails to provisioned cluster nodes causing Ansible jobs to fail.
Originally logged as a problem with gitlab-ctl reconfigure
with this noted failure:
As seen in https://gitlab.com/gitlab-org/distribution/gitlab-provisioner/-/jobs/517904488
Starting Chef Client, version 14.14.29
resolving cookbooks for run list: [gitlab-ee]
Synchronizing Cookbooks:
- gitlab-ee (0.0.1)
- package (0.1.0)
- gitlab (0.0.1)
- consul (0.1.0)
- repmgr (0.1.0)
- runit (4.3.0)
- postgresql (0.1.0)
- redis (0.1.0)
- monitoring (0.1.0)
- registry (0.1.0)
- mattermost (0.1.0)
- praefect (0.1.0)
- letsencrypt (0.1.0)
- nginx (0.1.0)
- gitaly (0.1.0)
- acme (4.1.1)
- crond (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb
================================================================================
NameError
---------
uninitialized constant #<Class:Gitlab>::Undefined
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:105:in `block in from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:29:in `auto_vivify'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:105:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:22:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:20:in `from_file'
Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:
98:
99: def from_file(_file_path)
100: # Throw errors for unrecognized top level calls (usually spelling mistakes)
101: config_strict_mode true
102: # Turn on node deprecation messages
103: Gitlab::Deprecations::NodeAttribute.log_deprecations = true
104: # Allow auto mash creation during from_file call
105>> Gitlab::ConfigMash.auto_vivify { super }
106: ensure
107: config_strict_mode false
108: Gitlab::Deprecations::NodeAttribute.log_deprecations = false
109: end
110:
111: # Enhance set so strict mode errors aren't thrown as long as the setting is witin our defined config
112: def internal_set(symbol, value)
113: if configuration.key?(symbol)
114: configuration[symbol] = value
System Info:
------------
chef_version=14.14.29
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
Running handlers complete
Edited by Robert Marshall