Chef bootstrapping is not a "one and done" and often fails on the first bootstrap run
When a patroni node (for example) initially runs its first Chef bootstrap, the iptables-persistent
package cannot be found, most likely due to an apt update
not being run. We should find where this happens and try to prevent it. As it is, this means a node takes an extra 30 minutes to provision, or having to stop and start the node to force Chef to run again.
Also, td_agent
seems to be unhappy with a ton of Ruby exceptions.
Example of the problem: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13324#note_571214265
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script:
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m Chef::Exceptions::Package#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: -------------------------#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: No candidate version available for iptables-persistent#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script:
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m Resource Declaration:#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: ---------------------#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: # In /var/chef/cache/cookbooks/iptables/recipes/_package.rb
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m 29: package 'iptables-persistent'
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m 30: end
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m Compiled Resource:#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: ------------------#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: # Declared in /var/chef/cache/cookbooks/iptables/recipes/_package.rb:29:in `from_file'
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m apt_package("iptables-persistent") do
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m package_name "iptables-persistent"
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m action [:install]
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m default_guard_interpreter :default
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m declared_type :package
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m cookbook_name "iptables"
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m recipe_name "_package"
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m end
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m System Info:#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: ------------#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: chef_version=14.13.11
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m platform=ubuntu
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m platform_version=16.04
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m ruby=ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux]
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m program_name=/usr/bin/chef-client
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m executable=/opt/chef/bin/chef-client#033[0m
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script:
May 12 18:35:57 patroni-v12-10-db-gprd GCEMetadataScripts[3545]: 2021/05/12 18:35:57 GCEMetadataScripts: startup-script: #033[0m#033[0m
Edited by Gerardo Lopez-Fernandez