Chef-client fails on Patroni DR nodes
UPDATE: Apparently we have different problems on patroni-01-db-dr and patroni-03-db-dr. The following describes only patroni-03-db-dr:
@mwasilewski-gitlab discovered that chef-client runs are failing on some of the Patroni nodes in the DR environment.
This has been failing for at least a week, probably much longer, as the error relates to the /etc/chef/ directory being missing.
Example output of failed chef-client run:
Dec 18 17:10:26 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:26+00:00] INFO: Forking chef instance to converge...
Dec 18 17:10:26 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:26+00:00] INFO: *** Chef 12.22.5 ***
Dec 18 17:10:26 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:26+00:00] INFO: Platform: x86_64-linux
Dec 18 17:10:26 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:26+00:00] INFO: Chef-client pid: 39997
Dec 18 17:10:26 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:26+00:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:28+00:00] INFO: Client key /etc/chef/client.pem is not present - registering
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: #033[0m
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: ================================================================================#033[0m
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: #033[31mChef encountered an error attempting to create the client "patroni-03-db-dr.c.gitlab-dr.internal"#033[0m
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: ================================================================================#033[0m
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: #033[0mSystem Info:#033[0m
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: ------------#033[0m
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: chef_version=12.22.5
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: #033[0mruby=ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: #033[0mprogram_name=chef-client worker: ppid=53394;start=17:10:26;
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: #033[0mexecutable=/opt/chef/bin/chef-client#033[0m
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: #033[0m[2019-12-18T17:10:28+00:00] ERROR: Running exception handlers
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:28+00:00] ERROR: Exception handlers complete
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:28+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:28+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:28+00:00] ERROR: I can't write your private key to /etc/chef/client.pem - check permissions?
Dec 18 17:10:28 patroni-03-db-dr chef-client[53394]: [2019-12-18T17:10:28+00:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)And the corresponding trace file mentioned in the above output:
msmiley@patroni-03-db-dr.c.gitlab-dr.internal:~$ sudo cat /var/chef/cache/chef-stacktrace.out ; echo
Generated at 2019-12-18 17:10:28 +0000
Chef::Exceptions::CannotWritePrivateKey: I can't write your private key to /etc/chef/client.pem - check permissions?
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/api_client/registration.rb:74:in `assert_destination_writable!'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/api_client/registration.rb:54:in `run'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/client.rb:669:in `register'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/client.rb:267:in `run'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application.rb:295:in `block in fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application.rb:283:in `fork'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application.rb:283:in `fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application.rb:248:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application.rb:236:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application/client.rb:450:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application/client.rb:450:in `interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application/client.rb:434:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/lib/chef/application.rb:59:in `run'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.22.5/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:57:in `load'
/usr/bin/chef-client:57:in `<main>'And to confirm, there is no /etc/chef directory (although there is an unexpected /etc/chef.patroni directory).
msmiley@patroni-03-db-dr.c.gitlab-dr.internal:~$ ls -ld /etc/chef
ls: cannot access '/etc/chef': No such file or directory
msmiley@patroni-03-db-dr.c.gitlab-dr.internal:~$ ls -l /etc/chef.patroni/
total 8
-rw------- 1 root root 1679 Feb 14 2019 client.pem
-rw-r--r-- 1 root root 196 Feb 14 2019 client.rbEdited by Matt Smiley