Prevent /runners chatops command from enabling chef-client on nodes that have specifically turned it off outside of normal deployment process
Overview
When someone wants to disable a deployment they run /runner run stop private green, at the moment this will disable chef-client from running as we can see below
steve@runners-manager-shared-blue-4.c.gitlab-ci-155816.internal:~$ chef-client-is-enabled
chef-client is currently disabled. To re-enable, run: chef-client-enable
Periodic runs via systemd: inactive and disabled on reboot.
Latest log entry from: /var/log/chef-client-disabler-shim.log
2021-12-15 17:44:45 UTC LOG (sa_111119524038469528205) Successfully disabled chef-client. Comment: Disable chef until the next deployment
When someone runs /runner run start private green this will enable chef-client once again, which is expected,
Problem
This can be problematic, for example, imagine someone is troubleshooting one of the nodes in the green deployment in the private shard and they disable chef-client. Whilst another person unaware will run /runner run start private green this will ignore the other developer work from disabling chef-client and start chef again which might result into a strange state.
We should add some checks in /runner run start private green and see if chef was disabled by deployer or something else.