Support locally disabling chef-client on a host
Problem statement
During an emergency, sometimes we need to temporarily prevent chef-client from running.
Our current practice is to disable its systemd unit:
$ sudo systemctl stop chef-client.service
This stops the automatic chef-client runs every 30 minutes, but by itself this is not sufficient.
We often explicitly run chef-client
as part of applying config changes or deployments, and stopping the systemd unit does not prevent this. Something that is normally considered safe and harmless becomes dangerous, and the caller has no context -- no clue about the risk or its consequences.
CORRECTION: We do have a documented 2nd step -- breaking chef-client my renaming its credentials/config directory. This is sufficient protection from accidental chef-client runs, but it does not give clear context for the change (when, by whom, why). So I'm still proceeding with this issue, as it makes chef-client give useful info when disabled, as opposed to just being broken. Documentation update: gitlab-com/runbooks!2080 (merged)
Potential solutions
Renaming the chef-client
executable in addition to disabling the systemd unit would prevent runs (which eliminates the risk), but it does not give any context to the caller.
Making a helper script to disable/enable chef-client could include giving the caller a descriptive message providing context as to why it's disabled and where to find more info (e.g. issue link, slack thread, etc.).
This could potentially look something like:
/usr/local/bin/chef-client-disable.sh -m "In-place changes to /etc/foo.conf, see: https://gitlab.com/..."
:
- Stops and disables chef-client systemd unit:
sudo systemctl stop chef-client
andsudo systemctl disable chef-client
- Overrides
/usr/bin/chef-client
with a shell script emitting the log message, its timestamp, and the user who logged it. This override might be accomplished by renaming chef-client or by manipulating /etc/alternatives or some other means.
/usr/local/bin/chef-client-enable.sh
- Starts and enables chef-client systemd unit:
sudo systemctl start chef-client
andsudo systemctl enable chef-client
- Restores the original
/usr/bin/chef-client
, and removes the log message associated with the disablement.