Add chef-client disabler scripts to Chef-managed hosts

Production Change - Criticality 4 C4

Change Objective	Install the chef-client-disable and chef-client-enable scripts on Chef managed hosts
Change Type	ConfigurationChange
Services Impacted	None (adds utility scripts for chef-client)
Change Team Members	@msmiley
Change Criticality	C4
Change Reviewer or tested in staging	tested on staging environment (see #1898 (comment 317510726))
Due Date	2020-04-06 21:00 UTC (14:00 PDT)
Time tracking	30 minutes ( same to rollback )

Detailed steps for the change

Pre-condition

Merge the chef-repo MR to bump the version of cookbook gitlab-server to 1.8.1 in the production environment: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3065

Change steps

Run the apply_to_prod job of the pipeline for the MR mentioned above.

Validation steps

Manually run chef-client on any example Chef-managed host in the target environment. Then verify the scripts and symlinks are present in /usr/local/bin:

$ sudo chef-client
...

$ ls -l /usr/local/bin/chef-client-*
lrwxrwxrwx 1 root root   40 Apr  3 19:06 /usr/local/bin/chef-client-disable -> /usr/local/bin/chef-client-disabler-shim
-rwxr-xr-x 1 root root 6496 Apr  3 19:06 /usr/local/bin/chef-client-disabler-shim
-rwxr-xr-x 1 root root 5627 Apr  3 19:06 /usr/local/bin/chef-client-disabler-shim-test
lrwxrwxrwx 1 root root   40 Apr  3 19:06 /usr/local/bin/chef-client-enable -> /usr/local/bin/chef-client-disabler-shim

Optionally, run the acceptance tests.

$ chef-client-disabler-shim-test

Optionally, use the scripts as they would be run in practice, per the updated runbook:

# Disable chef-client.

$ chef-client-disable 'Testing chef-client-disable script, see issue https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1898'

# Show that periodic runs are disabled.

$ sudo systemctl is-active chef-client.service

$ sudo systemctl is-enabled chef-client.service

# Show that manual runs are disabled.

$ sudo chef-client

# Re-enable chef-client.

$ chef-client-enable

# Show that chef-client behavior is restored to normal.

$ sudo chef-client

Rollback steps

Because this change adds script that are only run manually by humans, if rollback is needed, it should not need to be rushed.

Revert the MR, and re-run chef-client.

If chef-client has been disabled using these scripts, it can be manually re-enabled as follows:

$ sudo rm -v /usr/bin/chef-client && sudo ln -s /opt/chef/bin/chef-client /usr/bin/chef-client
$ sudo systemctl enable chef-client.service
$ sudo systemctl start chef-client.service

Changes checklist

• Detailed steps and rollback steps have been filled prior to commencing work
• Person on-call has been informed prior to change being rolled out