Remove file NFS mounts from gitlab.com

Procedure

Now that we have successfully been operating without NFS mounts on staging we would like to do the same on production.

• Remove nfs mounts on canary

knife ssh -e 'roles:gprd-base-fe-web-cny' 'sudo service chef-client stop'
knife ssh -e 'roles:gprd-base-fe-web-cny' 'for dir in $(sudo ls -d -1 /var/opt/gitlab/git-data-file*); do if [[ -d $dir ]]; then sudo umount $dir; fi; done'

• Verify everything is still working on canary
• Remove chef configuration for nfs mounts on production, merge and apply https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2471 (note that this will not remove existing mounts)
• Remove mounts on sidekiq

knife ssh -e 'roles:gprd-base-be-sidekiq' 'for dir in $(sudo ls -d -1 /var/opt/gitlab/git-data-file*); do if [[ -d $dir ]]; then sudo umount $dir; fi; done'

• Update the base role to remove nfs mounts

knife ssh -e 'roles:gprd-base' 'for dir in $(sudo ls -d -1 /var/opt/gitlab/git-data-file*); do if [[ -d $dir ]]; then sudo umount $dir; fi; done'

• Start chef on canary

knife ssh -e 'roles:gprd-base-fe-web-cny' 'sudo service chef-client start'

• Clean up old fstab entries since chef will not remove them, run on gstg (TBD)
• Clean up old fstab entries since chef will not remove them, run on gprd (TBD)

Merge and apply https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2473

Rolling back in case of problems

• Revert https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2471
• Update chef and force chef runs on the fleet