Make it possible to run gitlab-ctl reconfigure without manual intervention after failure
If gitlab-ctl reconfigure fails for any reason, then the nodes data in /opt/gitlab/embedded/nodes/<HOSTNAME>.json is removed, and manual intervention is needed to recover.
As I discussed in #8534 (closed), this happens because:
- Chef trashes the
/opt/gitlab/embedded/nodesdir when it first runs: https://github.com/chef/omnibus-ctl/blob/4e968f4cf713a545bbb70ee13b8ff7ebd37f7f3e/lib/omnibus-ctl.rb#L500 - Then Chef runs
Chef::Node#find_or_createfor the nodes file. - Since it was trashed, the node info returns 404, so it creates a file with almost no info.
- If Chef crashes for some reason (e.g. CTRL-C, some recipe failure), Chef doesn't get a chance to run
save_updated_nodeafter converging to update the info.
We might be able to improve this restoring the state of /opt/gitlab/embedded/nodes upon failure.
Edited by Stan Hu