Make it possible to run gitlab-ctl reconfigure without manual intervention after failure

If gitlab-ctl reconfigure fails for any reason, then the nodes data in /opt/gitlab/embedded/nodes/<HOSTNAME>.json is removed, and manual intervention is needed to recover.

As I discussed in #8534 (closed), this happens because:

  1. Chef trashes the /opt/gitlab/embedded/nodes dir when it first runs: https://github.com/chef/omnibus-ctl/blob/4e968f4cf713a545bbb70ee13b8ff7ebd37f7f3e/lib/omnibus-ctl.rb#L500
  2. Then Chef runs Chef::Node#find_or_create for the nodes file.
  3. Since it was trashed, the node info returns 404, so it creates a file with almost no info.
  4. If Chef crashes for some reason (e.g. CTRL-C, some recipe failure), Chef doesn't get a chance to run save_updated_node after converging to update the info.

We might be able to improve this restoring the state of /opt/gitlab/embedded/nodes upon failure.

Edited by Stan Hu