Consider using the GitLab API to update chef-repo after successful deploys
Recently it was discovered that the chef server had a missing key: default_attributes.omnibus-gitlab.package.version
for the DR environment.
We had also discovered that there were differing versions of the omnibus package installed across the fleet that were out of the ordinary. The current suspicion is that what's in chef-repo (file: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/dr-omnibus-version.json) was committed to the chef server which wipes out the aforementioned key. This has the potential to create an inconsistency on some servers and even worse, if servers are rebuilt they'll also get an incorrect version.
In the case above, we had differing versions of 11.9 RC's spread throughout the DR servers.
Internal Slack Conversation: https://gitlab.slack.com/archives/C8PKBH3M5/p1555615490026500
Utilize this issue to come up with a plan of action to help prevent this type of situation in the future.