Redis Convergence Outage - 2017-05-08
On Monday May 8th, 2017 at 19:23 UTC any GitLab nodes running the `chef-client` service and a member of the `gitlab-cluster-base` role downloaded, installed, and reconfigured the services upon the next automated chef run due to a change in the package deployment being a security release.
The security release, because it is a reconfiguration of the debian package source, causes an install on all nodes at the next chef run that have GitLab installed. This would have effected the database servers too, however we manually stopped the chef-client service from running on them several weeks ago due to this similar thing happening.
We are investigating via https://gitlab.com/gitlab-com/infrastructure/issues/1717 a more efficient way to resolve this problem.
issue