Standardize the process of investigating the diff between auto-deploy packages.
The problem
During incidents, a quite common request is to analyze if an incident was caused by a deployment. To investigate so, we need to find what application changes are being deployed or were deployed to a specific environment. We don't have a standard way for doing such investigation and it depends on where we're at in the deployment process (before, during, or after)
Some of the approaches we usually do are:
- Using
/chatops run auto_deploy statuscommand and use the diff link. This approach is not accurate if there's an ongoing deployment. - Finding the diff between environments by comparing the content of the package that is deployed and the package that we're investigating. This approach requires additional manual effort since we need to:
- Investigate the SHA of the deployed package, whether by looking into the #announcements Slack channel or in the Security environments page (e.g https://gitlab.com/gitlab-org/security/gitlab/-/environments).
- Grab the SHA of the package that we're investigating.
- Build a diff link and proportionate such link to engineers investigating the incident (e.g https://gitlab.com/gitlab-org/security/gitlab/compare/3e1d24dad25...2c06ed782b4)
- It gets more complicated if we want to find out the difference for satellite projects such as Gitaly, Omnibus and CNG
Let's improve this process by creating a standard and automated way to fetch the difference between auto-deploy packages. As a first iteration, we can limit this to only gitlab-org/gitlab diff and exclude satellite projects.
Solution
In gitlab-com/chatops!180 (merged) we added a Chatops command to check the difference between auto_deploy packages for gitlab-org/gitlab
Some ideas for possible future iterations:
- Add a diff link to the messages in the #announcements Slack channel or a diff when notifying RMs about an active incident while having a deployment in progress
- Include the diff link in the baking time message.
- Include the diff link in the Release Manager dashboard
- Built this option into our product (Probably a long term solution)