release-tools production checks are difficult to bypass
Problem Statement
In the case where we check for production issues or changes, if there's an on-going high severity or change that blocks the promotion, we'll fall back to asking for permission. At this point, if the current on-call team member deems safe for us to proceed, we must go into the CI Variable settings, set IGNORE_PRODUCTION_CHECKS to true, and play the job.
Now that the job was able to be played, we must go back and remove that variable.
Without this variable set, the job will always fail, as-is its current intent.
This is more work than necessary to allow a promotion to proceed.
Solution
Extending the deploy command to support --ignore-promotion-checks=reason. This will trigger a regular deployment setting IGNORE_PRODUCTION_CHECKS=reason.
Modify deploy-tooling bin/promote script to send IGNORE_PRODUCTION_CHECKS to release-tools so that we can log why the checks where ignored in the monthly issue. (fixing #944 (closed) as well)
Other proposals discussed
Proposal 1: Add an extra label
Add a label for us to check that allows us to skip the validation on that one issue alone. For example, creating a label of allow-deploy could be placed on an issue. If another issue shows up blocking the deploy without that label, the deploy should still be blocked.
This has the benefit that we now have an audit of what issues we ran into during the promotion phase, and we then have the label on the issue that blocked the promotion and an audit of that label being added by someone indicating that a deploy is still safe option despite the severity of the issue that is still on-going.
Proposal 2: Add a new ChatOps command
Have a ChatOps command to set and unset the needed environment variables. This removes the manual intervention from Release Managers and makes it less prone to errors
Without this, we rely on our manual messaging, which again, removes automation that we are driving towards with full CD pipelines.