How does Delivery prevent auto-deploy from building/deploy packages that are missing a target commit
Problem Statement
Let's say X
was rolled out in staging, and QA failed. Our now default behavior is to rollback staging to the last known good state (X-1
). At this point, a fix should be actively being worked on and merged, or the targeted commit reverted, etc. When the rollback is completed, auto-deploy doesn't know anything has happened, and when it comes around later to deploy X+1
to staging, there's a chance that this package may not yet contain the fix that is required that forced us to rollback.
Picks into auto-deploy are not guaranteed to be in the next package because the process of auto-deploy running and picking things have no connection, therefore there are no validations, outside of humans checking each package that is being prepared. How can we prevent auto-deploy from either deploying, or better yet, building, if the proposed package is missing a commit that contains a fix?
Options today include:
- pausing auto-deploy
- faking the lock mechanism on staging
Current Option Downfalls
Pausing Auto-deploy
This pauses everything. Picks won't happen, builds won't be triggered, etc. The best this can be used for, is waiting on green pipelines, manually picking, and triggering the build process. This is very error prone, requires a lot of watching and waiting and timing is crucial.
Faking the lock
Not all release managers have access to perform this action. It would involve modifying the chef role, setting the omnibus install flag to false
, which would then prevent a deploy from going out. The prepare
job simply waits and eventually times out. Other than the fact that this is something only SRE's are able to change, we wasted cycles by building a package which doesn't contain our fix.
Solutions
...
Milestones
-
Determine a method for which Release Managers can somehow mark a commit to be forced into the upcoming package. If it is not, a build will not occur