Sometimes deployer deploys too quickly
Release-tools triggers tags often enough where we may sometimes overlap deploys that are still running on staging. Our deploys into staging also trigger QA jobs, which this portion of the pipeline will take approximately 50 minutes. This has a negative consequence that while QA is running for a prior deploy, a deploy has been triggered. This is problematic as the QA jobs running may not be testing the correct version of the application the job is intended to test.
|Deploy||Deploy Start Time (EST)||Status|
||14:52||post deploy migrations completed, QA starting|
||15:25||cancelled by Release Manager|
Notice that we triggered 2 deploys in a short time span. This by itself is not entirely a problem. The final deploy was cancelled to allow QA to get to a state where it can finish and test the version of the product for which it is expected to test.
I think we need to consider modifying deployer's pipeline to prevent multiple deploys from going out if one is still on-going. Let's use this issue to discuss the pro's and cons of this issue and take action if necessary.