Stop deployment when deploying older package to environment

Summary

When a deployment to an environment starts, check the last successful deployment to the environment. If the current running deployment is older than the last deployment, then we should cancel the deployment.

This situation can happen when:

  1. An older version's pipeline gets delayed, and then continues after a newer pipeline has already deployed. In the following example, package 0620 is the older package, and package 1320 is the newer one.

    1. Package 0620 deployment to gstg-cny fails.
    2. Package 1320 deployment to gstg-cny succeeds.
    3. Package 1320 deployment to gstg succeeds.
    4. Package 0620 deployment to gstg-cny is retried, and passes.
    5. Package 0620 gets deployed to gstg, and so on.

  2. Another possibility is when we turn on the auto_deploy_tag_latest feature flag. If we turn off the feature flag before the pipeline on the latest commit has completed, the next run of the auto_deploy:tag job might tag an earlier commit. In this case, the version might be newer than the previous, but uses an older commit.

  3. During a rollback. This is the only valid case that should be allowed.

Proposal

  1. Call the API to compare commits, and compare the new commit being deployed and the commit already deployed.
  2. If the response.commits is empty, which means that there are no commits in the diff, raise an error. The commit list can be blank if we are comparing the same commit SHA, or if the source commit SHA is an ancestor (older) of the target commit SHA.
    1. Do not raise error if compare_same_ref is true, which means that we are comparing the same commit. This happens when a component has no changes to deploy.

This check should be done in deployer, after the prepare job that locks the environment. If the check is done before the environment is locked, it is possible for the check to pass while a newer deployment is running (because we compare with the latest successful deployment), and then continue with deploying the older package when the newer deployment completes.

Implementation

  • A job called *-check-package was added to the deployer pipeline (example: gstg-check-package).
  • The job raises an error if it finds that the package being deployed is older than the package already on the environment. This change is behind a feature flag: stop_deployment_if_old_package.
Edited by Reuben Pereira