Trigger separate deployments for each environment

Proposal

Release tools should trigger each environment deployment as a distinct job.

To roll out this change safely we will continue triggering deployer on master branch with the old multi-env deployment and in parallel, we will trigger the new deployments to the next-gen branch that defaults into check_mode.

Here follow a brief diagram of the new pipeline:

stateDiagram-v2
   [*] --> tag
    state fork_state <<fork>>
    tag --> fork_state
    fork_state --> wait_cng
    fork_state --> wait_omnibus

    state join_state <<join>>
    wait_cng --> join_state
    wait_omnibus --> join_state

    join_state --> trigger_gstg

    join_state --> trigger_complete_deployment
    note right of trigger_complete_deployment : already implemented up to here

    trigger_gstg --> trigger_cny

    state fork_state_cny <<fork>>
    trigger_cny --> fork_state_cny
    fork_state_cny --> baking_time
    fork_state_cny --> manual_promotion
    state join_state_cny <<join>>
    baking_time --> join_state_cny
    manual_promotion --> join_state_cny
    note right of baking_time : no longer a trigger
    note right of manual_promotion : no longer a trigger

    join_state_cny --> trigger_gprd

    trigger_gprd --> [*]

Baking time and manual promotions will be moved to release-tools as well (both triggers release-tools from deployer) as they are only available during a deployer multi-env pipeline.

We should evaluate if we can make use of multi-project pipeline status mirroring instead of active waiting.

Exit criteria

Individual deployments for staging, canary and production should be triggered by release tools

To do

  • Evaluate whether multi-project pipeline status mirroring can be used instead of active waiting
    • Since the next stage jobs are skipped if the child jobs are retried, active waiting will be used in this iteration, see #1578 (comment 520398890) for more details
  • Trigger a separate deployment for gstg
  • Trigger a separate deployment for cny
  • Baking time moved to release-tools
  • Trigger a separate deployment for production
  • Add a manual build to trigger production checks and then trigger a production deployment.
  • Testing
  • Create follow-ups

Implementation steps

Testing

Process

  • Prepare strategy
    • Currently, TRIGGER_REF env variable is used to indicate which deployer branch we use to perform deployments.
    • master is used for legacy deployments (gprd, gprd-cny, gstg) and next-gen is used for individual deployments
    • For testing we need to switch the deployer branch: master will be used for individual deployments and next-gen will be used for legacy deployments
    • MR with the implementation gitlab-org/release-tools!1423 (merged)
  • Cherry-pick a commit in the auto-deploy branch or create a new auto-deploy branch. Commit cherry-picked d5857130d59f3d7751d9c6f9471107148bab53d5
  • Add INDIVIDUAL_DEPLOYMENTS environment to release-tools
  • Ensure release-tools pipeline is generated correctly
  • Ensure the individual deployments to production are executed correctly.

Notes

  • First round of testing went partially successful. We missed some notifications due to a missing environment variable
  • Second round of testing was successful #1578 (comment 556457618). Minor follow ups to be created:
    • Rename Pipeline: deployer to Pipeline: Release Tools
    • Ignore gprd-checks when triggering an individual deployment
    • Remove INDIVIDUAL_DEPLOYMENTS environment variable

What to do if something goes wrong?

Development log

Follow ups

Edited by Mayra Cabrera