Test and rollout the independent post-deploy pipeline

As part of &585 (closed), the behavior of the independent post-deploy pipeline needs to be thoroughly tested before introducing it into release activities.

Exit criteria

  • Post-deploy migrations are executed on staging and production
  • The post-deploy pipeline doesn't interfere with the staging and production deployments.
  • Release managers are aware when the post-deploy pipeline is executed
  • Evidence of the post-migrations executed through the post-deploy pipeline is available
  • The execution of the post migrations is tracked across the database environments

Testing

Testing is based on the exit criteria and split into two phases:

  1. Phase 1: Initial testing to verify the configuration of the post-deploy pipeline
  2. Phase 2: Execution of post migrations in staging and production
  3. Phase 3: Rollout the post-deploy pipeline

✅ Phase 1: Initial testing

Details

Setup

  • Prepare release-tools to process and return fake post-migrations (this will prevent us waiting for actual post-migrations that can be time-consuming) gitlab-org/release-tools!1890 (merged)
  • For each of these scenarios notify release managers

Tests

Without pending post-migrations

  • Ensure there are no pending post-migrations
  • Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
  • Verify the prepare job fails since there are no post-deploy migrations

With an ongoing staging deployment

  • Ensure there are no pending post-migrations
  • Enable the fake_pending_post_migrations feature flag
  • Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
  • Verify the prepare job is canceled and a message with the ActiveGstgDeployment is sent to Slack

With an ongoing production deployment

  • Enable the fake_pending_post_migrations feature flag
  • Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
  • Verify the prepare job fails since there is an ongoing production deployment

With an active incident

Lists the post-migrations to be executed

  • Ensure there are no pending post-migrations
  • Enable the fake_pending_post_migrations feature flag
  • Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
  • Prevent post-migrations to be executed, e.g. cancel the jobs from the post_deploy_migrations:gstg bridge.
  • Verify a note has been added to release task issue with the list of pending post-migrations gitlab-org/release/tasks#3914 (comment 994680430)
  • Add a comment to the note in the release task indicating it's part of the testing.

✅ Phase 2: Execution of post migrations in staging and production

Details

Tests

Standard execution for staging and production

Jun 28th details

Results on #2352 (comment 1009830702)

Setup:

For staging:

  • Execute the pending post-deploy pipeline - /chatops run post_deploy_migrations execute
  • Verify a slack message is sent to notify the execution of the post-migration pipeline
  • Verify a message is posted in the release issue gitlab-org/release/tasks#4064 (comment 1009802310)
  • Tracking
    • Ensure a running product deployment is created
    • Verify a running deployment is tracked in Canonical and Security for db/gstg
  • Ensure the post-migrations are executed in staging.
  • Verify a slack message is sent notifying the successful execution
  • Tracking
    • Verify a success deployment is tracked in Canonical and Security
    • Verify a success product deployment is created
    • Verify the merge requests environment have been updated with db/gstg
    • Verify the label workflow::post-deploy-db-staging has been added.

For production:

  • Verify a slack message is sent to notify the execution of the post-migration pipeline
  • Tracking
    • Ensure a running product deployment is created
    • Verify a running deployment is tracked in Canonical and Security for db/gprd
  • Ensure the post-migrations are executed in production
  • Verify a slack message is sent notifying the successful execution
  • Tracking
    • Verify a success deployment is tracked in Canonical and Security
    • Verify a success product deployment is created
    • Verify the merge requests environment have been updated with db/gprd
    • Verify the label workflow::post-deploy-db-production has been added.

Clean up/rollback

  • Continue the execution of the coordinated pipeline

Setup:

For staging:

For production:

  • Verify a slack message is sent to notify the execution of the post-migration pipeline
  • Tracking https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/7462597
    • Ensure a running product deployment is created
    • Verify a running deployment is tracked in Canonical and Security for db/gprd
  • Ensure the post-migrations are executed in production https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/7462606
  • Verify a slack message is sent notifying the successful execution
  • Tracking
    • Verify a success deployment is tracked in Canonical and Security
    • Verify a success product deployment is created
    • Verify the merge requests environment have been updated with db/gprd
    • Verify the label workflow::post-deploy-db-production has been added.

Clean up/rollback

  • Continue the execution of the coordinated pipeline

When a post-migration fails in staging and production

Setup:

For staging:

  • Cancel the child job generated by the post_deploy_migrations:gstg bridge job
  • Verify a slack message is sent notifying the failed execution of the post migrations in staging
  • Retry the child job from the post_deploy_migrations:gstg bridge job
  • Verify a successful slack message is sent

For production:

  • Cancel the child job generated by the post_deploy_migrations:gprd bridge job
  • Verify a slack message is sent notifying the failed execution of the post migrations in gprd
  • Retry the child job from the post_deploy_migrations:gstg bridge job
  • Verify a successful slack message is sent

Quality is executed after the post-migrations are triggered on staging

Setup:

For staging:

Phase 3: Rollout of the post-deploy pipeline

Preparation

Rollout

  • Ensure the post-deploy pipeline is executed at least once a day by EMEA or AMER release manager.
  • Observe the behavior.

If a week has passed with no major problems, the rollout has been completed.

Handling post-migrations failures.

Clean up

After the testing has been completed:

Edited by Mayra Cabrera