Remove the blocking nature of post-deploy migrations
## Problem Statement
Post-migration execution is tied to our deployment process, which has the following disadvantages:
* Post-migrations are blockers for rollbacks, they can’t be rolled back due to the nature of their operations, when a package includes a post-migration and an incident needs to be fixed, we need to either roll forward or perform a hot-patch. These options have their disadvantages: preparing and deploying a merge request can take up to 6h, and hot-patching blocks auto-deploy processes, for reference rolling back a package only takes \~1h.
* Post-migrations are lengthy. The post-migration job has a timeout of 10 hours to fit all the possible operations a post-migration can execute. During this time the database can be impacted, on-call SREs are often paged about long-running transactions and other performance-related issues could be seen.
See https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1978 for more context and https://youtu.be/MCEI7b1RVxw for a general overview of this epic.
## Proposal
Execute post-deploy migrations independently from the coordinated pipeline by creating an autonomous post-deploy migration pipeline that can be triggered at the discretion of the Release Managers.
## DRI
@mayra-cabrera
## Exit criteria
* Post-migrations will be dissociated from the coordinated pipeline making all packages suitable to rollback if needed
* Post-migrations pipeline is executed manually at least once a day.
* Post-migrations need to be executed in GitLab.com to be included in the monthly release.
* Development velocity is not affected, the process for creating and merging an MR with a post-migration stays the same.
## Plan
- [Issue board](https://gitlab.com/groups/gitlab-com/gl-infra/-/boards/3312784?label_name[]=post-deploy%20migrations%3A%3Aphase%201)
Issues of this epic are split in three phases:
* **:white_check_mark: 1. Phase 1**: Implementation of the post-deploy pipeline
* **:white_check_mark: 2. Phase 2**: Adjust of the monthly and rollback process to consider the pipeline
* **:white_check_mark: 3. Phase 3**: Wrap up, documentation and announcements
To make use of the post-deploy pipeline, we require phase 1 and phase 2 to be completed.
#### :white_check_mark: Phase 1: Implementation of the post deploy pipeline
| Status | Issue | Link |
| ------ | ------ | ------ |
| :white_check_mark: | **Add visibility about pending post-migrations** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2045 |
| :white_check_mark: | **Allow Release Managers to identify which package introduced a specific post-migration** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2224 |
| :white_check_mark: | **Build a slack notification that informs the status of the post-deployment pipeline** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2047 |
| :white_check_mark: | **Build the independent post-deploy migration pipeline** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2046 |
| :white_check_mark: | **Post a message on the release issue when post-deploy pipeline is executed** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2346 |
| :white_check_mark: | **Make the list of pending post-deploy migrations available to release-tools** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2361 |
| :white_check_mark: | **Include post-migrations to be executed in the release task issue** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2362 |
| :white_check_mark: | **Only execute the post-deploy pipeline if there are pending post-deploy migrations** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2364 |
| :white_check_mark: | **Allow the post-deploy pipeline to be executed through ChatOps** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2376 |
#### :white_check_mark: Phase 2 - Adjust Release and rollback processes:
| Status | Issue | Link |
| ------ | ------ | ------ |
| :white_check_mark: | **Track the execution of the last post-deploy pipeline migration** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2363 |
| :white_check_mark: | **Extend the rollback logic to consider the post-deploy pipeline** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2365 |
| :white_check_mark: | **Adjust the monthly release process to consider the new post-deployment pipeline** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2049 |
| :white_check_mark: | **Adjust the patch release process to consider the post-deploy pipeline** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2415 |
| :white_check_mark: | **Adjust the security release process to consider the post-deploy pipeline** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2479 |
| :white_check_mark: | **Test and rollout the independent post-deploy pipeline** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352 |
| :white_check_mark: | **Communicate to Engineering and Infrastructure departments** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2478 |
| :white_check_mark: | **Add a runbook that explains how to deal with post-migrations failures** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2335 |
### :white_check_mark: Phase 3 - Final steps
| Status | Issue | Link |
| ------ | ------ | ------ |
| :white_check_mark: | **Remove the post-deploy jobs from the coordinated pipeline** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2353 |
| :white_check_mark: | **Remove post-migrations from deployer pipeline triggered by ChatOps** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2301 |
| :white_check_mark: | **Allow engineers to verify if a post-migration has been executed in GitLab.com** | https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2052 |
### Label admin
Using the following labels when creating issues for this Epic:
```
/label ~"post-deploy migrations::phase 1" ~AutoDeploy ~"team::Delivery" ~"Delivery::P4"
/epic &585
```
### Follow-ups
<details><summary>Expand for details</summary>
**Issues to be included in the [the second phase](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1978#phase-2-make-the-execution-of-post-deploy-migrations-smarter-option-d):**
* [#2136 - Post Deploy migrations safety indicator rollback](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2136).
* [Classify post-deploy migrations based on their nature](https://gitlab.com/gitlab-org/gitlab/-/issues/346604)
**Technical debt / cleanup**
* [#2490 - Move the detection of pending post migrations to the main stage](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2490)
* [#587 - Annotate pre and post-deployment migrations in Grafana](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/587)
* [#2417 - Expand the incident template to include the execution of post-migrations](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2417)
**Nice to have**
* [#2416 - Add the post-migration diff in the Slack threads](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2416)
* [#2334 - Build a Prometheus alert based on the pending post-deploy pipelines](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2334)
* [#2357 - Schedule post-deployment migrations as a job on k8s](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2357)
* [#2498 - Number of pending migrations on release manager dashboard considers main and ci databases](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2498)
* [#2504 - Proposal: Have a separate set of scoped labels for post-deployment migrations](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2504)
:white_check_mark: Completed
* [#2330 - Prevent duplicate list of post-deploy migrations from being posted to monthly release issue](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2330)
* [#2336 - Remove post_deploy_migrations_note feature flag](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2336)
* [#2377 - Consider removing the list of post-deploy migrations included in each package](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2377)
* [#2230 - Build a mechanism that reminds RM about executing post-deployment pipeline](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2230)
* [#2422 - Adjust the post-migration information sent by deployer to release-tools](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2422)
* [#2484 - Message posted by PDM on release issue contains duplicated entries](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2484)
* [#2470 - Update releases handbook page after post-deploy migrations move off auto-deploys](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2470)
* [#2423 - Ensure the pending migrations gauge metric is providing accurate information](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2423)
* [#2456 - Canceling the post-deploy pipeline prevents logs from being shown.](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2456)
* [#2490 - release-tools pipeline runs even when there are no pending migrations](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2489)
* [#2354 - Remove the deploy_version dependency from the post-deployment classes](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2354)
* [#2480 - Remove old rollback logic](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2480)
* [#2464 - The post-deploy pipeline tracks all merge requests regardless if they contain post-migrations](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2464)
* [#2494 - Proposal: Remove QA staging jobs from the coordinated pipeline](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2494)
* [#2476 - Update post-deploy migration references in the GitLab documentation](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2476)
</details>
### Development log
<details><summary>Expand for details</summary>
* **2022-02-10** - Broad plan has been adjusted based on the latest progress of https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/608. First four issues are ready to be picked up.
* **2022-03-15** - Visibility of pending post-migrations has been added to the [release manager](https://dashboards.gitlab.net/d/delivery-release_management/delivery-release-management?orgId=1&refresh=5m&from=now-6h&to=now) dashboard. Next step is to assist Release Managers to find out which [package introduced a specific post-migration](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2224).
* **2022-03-23** - Efforts for [adding visibility](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2224) to pending post-deploy migrations are still in-progress. At the same time, the implementation of the independent [post-deploy-pipeline](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2046) has started.
* **2022-04-08** - The implementation of the independent post-deployment pipeline is on review https://gitlab.com/gitlab-org/release-tools/-/merge_requests/1747/. After it's merged, the next step would be to test it before including its execution on the release manager's daily activities.
* **2022-04-15** - The implementation of the independent post-deployment pipeline is on review https://gitlab.com/gitlab-org/release-tools/-/merge_requests/1747/. After it's merged, the next step would be to test it before including its execution on the release manager's daily activities.
* **2022-04-22** - Initial [implementation of the post-deploy migration](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2046) was completed. The [current information available about post-deploy migrations](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2356) was evaluated, to make the post-deploy pipeline compliant with the release processes we require to know the [post-migrations executed in each pipeline run](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2362) and to establish a rollback window by [tracking the auto-deploy package](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2363) at the moment of the post-migration pipeline execution. To account for this scope, issues were opened and the epic was updated. Next step is to focus on making the [list of pending post-migrations](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2361) available to release-tools.
* **2022-04-29** - Work is in progress to make sure the pending [post-migrations information is available](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2361) to our tooling. Following the coordinated pipeline logic, an [initial note](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2346) is being implemented for the post-deploy pipeline, this one will serve as a historic record of the post migrations executions. Next step is to expand the initial note to include the [post-migrations to be executed](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2362) and to [track the execution of the post-deploy pipeline](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2363) to define a rollback window.
* **2022-05-23** - Initial tests to ensure release-tools receives the list of pending post-migrations were performed, as a result some modifications are being implemented to the deployer configuration, [details on the issue](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2361#development-log). Next step is to expand the initial note to include the [post-migrations to be executed](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2362) and to [track the execution of the post-deploy pipeline](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2363) to define a rollback window.
* **2022-05-25** - Implementation details for the post-deploy pipeline are in progress, including [adding a note on the release monthly issue](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2362) and [limiting the execution of the pipeline only if there are pending post-migrations](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2364). Next step would be to [allow the post-deploy pipeline to be executed through ChatOps](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2376) and to [track the execution of the post-deploy pipeline](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2363) to define a rollback window.
* **2022-06-03** - Implementation details for the post-deploy pipeline, including [adding a note on the release monthly issue](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2362), were completed. Next steps include [allowing the post-deploy pipeline to be executed through ChatOps](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2376) and [tracking the execution of the post-deploy pipeline](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2363) to define a rollback window. Both issues are necessary to start the [post-deploy pipeline testing](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352).
* **2022-06-10** - [Allowing the post-deploy pipeline to be executed through ChatOps](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2376) and [tracking the execution of the post-deploy pipeline](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2363) to define a rollback window are in progress. After the completion of these issues, the next step will be to start and define the [post-deploy pipeline testing](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352).
* **2022-06-23** - The 1st phase of [post-deploy pipeline testing](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352) has started. The 1st phase of testing verifying configuration and pre-checks was completed with positive results (details [here](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352#note_994717662) and [here](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352#note_999893690)). The next step is to proceed with the 2nd phase of testing to verify the execution of post-migrations through the post-deploy pipeline.
* **2022-07-01** - The 2nd phase of [post-deploy pipeline testing](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352) was initiated with good results (details [here](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352#note_1009830702)). The next steps are to complete the 2nd phase testing scenarios, prepare the [coordinated pipeline for a rollout](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2461), and [update the rollback checks](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2365) to consider the post-deploy pipeline.
* **2022-07-08** - The 2nd phase of the [post-deploy migration (PDM) pipeline testing](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352) was completed along with the [update the rollback checks](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2365). An [announcement](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2478) has been created to broadcast the rollout of the PDM to the Engineering departments. Next steps are to observe the behavior of the PDM in the auto-deploy daily activities and update the release templates to consider the PDM.
* **2022-07-15** - The PDM has been incorporated into the auto-deploy daily activities with no major issue (details [here](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352#note_1028035257)), also the release templates were updated to consider this new pipeline. Next steps are complete the rollout and start the [final steps](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/585#phase-3-final-steps).
* **2022-07-22** - The [post-deploy pipeline rollout](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2352) has been completed. The issues of Phase 3 were also completed (including removing the legacy post-migrations jobs from the [coordinated](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2353) and from the [deployer](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2301) pipelines, and making the status of the post-migrations [accessible to developers](https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2052)). Next steps are to address any missing issues and analyze the follow-up ones to determine if they're required for this iteration, or if they can be classified as technical debt or issues required for the next iteration.
* **2022-07-29** - Issues associated with the three different phases have been completed. Next steps are to gather all follow-up issues and to open up an epic describing the next iteration.
</details>
### Status - 2022-08-04
Exit criteria has been addressed, making the work for this phase completed. The next iteration for the post-deploy migration pipeline would be to make it smarter (https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/778) by relying on the classification of post-deploy migrations. The remaining follow-up issues have been moved to this epic.
epic