Run post-deployment migrations in Kubernetes (Staging)

Current architecture:

  1. PDMs are run by running a ChatOps command in Slack, which creates a pipeline in release/tools (sample)
  2. release/tools will trigger child pipelines in the Deployer project (sample for gstg)
    1. release/tools trigger YAML
  3. Pipelines in the Deployer project run the CI jobs inside the YAML definition files ci/environments/gstg-postdeploy-migrations.yml and ci/environments/gstg-postdeploy-migrations.yml

The goal for this issue would be to add jobs to that pipeline, which would call k-ctl upgrade without specifying the AUTO_DEPLOY and GITLAB_IMAGE_TAG environment variables, and SKIP_POST_DEPLOYMENT_MIGRATIONS set to false. That should create the right set of resources in the Kubernetes cluster.

The current specifications in the CI of gitlab-com only have auto-deploy CI jobs defined: https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/4494404

Steps

This change will be done in three stages because running PDMs can not be reversed.

  1. Dry-run PDM job to check the Kubernetes manifest of the PDM job
    1. MR: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/627 and gitlab-com/gl-infra/k8s-workloads/gitlab-com!4515 (merged)
    2. Verification thread: #21039 (comment 2578113249)
  2. Run PDM job for real but after the deploy VM (a no-op job)
    1. MR: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/628 and https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/629
    2. Verification thread: #21039 (comment 2589162560)
  3. Invert the order of the jobs and run PDMs in Kubernetes
    1. MR: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/631
    2. Verification thread: #21039 (comment 2606728969)
  4. Wait about a week and see whether any problems are discovered

Development - Related MRs on ops.gitlab.net

MRs on ops.gitlab.net are not listed here automatically by GitLab.

Edited by Siddharth Kannan