Skip to content

Draft: Remove code from problematic migration and restore structure.sql

drew stachon requested to merge dc-remove-problematic-index-drop into master

What does this MR do and why?

This MR is part of a larger plan to mitigate a migration which is taking too long and blocking deployments.

This is the plan.

PLEASE REVIEW THIS PLAN (in addition to the source code change included by this MR)

Via @drew from slack:

What has already happened:

  • We merged !84308 (merged) recently, during 14.10
  • We dropped the index on staging
  • We began the migration on production
  • A production incident has been opened because the migration has been running for over an hour

What we would like to do:

  1. [For @drew] Create the MR to revert the migration out of the release.
  2. [For EOC (@nnelson)] Manually kill the migration on production to end the transaction.
    1. Create a production change request for this.
    2. Kill the deployment job.
    3. Check the status of the 82178 statement: DROP INDEX CONCURRENTLY "ci_job_artifacts_expire_at_unlocked_idx"
    4. If the process is still running, then forcibly cancel it.
    5. If the process is still running, then forcible terminate it.
  3. Find a reviewer+merger (backend maintainer) to merge the new MR from step 1..
  4. Merge the new MR from step 1. that removes the actual migration code from #up and #down in the existing, previously merged migration effectively making it a no-op
  5. [For @rpereira2] Validate that the auto-deploy package contains the commit from the revert MR.
  6. [Anyone] Manually run migrate:down on staging only, to rebuild the index and bring the staging database back into parity with the production database
    1. Shell into the console-01-sv-gstg.c.gitlab-staging-1.internal system.
    2. Invoke: rake db:migrate:down VERSION=20220404184814
  7. Deploy again to staging and production, which will both run an empty migration, successfully, and leave the indexes in place in both environments

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by drew stachon

Merge request reports