Skip to content

Test runner for batched background migrations

Simon Tomlinson requested to merge test-batched-background-migrations into master

What does this MR do and why?

Describe in detail what your merge request does and why.

This allows for testing of batched background migrations as part of the database testing pipeline. It samples batches that will be executed on gitlab.com and tests a random subset to look for queries that fail or exceed timing bounds. Resolves gitlab-org/database-team/gitlab-com-database-testing#51 (closed).

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

I've used this code to test a real batched background migration in !85863 (comment 923916411).

In local testing, I produce the following directory structure when testing a CopyColumnUsingBackgroundMigrationJob background migration:

Batched Background Migration Testing Directory Structure 
.
└── CopyColumnUsingBackgroundMigrationJob
    ├── batch_1
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── batch_10
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── batch_2
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── batch_3
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── batch_4
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── batch_5
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── batch_6
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── batch_7
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── batch_8
    │   ├── migration-stats.json
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    └── batch_9
        ├── migration-stats.json
        ├── migration.log
        ├── query-details.json
        └── transaction-duration.json

11 directories, 40 files

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

If you have a local pending batched background migration queued within the last 2 days then you can just run rake gitlab:db:sample_batched_background_migrations and see the results in tmp/migration-testing/background_migrations.

If not, the specs include an integration test (runs sampled jobs from the batched background migration) that creates a new table, samples a migration against it, and writes the files to a temporary folder. This skips a lot of complex setup, so I recommend setting a breakpoint before the directory is removed and poking around.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Simon Tomlinson

Merge request reports