Instrument traditional background migrations

What does this MR do and why?

Instrumentation of traditional background migrations for the gitlab-com-database-testing pipeline.

Adds a new rake task, gitlab:db:migration_testing:sample_background_migrations, which samples currently queued background migrations for a default of 30 minutes.

A different duration can be specified by passing a number of seconds: rake gitlab:db:migration_testing:sample_background_migrations[3600] to sample for 1 hour.

Data about the sampled background migrations is recorded in the tmp/migration-testing/background_migrations dir, where the testing pipeline notifier will be enhanced to parse and display it.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

tmp/migration-testing/
├── background_migrations
│   └── TestBackgroundMigration
│       ├── batch_1
│       │   ├── migration.log
│       │   ├── query-details.json
│       │   └── transaction-duration.json
│       ├── batch_2
│       │   ├── migration.log
│       │   ├── query-details.json
│       │   └── transaction-duration.json
│       ├── batch_3
│       │   ├── migration.log
│       │   ├── query-details.json
│       │   └── transaction-duration.json
│       ├── batch_4
│       │   ├── migration.log
│       │   ├── query-details.json
│       │   └── transaction-duration.json
│       └── batch_5
│           ├── migration.log
│           ├── query-details.json
│           └── transaction-duration.json
└── up
    ├── QueueTestBackgroundMigration
    │   ├── migration.log
    │   ├── query-details.json
    │   └── transaction-duration.json
    ├── metadata.json
    └── migration-stats.json

9 directories, 20 files

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

1. Switch to a new branch so that clean-up will be easier later

2. Apply the following patch, containing a test background migration with 5 jobs: 0001-Patch-for-testing-background-migrations.patch (You can apply with git am < $patch_file)

3. Run normal migration testing to push the jobs into redis: rake gitlab:db:migration_testing:up

4. Sample the jobs from redis: rake gitlab:db:migration_testing:sample_background_migrations

5. Observe the output structure in tmp/migration-testing

6. Delete the branch you created to get rid of the testing commit

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.