Make Gitlab::BackgroundMigration.remaining work across queues

With worker routing, a Sidekiq queue can have jobs from multiple workers. This meant Gitlab::BackgroundMigration.remaining needed to account for that possibility. We do that in the same way as we already did with the scheduled, retry, and dead sets: iterate through and look for matching records.

To test, you can add this under sidekiq in config/gitlab.yml:

routing_rules: [['*', 'default']]

This makes every worker go to the default queue. Then, in a console (without Sidekiq running!):

[1] pry(main)> BackgroundMigrationWorker.queue
=> "default"
[2] pry(main)> BackgroundMigrationWorker.perform_async('Foo')
=> "237f3d21f4afe9606761b826"
[3] pry(main)> BackgroundMigrationWorker.perform_in(10.minutes, 'Foo')
=> "132fb9bc94011f6ccaeb668c"
[4] pry(main)> BackgroundMigrationWorker.perform_in(10.minutes, 'Foo')
=> "068c76b285a9cf9ebad6cdee"
[5] pry(main)> Gitlab::BackgroundMigration.remaining
=> 3

changed milestone to %14.2

added Background Processing groupscalability tooling (archive) labels

assigned to @smcgivern

mentioned in issue gitlab-com/gl-infra/scalability#1087

added databasereview pending label

added backend database documentation labels

	1 Message
	This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

Documentation review

The following files require a review from a technical writer:

• doc/update/index.md

The review does not need to block merging this merge request. See the:

• Metadata for the *.md files that you've changed. The first few lines of each *.md file identify the stage and group most closely associated with your docs change.
• The Technical Writer assigned for that stage and group.
• Documentation workflows for information on when to assign a merge request for review.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	George Koltsov (@georgekoltsov) (UTC+1, same timezone as @smcgivern)	Heinrich Lee Yu (@engwan) (UTC+8, 7 hours ahead of @smcgivern)
database	Simon Tomlinson (@stomlinson) (UTC-5, 6 hours behind @smcgivern)	Mayra Cabrera (@mayra-cabrera) (UTC-5, 6 hours behind @smcgivern)

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added 1 commit

• fcc0748d - Make Gitlab::BackgroundMigration.remaining work across queues

Compare with previous version

@qmnguyen0711 could you review, please? The failures look like they could be related, but they aren't: #337542 (closed)

requested review from @qmnguyen0711

added 1 commit

• d20d8748 - Make Gitlab::BackgroundMigration.remaining work across queues

Compare with previous version

added 1 commit

• b086b148 - Make Gitlab::BackgroundMigration.remaining work across queues

Compare with previous version

requested review from @mwoolf and removed review request for @qmnguyen0711

approved this merge request

removed review request for @mwoolf

@mwoolf, thanks for approving this merge request.

Please consider starting a new pipeline if:

• This is the first time the merge request is approved, or
• The merge request is ready to be merged, and there has not been a merge request pipeline in the last 2 hours.

For more info, refer to the guideline.

requested review from @mayra-cabrera

approved this merge request

Are these still accurate? If so, it'd be a good idea to leave them as guidance for people that do not have admin access.

@mayra-cabrera these instructions are accurate to a point if you're using an old GitLab version. But this version is so old that we don't have to keep it in the docs any more?

They will also become inaccurate if you use routing rules to change where background migrations run, unlike Gitlab::BackgroundMigration.remaining, which (thanks to this MR), will just work.

added databaseapproved label and removed databasereview pending label

resolved all threads

enabled an automatic merge when the pipeline for 01556dde succeeds

merged

mentioned in commit cb46d5b0

added workflowstaging label

added workflowcanary label and removed workflowstaging label

added workflowproduction label and removed workflowcanary label

added releasedcandidate label

added releasedpublished label and removed releasedcandidate label

added typemaintenance label and removed tooling (archive) label