Migrate Sidekiq jobs based on routing rules

What does this MR do and why?

As described in gitlab-com/gl-infra/scalability#1930, we are defaulting self-managed installations' Sidekiq (omnibus-gitlab!6289 (closed), gitlab-org/charts/gitlab!2789 (closed)) to listen to default and mailers queue only. To prevent jobs from being left out after user upgrades to the new version, we're leveraging post-deployment migration to automatically migrate jobs based on the queues that are outside of the current routing rules defined in gitlab.yml.

This is what a job looks like in redis:

> lrange resque:gitlab:queue:email_receiver 0 -1
1) "{\"retry\":3,\"queue\":\"email_receiver\",\"backtrace\":true,\"version\":0,\"args\":[],\"class\":\"EmailReceiverWorker\",\"jid\":\"a2cf84850e1506f5e5acb47d\",\"created_at\":1664962337.9325042,\"meta.feature_category\":\"team_planning\",\"correlation_id\":\"7bb10dedd488b4b8c8a6eadea2ad6df9\",\"worker_data_consistency\":\"always\",\"idempotency_key\":\"resque:gitlab:duplicate:email_receiver:e24f8243f3bdc71e54ba932255525eb20e3c5416e63890942eb16b1b38ed3f2f\",\"duplicate-of\":\"460f87d4dcdea1758213403f\",\"size_limiter\":\"validated\",\"enqueued_at\":1664962337.934328}"

This MR has to be included with below Related MRs in the same release at the very least.

Related MRs

These MRs are changing Sidekiq to listen to default and mailers queue by default, if no custom routing rules are defined. Otherwise, the queues are generated from the routing rules.

• omnibus-gitlab!6289 (closed)
• gitlab-org/charts/gitlab!2789 (closed)

Below MR has the same copy-pasted code as a Rake task:

• !101348 (merged)

How to set up and validate locally

1. Start GitLab.
2. Change gitlab.yml sidekiq.routing_rules to simulate queue per-worker configuration (default routing rules prior to !97908 (merged))

    routing_rules:
      - ["*", null]

3. Stop Sidekiq (so no jobs get processed).

4. Check queues before migration

redis /Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket[1]> keys resque:gitlab:queue:*
(empty array)

5. Schedule some jobs either from the console or the UI.

In rails console:
[2] pry(main)> EmailReceiverWorker.perform_async
=> "3b153d570496aa1f82a4af9f"
[3] pry(main)> ExportCsvWorker.perform_async
=> "e563fcdaa5bc13fd35b204a5"

6. Check the Sidekiq queues.

redis /Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket[1]> keys resque:gitlab:queue:*
1) "resque:gitlab:queue:email_receiver"
2) "resque:gitlab:queue:export_csv"
redis /Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket[1]> lrange resque:gitlab:queue:email_receiver 0 -1
1) "{\"retry\":3,\"queue\":\"email_receiver\",\"backtrace\":true,\"version\":0,\"args\":[],\"class\":\"EmailReceiverWorker\",\"jid\":\"3b153d570496aa1f82a4af9f\",\"created_at\":1667224064.14315,\"meta.feature_category\":\"team_planning\",\"correlation_id\":\"aeeaf1d8714cfbfe6b58ef74f3943e42\",\"worker_data_consistency\":\"always\",\"idempotency_key\":\"resque:gitlab:duplicate:email_receiver:e24f8243f3bdc71e54ba932255525eb20e3c5416e63890942eb16b1b38ed3f2f\",\"duplicate-of\":\"2faba86933661b0c02bf33c0\",\"size_limiter\":\"validated\",\"enqueued_at\":1667224064.151187}"
redis /Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket[1]> lrange resque:gitlab:queue:export_csv 0 -1
1) "{\"retry\":3,\"queue\":\"export_csv\",\"backtrace\":true,\"version\":0,\"args\":[],\"class\":\"ExportCsvWorker\",\"jid\":\"e563fcdaa5bc13fd35b204a5\",\"created_at\":1667224069.729505,\"meta.feature_category\":\"team_planning\",\"correlation_id\":\"97075e24957f895557707a225d7559ab\",\"worker_data_consistency\":\"always\",\"idempotency_key\":\"resque:gitlab:duplicate:export_csv:5a7377c023323988e6c86830ca9cc09d9276e04b7efb5dab402c84f319474907\",\"duplicate-of\":\"e8cf11cc821bfae428bd812e\",\"size_limiter\":\"validated\",\"enqueued_at\":1667224069.732599}"

7. Change the routing rules configuration. Set gitlab.yml sidekiq.routing_rules to null (Routing rules will be defaulted to [['*', 'default']] here).
8. Run the migration.

9. Check the Sidekiq queues again: they should match the new routing rules.

redis /Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket[1]> keys resque:gitlab:queue:*
1) "resque:gitlab:queue:default"
redis /Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket[1]> lrange resque:gitlab:queue:default 0 -1
1) "{\"retry\":3,\"queue\":\"default\",\"backtrace\":true,\"version\":0,\"args\":[],\"class\":\"ExportCsvWorker\",\"jid\":\"e563fcdaa5bc13fd35b204a5\",\"created_at\":1667224069.729505,\"meta.feature_category\":\"team_planning\",\"correlation_id\":\"97075e24957f895557707a225d7559ab\",\"worker_data_consistency\":\"always\",\"idempotency_key\":\"resque:gitlab:duplicate:export_csv:5a7377c023323988e6c86830ca9cc09d9276e04b7efb5dab402c84f319474907\",\"duplicate-of\":\"e8cf11cc821bfae428bd812e\",\"size_limiter\":\"validated\",\"enqueued_at\":1667224069.732599}"
2) "{\"retry\":3,\"queue\":\"default\",\"backtrace\":true,\"version\":0,\"args\":[],\"class\":\"EmailReceiverWorker\",\"jid\":\"3b153d570496aa1f82a4af9f\",\"created_at\":1667224064.14315,\"meta.feature_category\":\"team_planning\",\"correlation_id\":\"aeeaf1d8714cfbfe6b58ef74f3943e42\",\"worker_data_consistency\":\"always\",\"idempotency_key\":\"resque:gitlab:duplicate:email_receiver:e24f8243f3bdc71e54ba932255525eb20e3c5416e63890942eb16b1b38ed3f2f\",\"duplicate-of\":\"2faba86933661b0c02bf33c0\",\"size_limiter\":\"validated\",\"enqueued_at\":1667224064.151187}"

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

assigned to @marcogreg

changed milestone to %15.5

Suggested Reviewers (beta)

The individuals below may be good candidates to participate in the review based on various factors.

You can use slash commands in comments to quickly assign /assign_reviewer @user1.

Suggested Reviewers
@dzaporozhets, @mayra-cabrera, @smcgivern, @abrandl, @mkozono

If you do not believe these suggestions are useful, please apply the label Bad Suggested Reviewer. You can also provide feedback for this feature on this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/357923.

Automatically generated by Suggested Reviewers Bot - an experimental ML-based recommendation engine created by ~"group::applied ml".

added database databasereview pending labels

	2 Warnings
⚠	This merge request includes more than 10 commits. Each commit should meet the following criteria: Have a well-written commit message. Has all tests passing when used on its own (e.g. when using git checkout SHA). Can be reverted on its own without also requiring the revert of commit that came before it. Is small enough that it can be reviewed in isolation in under 30 minutes or so. If this merge request contains commits that do not meet this criteria and/or contains intermediate work, please rebase these commits into a smaller number of commits or split this merge request into multiple smaller merge requests.
⚠	New migrations added but db/structure.sql wasn't updated Usually, when adding new migrations, db/structure.sql should be updated too (unless the migration isn't changing the DB schema and isn't the most recent one).

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	Sam Figueroa (@sam.figueroa) (UTC+0)	George Koltsov (@georgekoltsov) (UTC+0)
database	Alexandru Croitor (@acroitor) (UTC+2)	Adam Hegyi (@ahegyi) (UTC+1)
~"migration"	No reviewer available	No maintainer available

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the 🔁 danger-review job that generated this comment.

Generated by 🚫 Danger

added maintenancescalability typemaintenance labels

mentioned in epic gitlab-com/gl-infra&596 (closed)

changed the description

added 1 commit

• 86c87b06 - Migrate Sidekiq jobs to default queue

Compare with previous version

mentioned in epic gitlab-com/gl-infra&148

Setting label groupscalability based on @marcogreg's group.

added groupscalability label

Setting label(s) devopsplatforms sectionplatforms based on groupscalability.

added devopsplatforms sectionplatforms labels

requested review from @marcogreg

removed review request for @marcogreg

added 1 commit

• 50c2df66 - Migrate Sidekiq jobs outside of current routing rules

Compare with previous version

changed title from Draft: Migrate Sidekiq jobs to default queue to Draft: Migrate Sidekiq jobs based on routing rules

changed the description

mentioned in merge request gitlab-org/charts/gitlab!2789 (closed)

mentioned in merge request omnibus-gitlab!6289 (closed)

marked this merge request as ready

changed the description

requested review from @smcgivern

Hi @smcgivern, as we discussed previously in gitlab-com/gl-infra/scalability#1491 (comment 1103618756), please help to review 🙏.

I believe there might be a cleaner way to write the migration/the spec, would love to hear your input.

marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

changed the description

removed review request for @smcgivern

mentioned in merge request !101348 (merged)

added 1 commit

• 1bc288f9 - Migrate Sidekiq jobs outside of current routing rules

Compare with previous version

changed the description

added 1 commit

• 9ab2609b - Migrate Sidekiq jobs outside of current routing rules

Compare with previous version

changed milestone to %15.6

added missed:15.5 label

@smcgivern Following up on our discussion about the copy-pasted code vs calling the Rake method !101348 (comment 1140220137), any further thoughts? Otherwise, I'll go ahead and call the Rake method instead.

resolved all threads

requested review from @mayra-cabrera

Database migrations (on the main database)

	1 Warnings
⚠	20220928093644 - MigrateSidekiqJobs did not complete successfully, check the job log for details

Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

Migration	Type	Total runtime	Result	DB size change
20220928093644 - MigrateSidekiqJobs	Post deploy	2.8 s	💥	+0.00 B

💥 Migration: 20220928093644 - MigrateSidekiqJobs

• Type: Post deploy
• Duration: 2.8 s
• Database size change: +0.00 B

---

Background migrations

---

Other migrations pending on GitLab.com

Migration	Type	Total runtime	Result	DB size change

Clone Details

Clone ID	Clone Created At	Clone Data Timestamp	Expected Removal Time
database-testing-1501818-8334464-main	2022-10-27T06:34:36Z	2022-10-27T04:10:17Z	2022-10-27 18:37:52 +0000
database-testing-1501818-8334464-ci	2022-10-27T06:34:36Z	2022-10-27T04:46:02Z	2022-10-27 18:37:52 +0000

Artifacts

---

Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

Database migrations (on the ci database)

	1 Warnings
⚠	20220928093644 - MigrateSidekiqJobs did not complete successfully, check the job log for details

Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

Migration	Type	Total runtime	Result	DB size change
20220928093644 - MigrateSidekiqJobs	Post deploy	2.4 s	💥	+0.00 B

💥 Migration: 20220928093644 - MigrateSidekiqJobs

• Type: Post deploy
• Duration: 2.4 s
• Database size change: +0.00 B

---

Background migrations

---

Other migrations pending on GitLab.com

Migration	Type	Total runtime	Result	DB size change

Clone Details

Clone ID	Clone Created At	Clone Data Timestamp	Expected Removal Time
database-testing-1501818-8334464-main	2022-10-27T06:34:36Z	2022-10-27T04:10:17Z	2022-10-27 18:37:52 +0000
database-testing-1501818-8334464-ci	2022-10-27T06:34:36Z	2022-10-27T04:46:02Z	2022-10-27 18:37:52 +0000

Artifacts

---

Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

added database-testing-automation label

To prevent jobs from being left out after user upgrades to the new version, we're leveraging post-deployment migration to automatically migrate jobs based on the queues that are outside of the current routing rules defined in gitlab.yml.

This makes sense to me. Post-migrations are optional for self-managed (they're required when updating to a new release though). Should we add some version-specific upgrading instructions for this one? If yes, it can be in another MR.

mentioned in merge request !102268 (merged)

added 2934 commits

• 9ab2609b...a517c9c2 - 2930 commits from branch master
• 1f84a739 - Migrate Sidekiq jobs outside of current routing rules
• e97ea75a - Copy implementation from sidekiq_migrate_jobs.rb
• 45111a17 - Call common migrate method from rake task
• f2fcc5f1 - Update LOG_FREQUENCY to 10

Compare with previous version

added backend label

added 1 commit

• 3b3fb5f0 - Add spec for migration without mappings mocked

Compare with previous version

Allure report

allure-report-publisher generated test report!

e2e-review-qa: ❗ test report for b47c07b0

expand test summary

+-----------------------------------------------------------------------------------------+
|                                     suites summary                                      |
+------------------------------------+--------+--------+---------+-------+-------+--------+
|                                    | passed | failed | skipped | flaky | total | result |
+------------------------------------+--------+--------+---------+-------+-------+--------+
| Create                             | 28     | 0      | 1       | 0     | 29    | ✅     |
| Plan                               | 49     | 0      | 1       | 0     | 50    | ✅     |
| Manage                             | 43     | 0      | 3       | 7     | 46    | ❗     |
| Verify                             | 12     | 0      | 1       | 0     | 13    | ✅     |
| Version sanity check               | 0      | 0      | 1       | 0     | 1     | ➖     |
| Govern                             | 10     | 0      | 5       | 0     | 15    | ✅     |
| Configure                          | 0      | 0      | 1       | 0     | 1     | ➖     |
| Feature flag handler sanity checks | 9      | 0      | 0       | 0     | 9     | ✅     |
| Package                            | 0      | 0      | 1       | 0     | 1     | ➖     |
+------------------------------------+--------+--------+---------+-------+-------+--------+
| Total                              | 151    | 0      | 14      | 7     | 165   | ❗     |
+------------------------------------+--------+--------+---------+-------+-------+--------+

added 1 commit

• 0467438a - Copy-paste back migration code

Compare with previous version

added 1 commit

• 81bda100 - Skip migration if running on GitLab.com

Compare with previous version

added 1 commit

• 223fce2b - Call method from migration

Compare with previous version

changed the description

@marcogreg Let me know when this is ready for another review, I'm unassigning myself in the mean time.

removed review request for @mayra-cabrera

requested review from @mayra-cabrera

requested review from @smcgivern

added 664 commits

• 223fce2b...f345c4bb - 655 commits from branch master
• 587bd60e - Migrate Sidekiq jobs outside of current routing rules
• d701a7ca - Copy implementation from sidekiq_migrate_jobs.rb
• edc75eb8 - Call common migrate method from rake task
• db071157 - Update LOG_FREQUENCY to 10
• c0224199 - Add spec for migration without mappings mocked
• 11d28ef7 - Copy-paste back migration code
• 56368878 - Skip migration if running on GitLab.com
• 53ba4aed - Call method from migration
• 49a549e1 - Fix Redis 5 compatibility

Compare with previous version

Not sure why the test is failing, doesn't seem to be related.

Database migrations (on the main database)

Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

Migration	Type	Total runtime	Result	DB size change
20220928093644 - MigrateSidekiqJobs	Post deploy	1.6 s	✅	+0.00 B

Migration: 20220928093644 - MigrateSidekiqJobs

• Type: Post deploy
• Duration: 1.6 s
• Database size change: +0.00 B

---

Background migrations

---

Other migrations pending on GitLab.com

Migration	Type	Total runtime	Result	DB size change

Clone Details

Clone ID	Clone Created At	Clone Data Timestamp	Expected Removal Time
database-testing-1516005-8389075-main	2022-11-02T20:00:14Z	2022-11-02T15:02:22Z	2022-11-03 08:03:49 +0000
database-testing-1516005-8389075-ci	2022-11-02T20:00:14Z	2022-11-02T16:48:23Z	2022-11-03 08:03:49 +0000

Artifacts

---

Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

Database migrations (on the ci database)

Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

Migration	Type	Total runtime	Result	DB size change
20220928093644 - MigrateSidekiqJobs	Post deploy	1.6 s	✅	+0.00 B

Migration: 20220928093644 - MigrateSidekiqJobs

• Type: Post deploy
• Duration: 1.6 s
• Database size change: +0.00 B

---

Background migrations

---

Other migrations pending on GitLab.com

Migration	Type	Total runtime	Result	DB size change

Clone Details

Clone ID	Clone Created At	Clone Data Timestamp	Expected Removal Time
database-testing-1516005-8389075-main	2022-11-02T20:00:14Z	2022-11-02T15:02:22Z	2022-11-03 08:03:49 +0000
database-testing-1516005-8389075-ci	2022-11-02T20:00:14Z	2022-11-02T16:48:23Z	2022-11-03 08:03:49 +0000

Artifacts

---

Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

approved this merge request

👋 @mayra-cabrera, thanks for approving this merge request.

This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.

For more info, please refer to the following links:

• rspec minimal jobs
• jest minimal jobs
• merging a merge request.

added 1 deleted label

added databaseapproved label and removed databasereview pending label

removed review request for @mayra-cabrera

added 1 commit

• 7c3cbcf2 - Bump migration version

Compare with previous version

added 1 commit

• b47c07b0 - Clear hardcoded queue in spec

Compare with previous version

approved this merge request

resolved all threads

enabled an automatic merge when the pipeline for 07406db4 succeeds

merged

mentioned in commit 49bf3819

added workflowstaging-canary label

added workflowcanary label and removed workflowstaging-canary label

added workflowstaging label and removed workflowcanary label

added workflowproduction label and removed workflowstaging label

added workflowpost-deploy-db-staging label and removed workflowproduction label

added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label

mentioned in merge request !103001 (closed)

mentioned in issue gitlab-com/gl-infra/scalability#1991

added releasedcandidate label

mentioned in merge request kubitus-project/kubitus-installer!1646 (merged)

added releasedpublished label and removed releasedcandidate label

mentioned in merge request !142676 (closed)

mentioned in merge request !142577 (merged)