Make self-managed instances use routing rules by default

After we resolve the workers that depend on their queue sizes, it's safe to migrate self-managed instances from queue selector to routing rules. From a long discussion in #1281 (closed), it turns out the best solution is the simplest solution. We agree that we will simply use [["*", "default"]] routing rules for self-managed instances. This configuration is good enough for most use cases.

Changes needed:

• Set default routing rules in Rails app
• Make sidekiq-cluster listen to an additional default and mailers queue, if default routing rules is used and regardless of whether queue selector is used. This logic ensures all Sidekiq processes are still processing all jobs regardless of queue selector.
• Add entry in version upgrade instructions for 16.0 about the above changes.

added to epic &596

changed the description

added ServiceSidekiq groupscalability workflow-infraProposal labels

marked this issue as related to #1493 (closed)

marked this issue as related to #1492 (closed)

mentioned in issue #1493 (closed)

added teamFrameworks (deprecated) label

assigned to @marcogreg

mentioned in merge request gitlab-org/omnibus-gitlab!6289 (closed)

mentioned in epic &596

mentioned in epic &148

mentioned in commit gitlab-org/omnibus-gitlab@fb57b514

added #1885 (closed) as child task

removed child task #1885 (closed)

added #1886 as child task

@qmnguyen0711 For the Kubernetes Sidekiq chart, should we update the fallback wildcard to default mailers instead?

According to the docs, default,mailers will spawn 1 process to process both queues. However, Omnibus runs the sidekiq with space delimited queues which results in spawning 1 process for each queue (e.g. default mailers).

So, should we align it with the omnibus also?

According to the docs, default,mailers will spawn 1 process to process both queues. However, Omnibus runs the sidekiq with space delimited queues which results in spawning 1 process for each queue (e.g. default mailers).

I wonder whether we do the way around in omnibus. So that one process handles both default and mailers queues

It means we can change the other MR's queue groups to the following config. Doing that keeps the prior behavior of omnibus, which it spawns 1 process to handle all queues.

default['gitlab']['sidekiq']['queue_groups'] = ['default,mailers']

Spawning 2 processes would be okay, but I wonder it would increase the default consumed resource.

Valid point. I was also thinking about the same since previously we also had 1 process to watch all queues also. Will do the necessary on the other MR then, thanks!

Omnibus by nature has access to more of a system than a container in Kubernetes. Kubernetes can see that it needs more workers, and spawn on demand. Omnibus must start them all at once.

mentioned in commit gitlab-org/omnibus-gitlab@f99dc36d

mentioned in commit gitlab-org/omnibus-gitlab@d7e08e1b

mentioned in merge request gitlab-org/build/CNG!1109 (closed)

mentioned in commit marcogreg/gitlab@805535e1

mentioned in merge request gitlab-org/charts/gitlab!2749 (closed)

So, why not cover everything at once? Update the Rails initializer for Sidekiq vaules, including Settings['sidekiq']['routing_rules'] ||= []

@WarheadsSE That is a very good idea. Just to verify your point, updating the Rails initializer will automatically update the default values for Omnibus and Helm chart, right?

Will there be any drawbacks to updating the initializer only?

@qmnguyen0711 Do you have any take on this?

Will there be any drawbacks to updating the initializer only?

If there is no setting passed which will override it, then this is what is used. If there is / was some value set by the end user, even setting defaults in multiple locations will not override what the user has provided.

By placing this here, we can continue without having to provide these default contents in any consumer.

Got it @WarheadsSE, I'm working on that change in gitlab-org/gitlab!96647 (closed). I will update the necessary once it's done.

mentioned in commit gitlab-org/gitlab@978c2256

mentioned in merge request gitlab-org/gitlab!96647 (closed)

mentioned in commit gitlab-org/gitlab@d9d230c9

mentioned in commit gitlab-org/gitlab@a5105c00

mentioned in commit gitlab-org/gitlab@4e00198d

mentioned in commit gitlab-org/gitlab@ca71d19d

mentioned in commit gitlab-org/gitlab@e2af93e7

mentioned in commit gitlab-org/gitlab@fb1967d2

mentioned in commit gitlab-org/gitlab@ce34e9e5

mentioned in commit gitlab-org/gitlab@ba4b3be6

mentioned in commit gitlab-org/gitlab@805b9ef2

mentioned in commit gitlab-org/gitlab@438b885e

mentioned in commit gitlab-org/omnibus-gitlab@0a8f2a84

mentioned in commit gitlab-org/omnibus-gitlab@d2ae6ee9

mentioned in commit gitlab-org/omnibus-gitlab@c44c2e54

changed the description

mentioned in commit gitlab-org/gitlab@cfb4033b

mentioned in commit gitlab-org/gitlab@efb9c20e

mentioned in commit gitlab-org/gitlab@5cffdf08

mentioned in commit gitlab-org/gitlab@a41a0dd9

mentioned in commit gitlab-org/gitlab@270941d7

mentioned in commit gitlab-org/gitlab@27cc4268

mentioned in merge request gitlab-org/gitlab!97908 (merged)

changed the description

mentioned in commit gitlab-org/omnibus-gitlab@a8bb63e1

@smcgivern @qmnguyen0711

1. Since gitlab-org/gitlab!97908 (merged) is already merged and going to be included in %15.4, I was wondering for self-managed clients, say the omnibus MR gitlab-org/omnibus-gitlab!6289 (closed) is also included in %15.4 or even later version, then wouldn't they have to perform job migration before they're upgrading?
   • Say currently we're running 15.0, and we have 400+ queues, if 15.4 only includes the Rails changes we'd be fine. But with the omnibus change on Sidekiq server default, some jobs won't be picked up including retry and scheduled jobs.
2. Another breaking change potential is when user currently uses custom routing rules as brought up here gitlab-org/omnibus-gitlab!6289 (comment 1103015440)

@marcogreg good questions:

1. Yes, we need to do job migration. I think we can include a Rails DB migration to trigger this? It will be fine to do that in a later release, before merging the omnibus MR.
2. As you mentioned in the omnibus MR, we should be able to automatically fill in the queues based on the routing rules if the user hasn't set anything.

added workflow-infraIn Progress label and removed workflow-infraProposal label

mentioned in issue #1930 (closed)

mentioned in merge request gitlab-org/charts/gitlab!2789 (closed)

mentioned in merge request gitlab-org/gitlab!100392 (merged)

mentioned in merge request gitlab-org/gitlab!100823 (merged)

changed milestone to %16.0

changed the description

changed the description

mentioned in merge request gitlab-org/gitlab!111675 (merged)

changed the description

removed the relation with #1492 (closed)

added workflow-infraProposal label and removed workflow-infraIn Progress label

changed due date to May 01, 2023

added workflow-infraReady label and removed workflow-infraProposal label

mentioned in merge request gitlab-org/gitlab!115296 (merged)

marked the checklist item Add entry in version upgrade instructions for 16.0 about the above changes. as completed

marked the checklist item Make sidekiq-cluster listen to an additional default and mailers queue, if default routing rules is used and regardless of whether queue selector is used. This logic ensures all Sidekiq processes are still processing all jobs regardless of queue selector. as completed

marked the checklist item Set default routing rules in Rails app as completed

With Reintroduce default routing rules for Sidekiq (gitlab-org/gitlab!111675 - merged) merged, self-managed instances in 16.0 will have [['*', 'default']] routing rules set by default in Rails app, and each sidekiq processes will listen to default and mailers queues.

Closing this issue as all tasks are completed.

closed

added workflow-infraDone label and removed workflow-infraReady label

mentioned in issue gitlab-org/gitlab#435095

mentioned in issue #2720 (closed)