## Definitions A Sidekiq worker is a class that performs a unit of work. A Sidekiq job is a task to be performed by a worker. A Sidekiq queue is a collection of jobs, which don't have to be from the same worker. A Sidekiq process listens to a set of queues and performs the jobs added to those queues by instantiating and running the worker classes. A Sidekiq process does this by using multiple threads so that it can perform work in parallel. ## History <sub>(Disclaimer, this history has been reconstructed from past discussions, may not be entirely accurate)<sub> Originally, the GitLab application used only a handful of queues for all Sidekiq background work. This meant that all workers would listen to (roughly) the same set small of queues to obtain work. The advantage of this approach is that it is easy to load-balance work - work is distributed between all workers without requiring specific balancing mechanisms. The disadvantage of this approach is that, when an incident occurred on GitLab.com (usually some sort of denial-of-service, or rate-limit incident), operators had limited capabilities to isolate the "bad traffic" from good. GitLab moved [first to multiple queues, based on features](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/815), and [then to a queue-per-worker approach](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3692/). This approach had several advantages: 1. It was easy to reason about. The destination of each job was well defined. 2. It was easy to manage. If a certain class became overloaded with junk messages, it was easy to address. > **I don't recommend having more than a handful of queues. Lots of queues makes for a more complex system and Sidekiq Pro cannot reliably handle multiple queues without polling**. M Sidekiq Pro processes polling N queues means O(M*N) operations per second slamming Redis. > > -- [https://github.com/mperham/sidekiq/wiki/Advanced-Options](https://github.com/mperham/sidekiq/wiki/Advanced-Options) However, there were also disadvantages: 1. Sidekiq is not designed for this approach. The Sidekiq author explicitly recommends against this approach. 1. As the number of background workers in the GitLab application grows, this approach is adding additional burden on our Sidekiq Redis servers. On GitLab.com, our `besteffort` workers monitor about 200 queues, and the Redis `BRPOP` commands used to monitor the queues consume a significant amount of time (by Redis latency standards). 1. Self-managed EE instances handle over 270 queues. The impact of this approach on these instances is unknown, but unlikely to be good. 1. Reliable fetching, in which jobs are simultaneously popped from a worker queue and stored in an intermediate queue in a single transaction using `BRPOPLPUSH`, is not possible when a worker is listening to multiple queues (at least without polling, which is highly preferable). It's likely that the number of workers in the application will continue to grow, putting additional pressure on Redis. For this reason, it has become apparent that we need to address the strategy of one-queue-per-worker-class, and reduce the number of queues in the application. ## Proposal We will describe the architecture in two parts: 1. Conceptually, without defaults 2. The default configuration, used in self-managed installs ### Conceptual Changes from existing Architecture The architecture is built on the **[Background Job Processing Improvements](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/96)** projects, and can be considered a next iteration forward beyond that work. * Each worker continues to publish a set of attributes (`urgency`, `feature_category`, `has_external_dependencies`, `resource_boundary` etc). * Work continues to be partitioned between worker nodes according to [the queue selector syntax](https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#queue-selector-experimental) built as part of the Background Job Processing Improvements project. Building on this foundation, we iterate with the following changes to the existing architecture: * Workers no longer specify a queue name. Instead of sending work to a fixed queue-per-worker-class, the worker will be configured, at startup time, with a queue that it will send its work to. The configuration will use the queue selector syntax. * The set of queues is fixed in number and name. For the sake of the proposal, we will assume a fixed size of 20 queues, and names: `alpha`, `beta`, `charlie`, `delta`, etc. In addition to these queues, there will also be `default` and `mailers` queues. * Sidekiq-cluster will lose it's `--experimental-queue-selector` flag. A fixed set of queues will be configured for a `sidekiq-cluster` server (and `--negate` will remain). #### Configuration Since routing of worker-classes to queues now happens in the application, rather than in sidekiq infrastructure, we move the queue selector configuration in the application, in `gitlab.yml` (plus appropriate Omnibus changes) ```yaml sidekiq: queue_routing: alpha: "urgency=high" beta: "urgency=throttled" ``` Any worker classes that do not match a queue will automatically be routed be routed to the `default` queue. This is not configurable. For example, in the above configuration, any worker `urgency=low` would not match and would be routed to the `default` queue. Additionally, all ActiveJob Mailer jobs would be hardwired to route to the `mailers` queue. Sidekiq-cluster would be configured to handle a small set of queues. Optimally, non-`catchall` queues (using `--negate`) would only listen to a single queue. ### Sensible Defaults The default configuration would be as follows: ```yaml sidekiq: queue_routing: alpha: "urgency=high" ``` This would lead to the following configuration: * `alpha` handles high-urgency jobs. * `mailers` handles emails. * `default` handles everything else. By default, `sidekiq-cluster` would be configured to listen to all 20 queues. This is a vast reduction on the ~270 that we listen to by default at present. ## Application Changes A new concern, `RoutingWorker`, translates the `sidekiq.queue_routing` configuration into queues names, given the attributes on a worker class. This is mapping is done on application initialisation, in the sidekiq client application (ie, GitLab Rails). Initially this concern will be rolled out on a subset of workers, most likely Cronjob workers, and gradually rolled out across the entire fleet. Eventually, `RoutingWorker` will be merged into `ApplicationWorker` and all worker classes will use it. The generator for `all_queues.yml` will need two specific changes: 1. It should exclude any worker classes that include `RoutingWorker`. 2. It should always include the fixed queues (`alpha`, `beta` etc) ## GitLab.com Infrastructure Changes For GitLab.com we will map our existing priorities to queues. I propose the following mapping, based strictly on the work done in https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/28. | new_queue | shard_name | selector | |-----------|-------------------------|---------------------------------------------------------------------| | alpha | urgent-cpu-bound | boundary="cpu", latency_sensitive="yes", feature_category!="search" | | beta | urgent-other | boundary="", latency_sensitive="yes", feature_category!="search" | | charlie | non-urgent-memory-bound | boundary="memory", feature_category!="search" | | delta | non-urgent-cpu-bound | boundary="cpu", latency_sensitive="no", feature_category!="search" | | echo | non-urgent-other | boundary="", latency_sensitive="no" | | foxtrot | elasticsearch | feature_category="search" | | default | catchall | | ## Migration The new fixed queues (`alpha`, `bravo` etc) will exist in `all_queues.yml`). At the very least, they will be picked up by the default queue and not be lost. We will also need to use some sort of migration script to ensure that jobs are not orphaned in abandoned queues during the migration process. There are several approaches that we could use to mitigate this (monitoring all "old" queue names for a while and automatically moving jobs), but I will keep that out of scope from this proposal. ## Reference architectures This also gives us a path to providing sensible defaults for reference architectures, which we can do once the migration is done. ## Status 2021-04-07 Our sidekiq-redis nodes are close to CPU saturation again (https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4049). In https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/961 we see that reducing the number of queues to watch can lead to a potential drop of ~40% in CPU usage on Redis Sidekiq. The first iteration will be to reduce the `catchall` shard to two queues.