Thanks to some outstanding investigative work by @mkaeppler in https://gitlab.com/gitlab-org/gitlab/issues/34396, and great prep work by Memory team, we are moving sidekiq-cluster to Core. ## Project management [DRI](https://about.gitlab.com/handbook/people-group/directly-responsible-individuals/) for this epic is: **@oswaldo** All new issues should at minimum contain the following labels: ``` /label ~"team::Scalability" /label ~"Service::Sidekiq" /label ~"sidekiq-cluster" /label ~"workflow::Triage" /epic https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181 /label ~P4 ``` |Resource| Link| |----- | ----- | |Work issue board | https://gitlab.com/gitlab-com/gl-infra/scalability/-/boards/1554230?&label_name[]=Service%3A%3ASidekiq&label_name[]=sidekiq-cluster | | Investigation issue | https://gitlab.com/gitlab-org/gitlab/issues/34396 | | Product approval | https://gitlab.com/gitlab-com/Product/issues/574 | --- ## Problem Statement From https://gitlab.com/gitlab-org/gitlab/issues/34396: Currently, running sidekiq in clustered mode (i.e. spawning more than one worker process) is technically only available as part of GitLab EE distributions, and for self-managed environments only in the Starter+ tiers. Because of that, when booting sidekiq up in a development env with the GDK, the least common denominator is assumed, which is to run sidekiq in a single-process setup. That can be a problem, because it means there is a divergence between the environment developers work on, and what will actually run in production (i.e. gitlab.com and higher-tier self-managed envs). We have already seen problems in production that went unnoticed and are specific to a multi-process sidekiq setup, such as [race conditions between worker processes to initialize prometheus](https://gitlab.com/gitlab-org/gitlab/issues/33125), as well as [initialization races leading to crashes](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8194). ## Reach Personas: * Sasha (engineer) - because they get more confidence in making changes to sidekiq and testing them in an environment closer to production * Devon (devops) - because it simplifies how we operate sidekiq in any environment Reach 10.0 = Impacts the vast majority (\~80% or greater) of our users, prospects, or customers. ## Goal - Having `sidekiq-cluster` being used by default for GDK and self-hosted instances, initially with a single process to handle all queues. - Deprecating `sidekiq_cluster` and standardizing `sidekiq` configurations using `sidekiq-cluster` under the hood (i.e. Omnibus). - Being able to disable the new default (`sidekiq-cluster`), falling back to `sidekiq` in a initial rollout. - Anticipate communication with Memory, Support, Distribution and Quality teams (self-hosted installations) and Developers regarding GDK changes. ## Release & Stages ### 1. Move script to Core - [x] In order to reference the `sidekiq-cluster` script through all environments, first we need to move it from EE to the Core. This work is mapped at https://gitlab.com/gitlab-com/gl-infra/scalability/issues/196. ### 2. GDK First step is releasing the change to GDK, considering `sidekiq-cluster` is already used by big EE instances (e.g. GitLab.com) and ideally it should also work well at developers environment (preventing bugs from flowing to other environments). 1. [x] With https://gitlab.com/gitlab-com/gl-infra/scalability/issues/170 we look forward to use `bin/background_jobs` to manage `sidekiq-cluster` instead `sidekiq` processes based in an ENV variable. That way we're able to release it for developers before Source installations (that also use this script) and give enough control to quickly fallback to the previous behavior. 2. [x] [Use sidekiq-cluster through environment variable at Procfile (GDK)](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/197) 3. [x] Gather feedback from release and iterate if needed ### 3. Source installation 1. [x] Make use of the changes at **1** https://gitlab.com/gitlab-com/gl-infra/scalability/issues/173 ### 4. Omnibus 1. [x] [Support sidekiq-cluster as opt-in on Omnibus sidekiq config](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/171) takes care of changing `sidekiq` Omnibus configuration to start a Sidekiq cluster instead, in a first step it'll be a opt-in option (documented as experimental) - we can move on to a opt-out approach once we're confident and have tests in place (`2`). 2. [x] [Setup tests for sidekiq-cluster](https://gitlab.com/gitlab-org/quality/performance/-/issues/204) ### 5. Helm charts - [x] [Use sidekiq-cluster for Helm charts](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/198). Considering how Helm charts use `sidekiq` today (one process per pod), using `sidekiq-cluster` needs further consideration after completing the first steps of this Epic (see https://gitlab.com/gitlab-com/gl-infra/scalability/issues/198) - Still ideally we should lean towards using it for making `sidekiq-cluster` the SSoT. ## Communication * [x] Initial step was handled at https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/227 (12.10). * [x] Communicate with Support that in 13.0 we'll have Sidekiq Cluster being used by default on Helm Charts and Omnibus ``` Hey team, just an update on https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181. On 13.0 we're enabling the usage of Sidekiq Cluster by default instead using Sidekiq directly for Omnibus and Helm charts instances (we've also just merged this small notice for 13.0 blogpost https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/49391). Technically it means that instead running one sidekiq process handling all queues, we'll by default be running a sidekiq-cluster instance (with a single child sidekiq process) handling all queues, making it easier to spin more sidekiq processes handling a smaller set of queues and so on (more documentation can be found at https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html). We don't expect it'll impact customers, but still, we've created an issue to track any unexpected behavior (https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/340) and customers are still able to disable this new cluster mode (https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html#disable-sidekiq-cluster). ``` https://gitlab.slack.com/archives/C4Y5DRKLK/p1590006609100000 ## Exit criterias * [x] Move script to Core, making it available to everyone * [x] Use `sidekiq-cluster` by default for Core and EE GDK's * [x] Support `sidekiq-cluster` in Omnibus installations (opt-in) * [x] Support `sidekiq-cluster` in Helm charts installations (opt-in) * [x] Use `sidekiq-cluster` by default for Source installations * [x] Use `sidekiq-cluster` by default for Helm charts installations (https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1989) * [x] Use `sidekiq-cluster` by default for Omnibus installations (https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/239) * [x] Make sure there are cleanup issues for GDK, Omnibus, Source and Helm chart `sidekiq-cluster` usage (we'll want to support fallbacks for a while after this is fully released, so we can't remove everything just yet). ## Status 2020-05-21 `sidekiq-cluster` is now the default for GDK and self-hosted instances which brings the environment that developers work on in line with what is run in production.