Handle Sidekiq namespace deprecation for self-managed
For self-managed users, the new deployment should write into the non-namespaced keys immediately. The migration can be done with a post-deployment migration of the following
- Queues: jobs from namespaced queues need to be RPOPLPUSH-ed to the non-namespaced queues
- Schedule sorted sets: jobs in the sorted set to be written into non-namespaced sorted set
- Retry sorted sets: same as (2)
- Deadset sorted sets: same as (2)
Take note that the relevant version of gitlab-exporter should be released before an envvar clean-up (and post-deployment migration) is released. Based on production#16264 (comment 1543098444), a post-deployment migration is feasible.
We would also need to elaborate about the expected behaviour in changelog/release notes (https://docs.gitlab.com/ee/update/versions/gitlab_16_changes.html).
Release plan
The tentative first release is 16.4, but since it is closing soon, we may have to start with 16.5 if we miss the milestone.
16.4
-
Gitlab: Introduce dual polling while using namespace (gitlab-org/gitlab!131478 (merged))
* Charts: Enable dual poll for gitlab-exporter (#2285 (closed)) -- gitlab-org/charts/gitlab!3388 (merged) - Omnibus: Enable dual poll for gitlab-exporter -- gitlab-org/omnibus-gitlab!7141 (merged)
Deployment behaviour: no change as all Sidekiq jobs are on namespaced queues
Rollback behaviour: no change as all Sidekiq jobs are on namespaced queues
note: charts MR missed 16.4 so we can only release the deprecation in 16.6.
16.5
note: charts MR missed 16.4 cutoff
- Charts: Enable dual poll for gitlab-exporter (#2285 (closed)) -- gitlab-org/charts/gitlab!3388 (merged)
16.6
-
Gitlab - Deprecate the use of namespace in Sidekiq
- Deprecate the use of namespace in Mailroom
- Add post-deployment migration for queues and sorted sets
-
Charts: Deprecate namespace usage in Mailroom -
Omnibus: Mailroom config is dependent on gitlab (NO CHANGES REQUIRED)
Note that both 16.4 and 16.5 will dual-poll to ensure smooth rollback. PDM will move bulk of the jobs across namespaces while dual-polling handles the rest.
Deployment behaviour: Sidekiq workers dual-poll to handle both groups of jobs. Non-namespaced jobs will gradually become the only type of jobs in the system.
Rollback behaviour: Sidekiq workers dual-poll to handle both groups of jobs. Namespaced jobs will gradually become the only type of jobs in the system.
16.7
-
Gitlab: Remove dual polling functionality (gitlab-org/gitlab!136397 (merged))
Deployment behaviour: Nothing applicable since all messages are on non-namespaced queues and sorted sets
Rollback behaviour: Nothing applicable since all messages are on non-namespaced queues and sorted sets
16.8
-
gitlab-exporter: remove dual-namespace polling functionality in gitlab-exporter (gitlab-org/ruby/gems/gitlab-exporter!198 (merged)) -
Charts: Set gitlab-explorer to poll non-namespaced only (https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/new?merge_request%5Bsource_branch%5D=sc1-stop-namespaced-probe) -
Omnibus: Set gitlab-explorer to poll non-namespaced only (gitlab-org/omnibus-gitlab!7237 (merged))
Note on performance: dual-polling would double BRPOP size. This would create a problem if the SM user's setup has a large number of queues and a large number of clients, leading to a cardinality spike. This was more applicable to .com as we had far more clients that self-managed installations.