# Production Change ## Change Summary Move Sidekiq concurrency-limit deferred-job storage in gprd from SharedState to the dedicated `concurrency-limit-redis` MemoryStore instance. The change disables the `sidekiq_concurrency_limit_middleware` ops flag so no worker holds a concurrency limit, waits for the deferred-job lists to drain to empty, flips `use_primary_store_as_default_for_concurrency_limit`, then re-enables the middleware. The final state sends reads and writes only to MemoryStore, and `redis-cluster-shared-state` stops holding deferred job payloads. With every limit at 0 and every list empty, nothing writes to the deferred-job lists and `ResumeWorker` skips every worker, so the store can be flipped in a quiescent window. No dual-write, and no console script moving live job payloads between stores. This replaces two earlier approaches: - production#22975 used MultiStore dual-write. Its 2026-09-23 attempt enabled `use_primary_and_secondary_stores_for_concurrency_limit` in gprd at 06:50 UTC and disabled it at 07:03 UTC ([enable](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/work_items/64837), [disable](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/work_items/64838)). - The 2026-09-28 revision of this issue paused `ResumeWorker` with `drop_sidekiq_jobs_ConcurrencyLimit::ResumeWorker` and moved entries between stores with a console script. Superseded per discussion on this issue; see the 2026-09-28 comments. Both migration flags are `false` in gprd at the time of writing. Related work: - Superseded change request: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22975 - Rollout plan: https://gitlab.com/gitlab-com/gl-infra/tenant-scale/tenant-services/team/-/work_items/485 - Staging change request: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22903 - Application change: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252919 - Provisioning: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/15106 - Kubernetes wiring: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/5839 - Chef wiring: https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/8017 - Observability: https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11570 - Parent epic: https://gitlab.com/groups/gitlab-com/gl-infra/tenant-scale/tenant-services/-/work_items/72 ## Background production#22975 required every deferred-job LIST on SharedState to reach `LLEN == 0` before dual-write and before the read flip. `resume_processing!` reads by position from the default store and applies `LTRIM` to both stores, so pre-existing SharedState entries could trim unrelated MemoryStore entries. Two attempts could not reach that state: - 2026-09-22 12:00 UTC: the backlog oscillated between 0 and 37,881 within 15 minutes. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22975#note_3884274397. - 2026-09-23 06:00 UTC: `PackageMetadata::GlobalAdvisoryScanWorker` held above 500 jobs and drained too slowly to reach zero. See https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22975#note_3889527428. Dual-write also carries an ordering race that no gate prevents. `MultiStore#write_both` (`lib/gitlab/redis/multi_store.rb`) writes the default store, then the other store, without atomicity. Concurrent `RPUSH` calls from different pods can therefore land in a different order on each store. A positional `LTRIM` then removes different jobs from each store. ### Why disable the middleware rather than move entries between stores `sidekiq_concurrency_limit_middleware` is an ops flag read by `WorkerAttributes#get_concurrency_limit` (`app/workers/concerns/worker_attributes.rb`). Disabled, every worker reports a limit of 0. At limit 0 with an empty list: - `Middleware#should_defer_schedule?` returns false on its `worker_limit == 0` guard, so no job is deferred on enqueue. - `Middleware#should_defer_perform?` finds `has_jobs_in_queue?` false, and `ConcurrencyLimitService.over_the_limit?` returns false because the limit is 0. No job is deferred on perform. - `ResumeWorker#schedule_workers` skips every worker on `next unless queue_size > 0`, so no resume runs and no positional `LTRIM` is issued. Nothing reads or writes the deferred-job lists in that state, so flipping the default store moves no data and races with nothing. This removes the move function, the dual-write window, and the job-ordering concern in one step. The cost is that concurrency limits are off fleet-wide for the duration. Workers that would otherwise be throttled run unthrottled. The window is therefore chosen when concurrency-limit queue usage is low, so few workers are affected. Concurrency trackers and limits live in `QueuesMetadata` (`worker_execution_tracker.rb`, `limit_manager.rb`) and are not touched by this change. ## Change Details 1. **Services Impacted** - ~"Service::RedisClusterSharedState", `memorystore-redis-concurrency-limit`, Sidekiq 2. **Change Technician** - @eguavoee 3. **Change Reviewers** - @jarv @fshabir 4. **Scheduled Date and Time (UTC in format YYYY-MM-DD HH:MM)** - 2026-10-05 06:00 5. **Time tracking** - 1 hour 6. **Downtime Component** - none ## Target instance The gprd instance `concurrency-limit-redis` is 20 GB, STANDARD_HA, `replica_count 1`, REDIS_7_0, persistence DISABLED, `maxmemory-policy=noeviction`, at `10.239.32.236:6379`. Sizing rationale is recorded in tenant-services/team#476 and config-mgmt!15106. Resizing a MemoryStore instance under load took \~21 minutes during the quarantine shard evaluation, versus \~33s idle. Resize is therefore not an incident lever, and 20 GB is fixed for the duration of this change. ## Preparation ### Change Reviewer checklist ~C4 ~C3 ~C2 ~C1: - [x] Check if the following applies: - The **scheduled day and time** of execution of the change is appropriate. - The [change plan](#detailed-steps-for-the-change) is technically accurate. - The change plan includes **estimated timing values** based on previous testing. - The change plan includes a viable [rollback plan](#rollback). - The specified [metrics/monitoring dashboards](#key-metrics-to-observe) provide sufficient visibility for the change. ~C2 ~C1: - [x] Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). - The change plan includes success measures for all steps/milestones during the execution. - The change adequately minimizes risk within the environment/service. - The performance implications of executing the change are well-understood and documented. - The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? - The change has a primary and secondary SRE with knowledge of the details available during the change window. - The change window has been agreed with Release Managers in advance of the change. If the change is planned for APAC hours, this issue has an agreed pre-change approval. - The labels ~"blocks deployments" and/or ~"blocks feature-flags" are applied as necessary. ### Change Technician checklist Infrastructure prerequisites were verified for production#22975 on 2026-09-21: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22975#note_3877586510. That covers the instance configuration, Kubernetes secret, dashboard data, Vault auth string, Chef convergence, and Mimir rules. - [x] Confirm `redis-cluster-shared-state` is not already degraded. - [x] Record the pre-cutover `redis-cluster-shared-state` memory saturation baseline in a comment on this issue. - [x] Confirm this change does not overlap a Production Change Lock. - [x] Confirm no other gprd change is ~"change::in-progress". The chatops production check blocks feature flag changes on in-progress changes. - [x] Obtain a `prod-rails-console-ro` Teleport grant for the pre-execution and verification checks (see [runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/teleport/Connect_to_Rails_Console_via_Teleport.md)). Read-write is not required by this procedure. - [x] Re-validate the revised procedure in gstg and post the results as a comment on this issue. The 2026-09-24 rehearsal covered the superseded pause-and-move procedure and does not carry over. - [x] A dry-run has been conducted and results noted in a comment on this issue. - [x] For ~C1 and ~C2 change issues, the change event is added to the [GitLab Production](https://calendar.google.com/calendar/embed?src=gitlab.com_si2ach70eb1j65cnu040m3alq0%40group.calendar.google.com) calendar by the [change-scheduler bot](https://gitlab.com/gitlab-com/gl-infra/ops-team/toolkit/change-scheduler). - [x] For ~C1 and ~C2 change issues, Platform Leadership provides approval with the ~platform_leadership_approved label on the issue. Mention `@gitlab-org/saas-platforms/change-review-leadership` in this issue with a reference to [review guidelines](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/platform-leadership-review/) to get approval and provide visibility to all infrastructure managers. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, confirm with Release managers that the change does not overlap or hinder any release process (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [x] Mark the change scheduled only after every preparation item passes: `/label ~"change::scheduled"` ## Detailed steps for the change ### Pre-execution steps - [x] Make sure all tasks in [Change Technician checklist](#change-technician-checklist) are done. - [x] For ~C1 and ~C2 change issues, the SRE on-call has been informed prior to change being rolled out and has provided approval with the ~eoc_approved label on the issue. - [x] Check for [active \~severity::1 / \~severity::2 incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?sort=created_date&state=opened&label_name%5B%5D=Incident%3A%3AActive&or%5Blabel_name%5D%5B%5D=severity%3A%3A1&or%5Blabel_name%5D%5B%5D=severity%3A%3A2&first_page_size=20) before requesting approval. If one overlaps with this change's blast radius, reschedule. If it does not overlap, include a brief risk assessment in the `@sre-oncall` approval request; if the EOC is unavailable or does not approve, reschedule. - [x] For ~C1, ~C2, or ~"blocks deployments" change issues, Release managers have been informed prior to change being rolled out. (In `#production` channel, mention `@release-managers` and this issue and await their acknowledgment.) - [x] Confirm no auto-deploy is rolling the gprd Sidekiq fleet. - [x] Confirm the migration flags are disabled in gprd: `/chatops run feature get use_primary_and_secondary_stores_for_concurrency_limit` and `/chatops run feature get use_primary_store_as_default_for_concurrency_limit`. A flag that has never been set reports "does not exist on gprd"; with `default_enabled: false` that is the required state. - [x] Confirm `sidekiq_concurrency_limit_middleware` is currently enabled in gprd: `/chatops run feature get sidekiq_concurrency_limit_middleware`. - [x] Confirm `sidekiq_concurrency_limit_queue_jobs` is low on the Sidekiq dashboard. This is the low-usage precondition for the window. - [x] From a gprd Rails console, confirm `Gitlab::Redis::ConcurrencyLimit.params[:host]` returns `10.239.32.236` and `Gitlab::Redis::ConcurrencyLimit.pool.with { |r| r.ping }` returns `PONG`. - [x] From the same console, confirm the two stores are distinct: `Gitlab::Redis::ConcurrencyLimit.pool.with(&:inspect) != Gitlab::Redis::SharedState.pool.with(&:inspect)`. This is the comparison `MultiStore#same_redis_store?` makes. Do not compare `connection[:host]`: SharedState is a `Redis::Cluster` client and raises `NotImplementedError`. - [x] Run the list census (below). Record the per-worker counts in a comment. ### Change steps - steps to take to execute the change _Estimated Time to Complete (mins)_ - 20 - [x] Set label ~"change::in-progress" `/label ~change::in-progress` - [x] Disable concurrency limits: `/chatops run feature set sidekiq_concurrency_limit_middleware false --gprd`. - [x] Wait 2 minutes for feature flag caches to expire. The process-local cache TTL is 1 minute (`lib/feature.rb`). - [x] Confirm `sidekiq_concurrency_limit_queue_jobs` is draining to zero and `batch_resumed` events are being logged. At limit 0, `QueueManager#num_jobs_to_resume` returns `MAX_BATCH_SIZE` (5,000), so drain is fast. - [x] Run the list census. Confirm every deferred-job list is empty on **both** stores. Do not proceed until it is. If lists are not converging, abort per the criteria below. - [x] Flip the default store: `/chatops run feature set use_primary_store_as_default_for_concurrency_limit true --gprd`. Leave dual-write disabled. - [x] Wait 2 minutes for feature flag caches to expire. - [x] Re-enable concurrency limits: `/chatops run feature set sidekiq_concurrency_limit_middleware true --gprd`. - [x] Confirm new deferred-job writes reach `memorystore-redis-concurrency-limit` on the gprd dashboard. - [x] Run the list census once more. Expect entries on MemoryStore only, and none on SharedState. - [x] Confirm the final state is `dual-write=false`, `primary-default=true`, `sidekiq_concurrency_limit_middleware=true`. - [x] Set label ~"change::complete" `/label ~change::complete` Concurrency limits are off for about 10 minutes. Workers that would otherwise be throttled run unthrottled for that period. If the chatops production check blocks a flag change because of an unrelated ~"blocks feature-flags" incident or in-progress change, confirm the blocker is unrelated, then add `--ignore-production-check`. ### List census Read-only. Used at pre-execution, at the drain gate, and at verification. ```ruby CL = Gitlab::SidekiqMiddleware::ConcurrencyLimit ss = Gitlab::Redis::SharedState.pool ms = Gitlab::Redis::ConcurrencyLimit.pool key_for = ->(name) { CL::QueueManager.new(worker_name: name, prefix: CL::ConcurrencyLimitService::REDIS_KEY_PREFIX).redis_key } workers = Gitlab::SidekiqConfig.workers_without_default.map { |w| w.klass.name } # SharedState counts: workers.to_h { |w| [w, ss.with { |r| r.llen(key_for.(w)) }] }.select { |_, n| n > 0 } # MemoryStore counts: workers.to_h { |w| [w, ms.with { |r| r.llen(key_for.(w)) }] }.select { |_, n| n > 0 } ``` ### Final forward state - `use_primary_and_secondary_stores_for_concurrency_limit=false` - `use_primary_store_as_default_for_concurrency_limit=true` - `sidekiq_concurrency_limit_middleware=true` Do not disable the primary-default flag on its own while deferred jobs are on MemoryStore. Use the rollback procedure, which drains the lists first. ### Post-cutover observation _Estimated Time to Complete (mins)_ - ongoing, outside the change window These are tracked as exit criteria on tenant-services/team#485 and do not hold the change window open. [Post-cutover verification, 2026-10-05](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23004#note_3955481416) records console results, memory baselines, and outstanding observation. - [ ] Observe through at least one real throttling event. Confirm no `redis-cluster-shared-state` memory movement is attributable to it. - [x] Confirm new-instance memory tracks `sidekiq_concurrency_limit_queue_jobs`. Divergence means keys are landing somewhere unexpected. - [x] Record the post-cutover `redis-cluster-shared-state` memory saturation against the recorded baseline. - [x] Confirm `concurrency_limit:queue_manager:{worker}` lease keys remain on SharedState, as expected per tenant-services/team#482. - [ ] Confirm `SidekiqConcurrencyLimitQueueBacklogged` still fires on the same conditions. Its `IMPACT` text still names shared-state; that is corrected in tenant-services/team#486. ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change _Estimated Time to Complete (mins)_ - 20 Rollback is the forward procedure with the flip reversed. It drains the lists first, so no entries are stranded. - [ ] Disable concurrency limits: `/chatops run feature set sidekiq_concurrency_limit_middleware false --gprd`, then wait 2 minutes. - [ ] Run the list census. Confirm every deferred-job list is empty on both stores. - [ ] Disable MemoryStore as default: `/chatops run feature set use_primary_store_as_default_for_concurrency_limit false --gprd`, then wait 2 minutes. - [ ] Re-enable concurrency limits: `/chatops run feature set sidekiq_concurrency_limit_middleware true --gprd`. - [ ] Confirm new deferred-job writes reach `redis-cluster-shared-state` and `sidekiq_concurrency_limit_queue_jobs` behaves normally. - [ ] Set label ~"change::aborted" `/label ~change::aborted` If the change aborts before the default-store flip, re-enabling `sidekiq_concurrency_limit_middleware` is the whole rollback. ### Abort criteria Roll back if any of the following holds: - The deferred-job lists do not reach empty within 10 minutes of disabling the middleware. Non-convergence means arrivals are outpacing the drain, and the window is not low-usage enough. - The list census raises an error, or reports entries on SharedState after the flip. - Unthrottled execution during the limits-off window causes a visible impact on a downstream service. - `rails_redis_client` error rate for `memorystore-redis-concurrency-limit` above the SLI threshold. - New-instance memory saturation above 0.40. That is the s2 threshold from tenant-services/team#478. - Rejected writes on the new instance. Under `noeviction`, saturation rejects `RPUSH` of new deferred jobs. ## Monitoring ### Key metrics to observe - Metric: MemoryStore Redis SLIs and saturation - Location: https://dashboards.gitlab.net/d/memorystore-redis-concurrency-limit-main/memorystore-redis-concurrency-limit-overview?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd - What changes to this metric should prompt a rollback: new Redis errors, rejected writes, or memory saturation above 0.40. - Metric: `rails_redis_client` apdex and error rate for `memorystore-redis-concurrency-limit` - Location: the service dashboard above - What changes to this metric should prompt a rollback: error rate above the SLI threshold. - Metric: `redis-cluster-shared-state` memory saturation - Location: https://dashboards.gitlab.net/d/redis-cluster-shared-state-main/redis-cluster-shared-state-overview?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd - What changes to this metric should prompt a rollback: no rollback trigger. Expect a drop. This is the outcome being measured. - Signal: `sidekiq_concurrency_limit_queue_jobs` - What changes to this metric should prompt a rollback: grows while resumes are stopped, as expected. After resumes restart, a backlog that does not drain indicates `ResumeWorker` cannot read from the new instance. - Signal: `sidekiq_jobs_skipped_total{worker="ConcurrencyLimit::ResumeWorker", action="dropped"}` - What changes to this metric should prompt a rollback: none. It confirms resumes are stopped, and must return to zero after the flag is disabled. `SidekiqJobsSkippedTooLong` fires only after 3 hours of drops, so it does not fire during this window. ## Known Risk - Concurrency limits are off fleet-wide for about 10 minutes. Workers that would otherwise be throttled run unthrottled. The window is chosen when concurrency-limit queue usage is low to bound how many workers this affects. - While the middleware is disabled and a list is still draining, a new non-resumed job for that worker is still deferred at perform time, because `Middleware#should_defer_perform?` returns true on `has_jobs_in_queue?` before it evaluates the limit. Convergence therefore needs the drain to outpace arrivals, which is why the drain gate is an explicit abort criterion rather than a timed wait. - `ResumeWorker` continues to run while the middleware is disabled and still trims by position. With the lists empty or near-empty in the chosen window there is little to trim, but the flip must not happen until the census reads zero on both stores. - `sidekiq_concurrency_limit_middleware` is an ops flag and global. It removes limits for every worker, not only those with a backlog. - An admin `drop_jobs!` call during the window removes jobs by value from one store only. Do not run `drop_jobs!` during the change. ## Criticality C2. The procedure disables concurrency limits fleet-wide for a short window, which is more than a feature-flag flip on a single code path. Rollback is the same procedure with the flip reversed and is exercised in gstg before execution.