Resolve sidekiq spikes when a User is banned
Problem
Banning a user via the API generated 2M+ Sidekiq jobs, caused production outages, and timed out at 60 seconds leaving users partially banned.
POST /api/v4/users/:id/block → user.block! → after_transition → run_after_commit → calls DropPipelinesAndDisableSchedulesForUserService synchronously within the same Rack request → walks all projects, enqueues thousands of DropPipelineWorker jobs → hits 60-second timeout, leaves user partially blocked.
Solution
POST /api/v4/users/:id/block → user.block! → after_transition → run_after_commit → enqueues one BlockUserPipelineCleanupWorker job → API returns immediately
Then asynchronously in Sidekiq:
BlockUserPipelineCleanupWorker(low urgency) walks projects/pipelines, enqueues oneBlockUserDropPipelineWorkerper pipeline- Each
BlockUserDropPipelineWorker(low urgency) drops the jobs within that pipeline
So the three fixes together:
.indexed_alive_statusesfilter — includes pipelines with statuses["preparing", "pending", "running", "waiting_for_resource", "created"]entirely, reducing the total job countBlockUserPipelineCleanupWorker— moves orchestration out of the Rack request, fixing the timeout and partial-block problemBlockUserDropPipelineWorker— low urgency, so the flood of drop jobs can't starve PipelineProcessWorker for other users' active pipelines
This fixes https://gitlab.com/gitlab-org/gitlab/-/work_items/538338
Edited by Shabini Rajadas