Resolve sidekiq spikes when a User is banned

Problem

Banning a user via the API generated 2M+ Sidekiq jobs, caused production outages, and timed out at 60 seconds leaving users partially banned.

POST /api/v4/users/:id/blockuser.block!after_transitionrun_after_commit → calls DropPipelinesAndDisableSchedulesForUserService synchronously within the same Rack request → walks all projects, enqueues thousands of DropPipelineWorker jobs → hits 60-second timeout, leaves user partially blocked.

Solution

POST /api/v4/users/:id/blockuser.block!after_transitionrun_after_commit → enqueues one BlockUserPipelineCleanupWorker job → API returns immediately

Then asynchronously in Sidekiq:

  • BlockUserPipelineCleanupWorker (low urgency) walks projects/pipelines, enqueues one BlockUserDropPipelineWorker per pipeline
  • Each BlockUserDropPipelineWorker (low urgency) drops the jobs within that pipeline

So the three fixes together:

  • .indexed_alive_statuses filter — includes pipelines with statuses ["preparing", "pending", "running", "waiting_for_resource", "created"] entirely, reducing the total job count
  • BlockUserPipelineCleanupWorker — moves orchestration out of the Rack request, fixing the timeout and partial-block problem
  • BlockUserDropPipelineWorker — low urgency, so the flood of drop jobs can't starve PipelineProcessWorker for other users' active pipelines

This fixes https://gitlab.com/gitlab-org/gitlab/-/work_items/538338

Edited by Shabini Rajadas

Merge request reports

Loading