Add limited capacity worker for scheduled member deletion
What does this MR do and why?
In order to delete users from large hierarchies efficiently and without timeout, we are moving to an async process whereby we will add a limited capacity worker to process scheduled deletions.
A user will be scheduled for removal from a namespace hierarchy via the Members::DeletionSchedule
table (added in !170268 (merged)).
In this MR, we add a limited capacity worker that will look up records in that table, then proceed to handle removal of matching Member
records in the hierarchy in batches.
To err on the side of caution, we are also going to limit the worker to performing the removals in batches of 100 and limit execution time to 60 seconds.
We will keep the worker limited to 1 running job at any one time, which means we avoid the risk of saturating sidekiq with workers for 1000s of removals/memberships.
Refs #501248 (closed)
References
Please include cross links to any resources that are relevant to this MR This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.
- #501248 (closed) - issue for this change
- #501247 (closed) - issue for the related table
- #467281 (comment 2135360209) - issue/thread for discussion resulting in this approach
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- In rails console enable the experiment fully
Feature.enable(:limited_capacity_member_destruction)
- Visit any group or project member pages such as
http://127.0.0.1:3000/groups/flightjs/-/group_members
- Click the
invite members
button and add a user - Back in the rails console, create a scheduled deletion:
member = Member.last # the membership you just created user = member.user source = member.source Members::DeletionSchedule.create(user: user, namespace: source.root_ancestor, scheduled_by: User.first) # replace User.first with your user if you wish
- Next, we can schedule the new worker(s):
Members::SchedulePruneDeletionsWorker.new.perform
- Wait a few moments (for the workers to process) and confirm the user has been removed from the group/project hierarchy -->