Add support of worker_class param in loose_foreign_keys config
What does this MR do and why?
This change adds the ability to assign specific database cleanup tasks to different worker processes instead of having one worker handle everything. Previously, all loose foreign key cleanup (which maintains data consistency when records are deleted) was handled by a single default worker. Now you can configure certain database tables to be processed by custom workers, allowing for better load distribution and specialized handling of different types of data.
The main changes include:
- Adding a new
worker_classconfiguration option that lets you specify which worker should handle cleanup for specific tables - Modifying the service to filter tables based on which worker is running, so each worker only processes its assigned tables
- Updating documentation to explain how to configure custom workers
- Adding comprehensive tests to ensure workers only process their assigned tables
We implemented the worker_class attribute for loose foreign keys because we need to add a custom worker specifically for processing merge_request_diff_commit records.
- The default
LooseForeignKeys::CleanupWorkerprocesses all loose foreign key cleanup in a single worker-
merge_request_diff_commitrecords have unique characteristics that require specialized handling:- High volume of records
- Different cleanup pattern or performance requirements
- Potential need for custom processing logic
-
References
- More discussion on this is present in this confidential issue https://gitlab.com/gitlab-org/gitlab/-/issues/556717#note_2663377065
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #559200