## Short description Build/adapt a framework which shares code with batched background migrations, that will allow teams to perform large data changes during operation, but outside of the migration framework, with the ability to be repeated long term, on a cron schedule, or triggered by an event. NOTE: The intention is not to improve the existing BBM framework, but to extend it with its current state to more players (sidekiq, cron, adhoc actions). ### Use cases https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/batched_background_operations/#use-cases ## Business Justification Performing large data operations without enough measures are the common problem for multiple spikes in database usage, which also affects other areas of the application. Without a system, it increases the debug and fix time around these issues. This aims to avoid them by sub-batching large data operations with enough throttling measures based on the db health status. This aligns with FY26 Objective 3, to be more thorough and improve the efficiency to provide better value for the customers. ## Expected impact - Less database spikes/incidents because of long running large data operations. - Workers don't have to manually apply necessary measures (db throttling, sub-batching, sleep between batches, etc) but can rely on the new framework for them. ## Exit criteria - [x] Authors are able to create jobs that safely process data in the background using a BBM like framework - [ ] Jobs should be able to be scoped to cell or organization - [ ] Jobs must work with cells - [ ] Must have Documentation - [x] The lifecycle of BBOs, how it gets processed and it's workflow. - [ ] Management through chatops - [ ] How to monitor failed operations ## Deliverables ### %&quot;18.3&quot; * [x] PoC to make sure tables are designed in a scalable manner. * [x] Analyse existing workers to find the pattern for BBO design. ### %&quot;18.4&quot; * [x] Complete the design doc and share with interested feature teams. * [x] Remove the Legacy Background worker code, as it's all over the place and confusing with BBM code. ### %&quot;18.5&quot; * [x] Create tables to support Background operations. * [x] Add support to enqueuing Background operations. * [x] Cell Local: Instance wide, mostly queued from cronjobs. * [x] Organization Level: User specific, can be triggered by user action from anywhere in the application. ### %&quot;18.6&quot; * [x] Add support to consume Background operations enqueued to the table * [x] Concurrent workers are setup to consume the operations, and each BBO is consumed only once. * [x] Cron to call the Background operations consumer every minute * [x] Add actual processing logic for Background operations * [x] Sub batches similar to BBM. * [x] Db based health check is in place. * [x] Logs are generated correctly. ### %18.7 - %&quot;18.8&quot; * [x] Integrate all the pieces and fix any bugs that comes up. * [x] Do a dry run by migrating an existing worker. **Note:** Team was at low capacity with `@praba.m7n` becoming Acting EM for DBF, `@l.rosa` helping with DDL migrations locks and incidents and the Christmas and followed by F&F days. * We also worked on a few items from %18.6 in these milestone, also could take a couple from %&quot;18.9&quot; post holidays based on the availability. ### %"19.0" * [ ] Sign off on cells compatibility for Background operations framework * [ ] Chatops support to see the progress of Background Operations. ### %"19.1" * [ ] Feature flags are setup to disable Background operations based on the worker name. * [ ] Documentation completion * [ ] Announce availability to #engineering-fyi, #database, and anywhere else deemed a good idea <!-- STATUS NOTE START --> ## Status 2026-05-26 We are close to the completion of this epic! Pending items are making it compatible with cells, providing chatops support and more documentation. :tada: **achievements**: * Chatops support: List endpoint was merged in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/236232 and show endpoint https://gitlab.com/gitlab-org/gitlab/-/merge_requests/237673 is in review. * Cells compatibility: Using Current.organization on enqueuing background operations is in final review https://gitlab.com/gitlab-org/gitlab/-/merge_requests/236803. :arrow_forward: **next**: * Similar to sidekiq ([org read-only adr](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/organization/decisions/010_organization_read_only_mode/#policy)) we have to make org-specific background operations skip processing workers on org in read-only mode. * Capture cells compatibility and chatops support sections in the docs https://docs.gitlab.com/development/database/background_operations. cc: @morefice @l.rosa @panoskanell _Copied from https://gitlab.com/groups/gitlab-org/-/epics/16152#note_3383252136_ <!-- STATUS NOTE END -->