Implement first iteration of (async) BBM process in container registry
Implement the first iteration of async BBM process discussed in PoC: evaluate proposed database background migr... (#1241 - closed) in the container repository repo. This first iterations must be behind a feature flag that starts the bbm process (off by default) and follows the proposal defined in https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/spec/gitlab/database-background-migrations.md?ref_type=heads.
The first iteration should have/have-support for:
- checking for and scheduling asynchronous BBM and periodically queuing eligible jobs
- Using user defined configurations for setting bbm run frequency
- Distributed bbm run lock synchronization
- Creating and updating the statuses of bbm and bbm jobs
- unit and integration tests
- retry and recovery functionality
- observability (metrics and logs)
- Querying the background_migration tables in-order to manage the background migration job runs
There's a lot of moving pieces in this issue and so it is important to tackle this iteratively through smaller MRs targeting the same goal. The impact of this change on the functionality of the registry at the moment is low given the feature will be initially off.
This first version will potentially generate a few follow up issues from the team after review, the follow up issues will need to be evaluated and addressed where necessary before we can proceed with potentially rolling this out to testing and staging environments