Skip to content

Convert to one batched migrations execution worker per database

Krasimir Angelov requested to merge 372316-split-execution-worker-per-database into master

What does this MR do and why?

While working on Convert batched migrations execution workers to... (!104027 - merged) I realized that, because of the way the redis key for LimitedCapacity::JobTracker is generated, max_running_jobs will be shared between migrations for main and ci databases, which is not what we want.

This MR introduce two new classes, MainExecutionWorker and CiExecutionWorker, that include the shared code form the ExecutionWorker module. This is the same approach we use for the scheduling cron workers, for example CiDatabaseWorker. This way when we convert them to limited capacity workers,running migrations for each database will be tracked in a separate redis set, and independent from each other.

I was hoping we can keep having just one class, and generate the redis key dynamically, but it's not easy, as the job_tracker instance is generated in LimitedCapacity::Worker#perform which is before any code from the workers is called.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #372316 (closed)

Edited by Marius Bobin

Merge request reports