Skip to content

Change batch size depending on how far off we are

Andreas Brandl requested to merge ab/optimize-batched-migrations into master

What does this MR do?

This changes the batch size optimizer so that it does not increase/decrease batch size by a static multiplier (20% up, 10% down) but rather tries to target the desired "time efficiency" and adjust the batch size so that theoretically we would hit the target efficiency (0.95).

This is based on the assumption that the duration of a job is linear in the batch size. We calculate how far off we are and adjust the batch size according to that.

Additionally, we limit the increase step to 20% maximum.

Another change here is to allow the batch size to go up to 2M.


The current version of this has been enabled on .com and works out fine.

There is no risk in increasing the batch size to very high values - we won't overload the database with this, given that the sub batch size stays untouched. However, jobs will take longer breaks in between - but that's all.

Does this MR meet the acceptance criteria?

Conformity

Edited by Andreas Brandl

Merge request reports