Change batch size depending on how far off we are
What does this MR do?
This changes the batch size optimizer so that it does not increase/decrease batch size by a static multiplier (20% up, 10% down) but rather tries to target the desired "time efficiency" and adjust the batch size so that theoretically we would hit the target efficiency (0.95).
This is based on the assumption that the duration of a job is linear in the batch size. We calculate how far off we are and adjust the batch size according to that.
Additionally, we limit the increase step to 20% maximum.
Another change here is to allow the batch size to go up to 2M.
The current version of this has been enabled on .com and works out fine.
There is no risk in increasing the batch size to very high values - we won't overload the database with this, given that the sub batch size stays untouched. However, jobs will take longer breaks in between - but that's all.
Does this MR meet the acceptance criteria?
Conformity
-
📋 Does this MR need a changelog?-
I have included a changelog entry. -
I have not included a changelog entry because feature flag.
-
-
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content