Optimize background migrations specs
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
While reviewing MTTP in the context of a recent incident, the rspec background_migration jobs jumped out as being in the critical path, taking ~80 minutes of gitlab-org/gitlab build time.
There had been some previous analysis in this space, as well as some previous work to improve things.
However, it looks like that work only addressed regular migrations, not background migrations.
From some initial analysis, it looks like the bulk of the overhead is coming from migrating to the finalized_by schema of the background migration. A few (semi-informed) ideas on how we might improve this situation:
- Extend the cutoff logic that we have for regular migrations to background migrations, skipping older specs.
- Reduce the set of unique
finalized_byschemas and group specs by schema version. - Run specs in order of
finalized_byschema, so that we can migrate incrementally instead of full-blownupanddownon every spec.
There's probably other approaches we can take to improve things here.
This has the potential to reduce build pipelines and MTTP quite a bit.
Side-note: The sample job shows big discrepancies between knapsack expected runtimes and actual time. We should check if knapsack measurement includes the database setup.