Explore migration testing for multi-version upgrades without building an environment
Problem
Self-managed customers catch migration errors when doing multi-version upgrades which are not always caught prior to release by existing test coverage for migration and upgrade testing.
Context
There are cases when update-major
and update-minor
jobs that use Test::Omnibus::UpdateFromPrevious
QA scenario catch migration errors (gitlab-org/gitlab#409331 (closed), gitlab-org/gitlab#408304 (closed)).
However these tests are not triggered in every MR and it takes a long time for job to finish due the nature of package-and-test
(1.5-2 hours).
multi-minor-version is an upgrade like 15.4 -> 15.7 that skips 15.5 and 15.6 (#1270 (comment 1378880407))
We plan to work on end-to-end upgrade testing solution at Improve test coverage for planned upgrade stops (gitlab-org&9201), however we should explore if there is a way to add light-weight test and shifting-left that would allow to skip building an env and doing the actual upgrade.
More context: #1270 (comment 1374137658)
Goal
Emulate multi-version upgrades migrations without building an actual environment or GitLab docker image. If it's possible to achieve, explore making such job required for all DB schema changes.
Concept
- There is DB dump with extensive test data seeded available from the latest known GitLab Upgrade stop. (for example 15.4.6)
- MR is created that changes DB schema, it triggers multi-version upgrade migration test job
- Job imports DB dump
- Emulates migration against the DB dump using the code changes from MR (emulate jump from 15.4.6 to 16.0)
- Each known upgrade stop has DB dump
Proposal
- Create DB dump from latest known GitLab version stop with some test data in place so that it's not empty and store it somewhere (
db:schema:dump
or another way) - Create a job which will
- Import DB dump from step 1 (
db:schema:load
or another way) - Use latest code from GitLab master, runs
gitlab:db:configure
or a similar command to replicate what Omnibus does during upgrades
- Import DB dump from step 1 (
- Create DB dumps for all releases and extend testing above to follow multiple version upgrades
Questions
- Is it possible to create such a job and will it be helpful?
- Are there other ways to catch multi-version migration errors without building an environment/image? (emulate what Omnibus does during upgrade and reconfigure)
- Are there other areas that could be helpful to focus on to shift left and catch such errors earlier (ideally in MR)?
- Is there some existing test for data/schema changes? If yes, can it be updated to catch multi-version upgrades issues?
- What are the limitations of such lower level testing? For example, not being able to test background migrations?
- Are there other quick-win issues that can be helpful to expand multi-version migration testing?