## Goal :dart: Add CI job which would emulate multi-version* upgrades by running `bundle exec rake gitlab:db:configure` against DB dump from latest known GitLab version stop with test data. :star: Parent epic - https://gitlab.com/groups/gitlab-org/-/epics/9201+ *multi-version upgrades - upgrade that follows [upgrade path](https://docs.gitlab.com/ee/update/index.html#upgrade-paths) and skipping some versions, for example 16.0 -\> 16.3 -\> 16.7 #### Challenge No existing unit-level tests  for multi-version migration testing in GitLab project CI. #### Business Outcome - Catch migration errors for multi-version upgrades in merge requests at unit-level. - Save costs by running unit tests without building actual environments ## High-level overview :notepad_spiral: [Epic DRI](https://handbook.gitlab.com/handbook/engineering/infrastructure/team/gitlab-dedicated/#epic-owner-responsibilities): @niskhakova - [x] Proof-of-concept https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1806+ - [x] Explore seeding options for the DB dump with data - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1919+s - [x] Prepare and store DB dump for the latest known GitLab version - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1920+s - [x] Add multi-version upgrades test to GitLab CI - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1921+s - [x] Expand data seeding configuration for multi-version upgrades testing - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354+s - [x] Identify solutions for storing DB dumps for multi-version upgrade testing - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2355+s - [x] Iterate on multi-version upgrades CI test configuration - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2356+s - [x] Monitor test stability and results - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1922+s - [x] Make multi-version upgrades job required to pass - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1923+s - [x] Create follow up for background migrations testing - https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/62 ## Latest status Completed :tada: See https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/19#note_1859948872 <details><summary>Previous status updates </summary> #### 2024-03-28 https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5928#note_1837062985 #### 2024-03-22 https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5928#note_1828217200 #### 2024-03-15 https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5928#note_1817440697 #### 2024-03-08 Latest status update in OKR: https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5928#note_1807812515 #### 2024-03-01 Latest status update in OKR: https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5928#note_1797676186 #### 2024-02-23 Done: - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354+s - ~"Progress::ongoing" - Tested out suggested scenario for breaking migrations for validating the seeding approach - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354#note_1786905633 - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2355+s - ~"Progress::ongoing" - Implement script for PG Dump Generator that follows outlined design https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2355#note_1783504407, Updated existing multi-version upgrade test job and switched it to allow to fail from previous `manual`.https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2355#note_1783749422 - `db:migrate:multi-version-upgrade` is now running in MRs as allowed to fail :tada: Next steps: * https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354+ - sync with Database team for scenario that intentionally breaks migrations for validating the seeding approach * https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2355+ - address https://gitlab.com/gitlab-org/gitlab/-/issues/442778+, enable CI pipelines for generating PG dumps * https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2356+ - review how multi-version upgrades test job will work with multiple PG version support, review rules when job should be triggered * https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1922+ - start looking into monitoring the stability of the job #### 2024-02-16 Done: - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354+s - ~"Progress::ongoing" - Created https://gitlab.com/gitlab-org/gitlab/-/merge_requests/144631 which adds config to seed all fixtures using Data Seeder, Resolved most frequent fixture seeding error related to bundle files (collaborated with Gitaly team) - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354#note_1775000308, Reached out to Database team for review of the approach and advise on validation - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354#note_1775308818 - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2355+s - ~"Progress::started" - created script design for creating PG dumps - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2355#note_1777067946 Next steps: * https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354+ - work with Database team on the feedback for approach, explore options for intentionally breaking migrations for validating the seeding approach * https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2355+ - implement initial script for PG Dump Generator #### 2024-02-09 Done: - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1919+s - ~"workflow::complete" - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 was merged, @ddavison closed relevant issue as done - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1920+s - ~"workflow::complete" - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 was merged, @ddavison closed relevant issue as done - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1921+s - ~"workflow::complete" - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 was merged, @ddavison closed relevant issue as done - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354+s - ~"Progress::ongoing" - @niskhakova caught up with Seeder and MR!137582 changes, [validated](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354#note_1758876296) that data selected approach work by creating DB dump for known problematic upgrade path and verify that there is a migration error, [started](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354#note_1762969866) exploring options for seeding environment with all possible fixtures Next steps: - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354+s - Proceed exploring options for seeding environment with all possible fixtures, work on design for realistic data for upgrade testing with Seeder #### 2024-02-02 Done: - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1919+s - ~"Progress::ongoing" - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 is in final review stages, @ddavison worked on review feedback - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1920+s - ~"Progress::ongoing" - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 is in final review stages, @ddavison worked on review feedback - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1921+s - ~"Progress::ongoing" - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 is in final review stages, @ddavison worked on review feedback - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354+s - ~"Progress::started" - @niskhakova catches up with Seeder and MR!137582 changes Next steps: - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 - continue to work on review feedback - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2354+s - Validate that data selected approach work - create DB dump for known problematic upgrade path and verify that there is a migration error #### FY24Q4 / 2024-01-26: - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1919+s - ~"Progress::ongoing" - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 is in review - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1920+s - ~"Progress::ongoing" - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 is in review - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1921+s - ~"Progress::ongoing" - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137582 is in review </details>