:construction: This roadmap is still in progress :construction: ## Goal Increase test coverage for GitLab Upgrades to ensure that if customer is following [upgrade path](https://docs.gitlab.com/ee/update/index.html#upgrade-paths) the upgrade is successful. ## Customer facing problems From previous discussions with Support team, the main problems that customers encounter during upgrade: 1. Unplanned upgrade stops - errors caused by migrations ( https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1270#note_1374819271) - Example: https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1529 2. Functionality degradation between upgrades - feature regression between upgrades which requires customers to perform multiple upgrades to leverage bug fixes. - Example: When customer finds a bug, GitLab team recommends to upgrade to X where bug is fixed, then customer upgrades to X - and finds another bug, and GitLab team recommends upgrade to X+n where it was fixed in Y -\> then they need to repeat this all again <details> <summary> Current coverage :mag_right: </summary> 1. Lower level testing - [Unit tests for `db` rake tasks without building an environment](https://docs.gitlab.com/ee/development/database/database_migration_pipeline.html) 2. System level testing - upgrade tests against GitLab environments 1. Docker Upgrade testing - [GitLab QA upgrade testing jobs using GitLab Docker images](https://handbook.gitlab.com/handbook/engineering/infrastructure/test-platform/test-coverage/#gitlab-qa-update-scenario) - covers multi-version upgrades. For minor update test jobs - tests upgrade from latest known upgrade stop to the latest code from `master` (example 16.3 -\> `master`). For major update test jobs - tests upgrade from latest known major-1 version to the latest master (example 15.11 -\> 16.1 -\> 16.3 -\> `master`) 2. Self-Managed Upgrade testing - [Performance environments nightly upgrades](https://handbook.gitlab.com/handbook/engineering/infrastructure/test-platform/test-coverage/#performance-environments-nightly-upgrades) </details> # High-level work overview :mag: The below is proposed work for improving Upgrade testing coverage: <table> <tr> <td>Milestone</td> <td>Goal(s)</td> <td>Issue(s) / Epic (s)</td> <td>Status</td> </tr> <tr> <td rowspan="3">Test gaps review</td> <td rowspan="3">Review existing test coverage, identify test gaps</td> <td> https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1270+s  </td> <td> Done :white_check_mark: </td> </tr> <tr> <td> https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2320+s </td> <td> Done :white_check_mark: </td> </tr> <tr> <td> https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1846+s </td> <td> To be scheduled :calendar_spiral: </td> </tr> <tr> <td>Planned upgrade stops</td> <td> _\[Stage 1\]_ Ensure that if customer is following [upgrade path](https://docs.gitlab.com/ee/update/index.html#upgrade-paths) there are no unknown/unplanned upgrade stops. </td> <td> Epic: https://gitlab.com/groups/gitlab-org/-/epics/9201+s </td> <td> FY25 Q1, FY25 Q2 Done :white_check_mark: </td> </tr> <tr> <td>Planned upgrade stops</td> <td> _\[Stage 2\]_ Ensure that if customer is following [upgrade path](https://docs.gitlab.com/ee/update/index.html#upgrade-paths) there are no unknown/unplanned upgrade stops. </td> <td> Epic: https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/framework/-/epics/3+ </td> <td> To be scheduled :calendar_spiral: </td> </tr> <tr> <td>Functionality consistency</td> <td>Reduce functionality degradation between upgrades by building processes to prevent regressions</td> <td> Epic:  https://gitlab.com/groups/gitlab-org/-/epics/12457+s </td> <td> To be scheduled :calendar_spiral: </td> </tr> <tr> <td>Downgrade testing</td> <td>Ensure that if customer is following documented guidance the downgrade is successful</td> <td> Epic: https://gitlab.com/groups/gitlab-org/-/epics/13023+ </td> <td> To be scheduled :calendar_spiral: </td> </tr> </table>