The issue is to track areas where QE can help to increase quality and stability working with Distribution team. ### Identified areas - Add initial test coverage for GitLab **upgrades** - Problem: No known existing frameworks for testing GitLab upgrades. Our performance pipelines are updated mostly with nightlies and it's different from the path with which customers are updating their environments. - Actions: - Review the current test coverage and improve it - https://gitlab.com/groups/gitlab-org/-/epics/9201 - Priority: 1 - GitLab **installation** test coverage - Problem: Distribution has added multiple new OS'es recently like Almalinux, AL2, etc. No pipelines to verify the packages for these OS'es continue to work. - Actions: - Review the current test coverage - https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/882 - Create pipeline to trigger GitLab installation for supported OS - TBD - Priority: 2 - Review and streamline **QA pipelines** in Distribution projects - Problem: There are multiple QA pipelines, but it's not well documented what to do if something fails - Actions: - Review current pipelines, explore what can be improved, how failures are tracked, document current coverage, document how to debug failures, collaborate with team if a demo is needed - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1498 - Priority: 2 - **Integration tests** for Omnibus-GitLab specific functionality - Problem: Team needs to manually test scenarios like PG failover and upgrades - Actions: - Review current test coverage and what scenarios are needed to be add (pg-upgrade, automatic PG failover), explore adding automated test pipeline - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1499 - Priority: from Design thinking sessions - **Ref Arch testing for helm charts** - Details: A set of test validation 'ref architectures' for helm charts. So we are exercising more than one deployment type, and do it in such a way that devs can deploy the same arch for their local clusters - Actions: - Add Cloud Native Reference Architecture test pipeline in Helm Charts - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1500 - Priority: from Design thinking sessions - **Dependency updates** test automation - Problem: We often have to bump the bundled softwares to keep up with latest releases, security patches etc. While we can create a new package with the new version and test it, there is currently no general procedure depicting what all to test for. - Actions: - Gather the list of tests that should be done before dependency upgrade - https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2347 - Explore what can be automated - TBD - Priority: 1 - Test **Operator upgrades** - https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1616 - Problem: currently we're not testing for Operator upgrades (nor for upgrades facilitated by Operator) - Actions: - Add test automation for Operator upgrades - TBD 1. provision "previous" version of operator with "previous" version of chart deployed 1. upgrade operator 1. upgrade GitLab deployment - Priority: 3 - **Cloud Native test coverage** - https://gitlab.com/groups/gitlab-org/quality/quality-engineering/-/epics/7 - Problem: Test gaps in CNG - Actions: - Review and improve Cloud Native test coverage - Priority: 3 #### Sources - QE Distribution dashboard: https://gitlab.com/groups/gitlab-org/-/boards/2187925?label_name[]=Quality&label_name[]=devops%3A%3Asystems&label_name[]=group%3A%3Adistribution - Distribution brainstorm: https://app.mural.co/t/gitlab2474/m/gitlab2474/1654609389941/c8ce8561287f4e1d63091e0fe364ca5916f6c3fa?sender=u3a529c23c09585bede780421 - Distribution team retrospectives