The QA testing procedure is not particularly helpful
With every release we produce, we create a corresponding QA issue, such as gitlab-org/release/tasks#623 (closed). There are a few problems with QA issues in their current state:
- Due to the size of our RCs, they tend to be really big. The QA issue mentioned above has 78 tasks to complete.
- Many of these tasks are never completed. This can be be due to any number of reasons such as: tasks hard to test, lack of ownership, or the sheer number of tasks.
- As a result of the above, I'm not certain they help us ensure quality of our changes. Even if all tasks were tested, it's likely most testing just comes down to "Browse to page X, check if it doesn't HTTP 503s". In other words, I'm not convinced the testing is in-depth enough to catch bugs we haven't already caught. Of course there might be exceptions to this.
In particular the QA issue model doesn't really work as we move towards continuous deployments, as we'd have hundreds of QA issues every week.
In the short term we might not be able to change anything, but on the longer term we need to phase out these QA issues with automated QA tests.