Make GitLab QA production ready

Description

We are going to add more tests to GitLab QA, and we want to make this project really useful in the future. But it is not production ready yet.

TL;DR - minimal steps to make GitLab QA production ready:

1. qa-package button incorporated into development process, green pipeline before merging every merge request.
2. Make it easier for developers to add tests in a way that GitLab QA itself keeps a high quality and is maintainable in a long term.

What does it mean?

By saying production ready GitLab QA I mean that:

1. GitLab QA is able to provide quick feedback about integration problems in merge requests, before these are merged.
2. GitLab QA is easy to extend, and developers can easily implement new tests.
3. GitLab QA can quickly adapt to changes in GitLab itself.
4. GitLab QA is fast and reliable, false negatives are rare.

What is the current status

1. Currently we are running GitLab QA against nightly builds.

   This already allowed us to catch some major integration problems in the past, but it is a fragile approach ➡️ gitlab-org/gitlab-qa#6 (closed).

   We are also working on making it possible to run GitLab QA from within merge requests ➡️ https://gitlab.com/gitlab-org/gitlab-ce/issues/39637, but it is still not finished. We still need to make it fast, and there is still a single source of truth problem that we need to solve. Nevertheless we made some good progress and this can also help to make GitLab QA more reliable and reduce false negatives in nightly pipelines.

2. GitLab QA is not easy to extend yet.

   Because we wanted to implement fully click-driven black-box testing strategy we had to resolve a lot of tricky problems. Managing side-effects, idempotent and isolated tests, deterministic scenarios, all these things impose a lot of essential complexity. It works, so I think that we managed to solve some of these problems, but it is not done yet in the extent that we can say that GitLab QA is predictable and we have an established convention of adding new tests.

   We do not have much tests implemented yet, because of fragile tests problem ➡️ gitlab-org/gitlab-qa#6 (closed) we avoided adding new tests before having qa-package manual action ➡️ https://gitlab.com/gitlab-org/gitlab-ce/issues/39637. This means that we had too few tests to triangulate and devise a good interface for managing click-driven approach.

   What we have now was the boring solution, but it needs a few refinements. We had a call about it with @mbergeron and @felipe_artur recently, and it is apparent that we need to reduce coupling between Scenario / Factory objects ➡️ gitlab-org/gitlab-qa#86 (closed), and we need more features like traits ➡️ gitlab-org/gitlab-qa#122 (closed).

   It shouldn't be too hard to refactor our codebase to reduce this coupling and to make factories more reusable + support traits, we can use common dependency inversion techniques like dependency injection to make it possible to return value objects from factories, eg. a project object that will contain the project URL, which can lead to a significant performance improvement too ➡️ gitlab-org/gitlab-qa#123 (closed).

3. Making it easier to adapt to changes in GitLab itself is important too.

   When someone removes some page in GitLab, or updates navigation menus (what happens around three times during last year or two), we should not spend weeks on updating GitLab QA. I think that is the problem that we actively work to solve since some time, and qa-package button + scenarios / factories that minimize duplication help a lot already. Redesigning factories to return use value objects + using dependency injection should improve this even further.

4. 🚀 🌲 Fast and reliable.

   Like every test suite, GitLab QA should be fast and reliable. We made some decent progress with making it fast, and with bringing qa-package to merge requests. We are working with ~Build team to improve package build duration even more and since a few days we can use qa-package button in merge requests. That said, we should keep in mind that complexity of GitLab is not going to decrease a lot in the future, and that we need to manage complexity of GitLab too. It might be easier to optimize something that we can actually measure, so we might want to store test metadata ➡️ gitlab-org/gitlab-qa#125 (closed) use this data to generate pages / push to an external service to see measurements and progress 🚀

Please take a look at this issue @rymai @bjgopinath. Thanks!