Test every merge request with GitLab QA

Description

We want to catch integration bugs before these are merged into the master. To do that, we need to test most of our merge requests with GitLab QA.

Currently package-and-qa is a manual action in the pipeline. It is still an optional manual action (we do support blocking manual actions already) because we don't want to negatively affect developers' productivity since GitLab QA needs a significant amount of time to finish and sometimes can produce false negatives. GitLab QA is currently production ready ➡ #126 (closed), but testing every merge request with it is still something that we do not enforce.

As a part of GitLab QA production readiness we decided to make GitLab QA fast and reliable. However there is still some room for improvement here, if we would like to run GitLab QA for every merge request. I below describe some things we could consider doing before making GitLab QA a required / blocking manual action on every merge request.

Problems

1. It currently takes around 10075 (as of June 14th, 2018) minutes to test a merge request with GitLab QA

   1. It takes around 6035 (as of June 14th, 2018) minutes to build a package in Omnibus
   2. It takes around 3035 (as of June 14th, 2018) minutes to go through GitLab QA pipeline
   3. Lack of multi-project pipeline adds up to 5 minutes to this process because of the polling interval.

2. False negatives are still a thing

   Because we do not enforce testing every merge request with GitLab QA, integration bugs can still sneak into the master. We recently started creating an issue for every legitimate failure using ~failure label in the issue tracker, but failures that are false negatives are sometimes also a case on master. To mitigate false negatives we recently implemented selectors validation mechanism ➡ #58 (closed), it helped a lot, but this is a 80 results / 20 percent effort solution. It would be great to increase accuracy of this check up to 90% percent if we want to avoid false negatives, that are usually caused by selectors changed in CE / EE, and qa/ code not updated accordingly. One thing that can help is ➡ #164 (closed).

3. Cost of running a GitLab QA test for a merge request

   We strive for being frugal. Currently we estimate that the cost of running GitLab QA for one merge request might be around 2$. It is a lot, and running GitLab QA for every merge request might result in a significant bill in the scale of a year, depending on our growth and number of merge requests tested. There are few ways of reducing this cost, and we should consider doing that before enabling GitLab QA everywhere.

4. Not testing every merge request

   I'm not sure if this makes sense to run GitLab QA for literally every merge request, given the cost and time needed to do it. Even we if plan to test 90% of merge requests with GitLab QA, we still might want to make it possible to skip it for documentation changes, minor changes like typos etc. In order to do that we can use variables expression feature, we are going to ship in this release (10.7) with a few more features that are on road map to do something like in gitlab-ci.yml: qa: except: variables: - $CI_COMMIT_MESSAGE =~ /skip-qa-tests/.

Solution

After resolving concerns / problems, and deciding that we can now test every merge request, we can make package-and-qa manual action, a blocking manual action easily. We currently are not using this feature in GitLab, but it is there to allow us to implement gating in the pipeline.

/cc @bjgopinath @meks @jacobvosmaer-gitlab @rymai