[Experiment] Always run minimal jobs for fork pipelines

Abstract

In order to reduce the CI minutes consumed by fork pipelines, we want to experiment only running minimal jobs in fork pipelines.

The "full" jobs would only run for canonical pipelines (i.e. pipelines manually started by a member of the project) once the MR is approved.

Pros

• Automatically reduce the CI minutes consumed by wider community contributors, which is good for them and GitLab Inc (since GitLab Inc pay for shared runners).

Cons

• If maintainers don't run a canonical pipeline prior to merging, we could end up with broken masters. We don't expect this to happen as the process is well followed.

Timeframe

• 14 days - from 2022-02-17 to 2022-03-03

Expected results

• Less CI minutes consumed by MR pipelines from forks on average (metric to be created)
• 10-day moving average master success rate stays stable
• RSpec Minimal to Full pipeline transition success rate stays at or above 85%.
• Jest Minimal to Full pipeline transition success rate stays at or above 99%.

Risks

• Running less tests theoretically increases the chance to miss regressions, but given we only have 175 flaky tests tracked and our total number of tests is 159,590, it would only skip 0.1% of all our tests
• It could happen that master broken notifications wouldn't go down as flaky tests that we tracked are tests that end up passing on second or third try, so they might not end up actually breaking pipelines due to that

MR

• gitlab-org/gitlab!78707 (merged)

Communication

• 2022-02-17 Communicate in #development

  📣 We’ve started a 14-days experiment to only run minimal jobs for fork pipelines in order to reduce the CI minutes consumed by forks: #1170 (closed). If you notice anything weird related to RSpec jobs, please let us know in #g_engineering_productivity. 📣

• Add an item in the upcoming Eng. Week in Review

Once the experiment is validated:

• Add an item in the upcoming Eng. Week in Review
• Communicate in #i_saas_free_user_efficiency, #development, #eng-managers, #quality and more

How to enable

See https://docs.gitlab.com/ee/development/pipelines.html#flaky-tests.

• Run only minimal jobs in fork pipelines (gitlab-org/gitlab!78707 - merged) merged on 2022-02-17 at 12:07 UTC.

How to disable

Revert gitlab-org/gitlab!78707 (diffs) or manually set pipeline:run-all-rspec on MRs.