Skip to content

[Experiment] Always run minimal jobs for fork pipelines

Abstract

In order to reduce the CI minutes consumed by fork pipelines, we want to experiment only running minimal jobs in fork pipelines.

The "full" jobs would only run for canonical pipelines (i.e. pipelines manually started by a member of the project) once the MR is approved.

Pros

  • Automatically reduce the CI minutes consumed by wider community contributors, which is good for them and GitLab Inc (since GitLab Inc pay for shared runners).

Cons

  • If maintainers don't run a canonical pipeline prior to merging, we could end up with broken masters. We don't expect this to happen as the process is well followed.

Timeframe

  • 14 days - from 2022-02-17 to 2022-03-03

Expected results

Risks

  • Running less tests theoretically increases the chance to miss regressions, but given we only have 175 flaky tests tracked and our total number of tests is 159,590, it would only skip 0.1% of all our tests
  • It could happen that master broken notifications wouldn't go down as flaky tests that we tracked are tests that end up passing on second or third try, so they might not end up actually breaking pipelines due to that

MR

Communication

  • 2022-02-17 Communicate in #development

    📣 We’ve started a 14-days experiment to only run minimal jobs for fork pipelines in order to reduce the CI minutes consumed by forks: https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1170. If you notice anything weird related to RSpec jobs, please let us know in #g_engineering_productivity. 📣

  • Add an item in the upcoming Eng. Week in Review

Once the experiment is validated:

  • Add an item in the upcoming Eng. Week in Review
  • Communicate in #i_saas_free_user_efficiency, #development, #eng-managers, #quality and more

How to enable

See https://docs.gitlab.com/ee/development/pipelines.html#flaky-tests.

How to disable

Revert gitlab-org/gitlab!78707 (diffs) or manually set pipeline:run-all-rspec on MRs.

Edited by Rémy Coutable