Skip to content

Experiment with failing pipelines early in Merge Request pipelines

What is this experiment?

To provide quicker feedback when a Merge Request introduces failing tests, the Engineering Productivity team is experimenting with a fail fast mechanism on Merge Request pipelines.

With this, an MR that introduces failing tests would receive earlier feedback as the pipeline would be ended early, without running most of other tests that may not be immediately related to the MR.

How it works?

An rspec fail-fast job is added in parallel to all other rspec jobs in a Merge Request pipeline. This job would run the test files that are directly related to the changes in the Merge Request. If any of these tests fail, the rspec fail-fast job would fail, triggering a fail-pipeline-early job to run. The fail-pipeline-early job would cancel the currently running pipeline and all in-progress jobs, and would the pipeline would have status failed.

graph LR
    subgraph "prepare stage";
        A["detect-tests"]
    end

    subgraph "test stage";
        B["jest"];
        C["rspec migration"];
        D["rspec unit"];
        E["rspec integration"];
        F["rspec system"];
        G["rspec fail-fast"];
    end

    subgraph "post-test stage";
        Z["fail-pipeline-early"];
    end
    
    A --"artifact: list of test files"--> G
    G --"on failure"--> Z

Will my Merge Request be affected, how?

If your MR:

  • changes only documentation files:
    • No, your MR pipeline will not be affected
  • changes only QA code:
    • No, your MR pipeline will not be affected
  • changes only frontend code:
    • No, your MR pipeline will not be affected
  • changes only backend code:
    • Yes, your MR pipeline would be affected.
    • You'll see rspec fail-fast job in your pipeline. If this job fails, it means that your changes have broken some existing tests that is in close relation to your changes. As usual, the job log would contain the list of failing tests. The pipeline will then be ended early with a "failed" status.
  • changes both frontend and backend code:
    • Yes, your MR pipeline would be affected.
    • You'll see rspec fail-fast job in your pipeline. If this job fails, it means that your changes have broken some existing tests that is in close relation to your changes. As usual, the job log would contain the list of failing tests. The pipeline will then be ended early with a "failed" status. Due to how the pipeline is structured, the jest job may also be cancelled if rspec fail-fast has failed.
    • If you would like to run jest without being affected by rspec fail-fast, you may opt-out from fail fast using one of the methods below.

How to opt-out of this?

A Merge Request author may choose to opt-out of the fail fast mechanism by doing one of the following:

  1. Including [SKIP RSPEC FAIL-FAST] in the Merge Request title.
  2. Starting the dont-interrupt-me job found in the sync stage of a Merge Request pipeline.

When will this happen?

We are expecting to enable this experiment in the gitlab-org/gitlab project sometime on Tuesday, 15 September 2020.

Anything else I need to know?

The rspec fail-fast job would no-op if there are more than 10 test files related to the changes in the Merge Request. This is to prevent rspec fail-fast duration from exceeding the average rspec job duration and defeating its purpose.

I have some feedback

We would like to hear any feedback related to productivity changes resulting from this experiment. If you have benefited from the shorter pipelines, please let us know below. If you are negatively affected by the change, we would also like to hear about it.

For more detail:

Edited by Albert Salim