Skip to content

Draft: POC - Find and Skip flaky tests

David Dieulivol requested to merge ddieulivol-create_flaky_tests_issues_for_mrs into master

Context

Related to gitlab-org/quality/engineering-productivity/team#297 (closed).

The goal is to identify flaky tests we see in MRs, and skip them for other MRs.

Solution overview

Definitions

  • Flaky test: If a spec was retried in a new RSpec process, and succeeded, we consider it a flaky test.
  • Flaky test from the MR or outside the MR: If the file containing the flaky test was introduced/modified in the MR, we consider that the MR introduced the flaky test. Otherwise, the flaky test happened outside of the MR.

Algorithm

Part 1 - Find and report flaky tests

  1. In each Rspec job, we check whether specs were retried and successful after the retry. All those specs are considered flaky.
  2. For each flaky spec, check whether they were part of the MR. If so, either fail the job or issue a warning in the MR.
  3. If the flaky spec is not part of the MR, find/create the flaky test issue in a new dedicated GitLab project.

Part 2 - Skip flaky tests

  1. In scheduled pipelines, we create the list of flaky tests from the dedicated GitLab flaky tests project
  2. We publish this list of flaky tests to GitLab pages
  3. Each RSpec job downloads this flaky tests list, and skips the tests present in it

What does this MR do and why?

This MR is a "non-functional proof-of-concept" 😆. With it, I'd like to validate the approach I'd like to follow as very-early feedback mechanism.

I made two commits, each of them is a part of the POC.

Part 1 - Find and report flaky tests

commit

We find and create GitLab issues for flaky tests we discovered in MRs.

Part 2 - Skip flaky tests in MRs

commit

In scheduled pipelines, we construct a list of flaky tests to skip from the open issues in the GItLab project we use to report flaky tests.

We then use this list to skip the tests we want to skip.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Dieulivol

Merge request reports