Skip to content

GitLab Next

Merged
Created by Rémy Coutable@rymaiMaintainer1 of 1 task completed1/1 task

Detect and keep track of flaky specs

EE MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2626

What does this MR do?

This detects flaky specs by checking if a spec that passes had passed after more than 1 attempt.

This doesn't take in account a spec that fails in a first job, and the pass in a job retry, as this would need a more complex logic.

Each RSpec job generates a report (similarly to what Knapsack does) and then all the reports are merged into a single one and uploaded to an S3 bucket (similarly to what Knapsack does). New flaky specs (if any) are also printed at the end of each job.

I've build upon what we already had:

Following is an example of the report generated:

{
  "d1371e00c3b1abe1a4c1e5cc64c076d0": {
    "example_id": "./spec/models/repository_spec.rb[1:77:1:1]",
    "file": "./spec/models/repository_spec.rb",
    "line": 1806,
    "file_mtime_when_first_flaky": "2017-07-24 08:18:47 +0000",
    "description": "Repository#commit_count_for_ref with a non-existing repository returns 0",
    "first_flaky_at": "2017-07-24 12:12:31 +0000",
    "last_flaky_at": "2017-07-24 12:12:31 +0000",
    "last_attempts_count": 2,
    "flaky_reports": 1,
    "last_flaky_job": "https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/42"
  },
  "ad6ab3504d10ca04aebd8a51970a7879": {
    "example_id": "./spec/features/boards/modal_filter_spec.rb[1:4:1]",
    "file": "./spec/features/boards/modal_filter_spec.rb",
    "line": 84,
    "file_mtime_when_first_flaky": "2017-07-24 08:18:47 +0000",
    "description": "Issue Boards add issue modal filtering author filters by selected user",
    "first_flaky_at": "2017-07-24 12:13:56 +0000",
    "last_flaky_at": "2017-07-24 12:13:56 +0000",
    "last_attempts_count": 2,
    "flaky_reports": 1,
    "last_flaky_job": "https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/42"
  },
  "5c9f7f8936916cd975561649432cd18d": {
    "example_id": "./spec/features/variables_spec.rb[1:6]",
    "file": "./spec/features/variables_spec.rb",
    "line": 83,
    "file_mtime_when_first_flaky": "2017-07-24 08:18:47 +0000",
    "description": "Project variables deletes variable",
    "first_flaky_at": "2017-07-24 12:16:12 +0000",
    "last_flaky_at": "2017-07-24 12:16:12 +0000",
    "last_attempts_count": 2,
    "flaky_reports": 1,
    "last_flaky_job": "https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/42"
  }
}

Are there points in the code the reviewer needs to double check?

For now, we only keep track of the flaky specs, we don't:

  • remove not longer flaky specs (probably based on last occurence)
  • remove flaky specs that have been modified (based on the file mtime and/or example description/ID)
  • fail a pipeline if a commit introduces a new flaky spec: for now the job is allowed to fail as we'll need to build a comprehensive list of flaky specs before not allowing this job to fail

These can be next steps (to be done in other MRs).

Why was this MR needed?

Because flaky specs are a productivity killer so we should detect, retry, and ignore them when appropriate.

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

Related to #32308 (moved).

Edited by Rémy Coutable