Skip to content

Support fast-quarantine for E2E tests

Rémy Coutable requested to merge support-fast-quarantining-e2e-tests-2 into master

What does this MR do and why?

This adds support for fast-quarantining E2E examples. This depends on gitlab-qa!1168 (merged).

Related to gitlab-org/quality/quality-engineering/team-tasks#1777 (closed).

The example path should be given relatively to the top-level qa/ folder from where the E2E tests are run, e.g.

qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb:14

Screenshots or screen recordings

https://gitlab.com/gitlab-org/gitlab/-/jobs/4379911523#L2162:

Create
  Branch Rules Overview
    adds a new branch rule (PENDING: Skipping ./qa/specs/features/browser_ui/3_create/repository/add_new_branch_rule_spec.rb[1:1:1] because it's been fast-quarantined.)

How to set up and validate locally

  1. Create a rspec/fast_quarantine-gitlab.txt file with the example identifiers that you'd like to quarantine, e.g. qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb:14
  2. Run the test that's supposed to be quarantine, e.g. cd qa && bundle exec rspec qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb:14
  3. You should see an output saying that examples are skipped, e.g.
    sets the default branch name for a new project (PENDING: Skipping ./qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb[1:1:1] because it's been fast-quarantined.)

[...]

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Create Default branch name instance setting sets the default branch name for a new project
     # Skipping ./qa/specs/features/api/3_create/repository/default_branch_name_setting_spec.rb[1:1:1] because it's been fast-quarantined.
     Failure/Error: raise SkipDeclaredInExample.new(message)
       RSpec::Core::Pending::SkipDeclaredInExample
     # /Users/remy/Code/GitLab/gdk/gitlab/spec/support/fast_quarantine.rb:20:in `block (2 levels) in <top (required)>'

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rémy Coutable

Merge request reports