Skip to content
Snippets Groups Projects
Commit 4d3a8d69 authored by Sanad Liaquat's avatar Sanad Liaquat :palm_tree: Committed by Kassandra Svoboda
Browse files

Remove text related to the blocking e2e tests

parent d3c81eb8
No related branches found
No related tags found
1 merge request!10673Remove text related to the blocking e2e tests
......@@ -27,8 +27,8 @@ The testing strategy for [this level of the pyramid](https://docs.gitlab.com/ee/
**Merge Request** - The whole QA End-to-End test suite can be run on your MR by triggering manually the `package-and-qa` job.
**Scheduled Pipelines** - [Schedule](/handbook/engineering/infrastructure/test-platform/debugging-qa-test-failures/#qa-test-pipelines).
Package tests run in various pipelines and we have a few tests tagged as `:blocking`. `:blocking` tests block merge requests on failure and `:smoke` tests block deployments from going further in case of failure.
Other Package related tests that are not tagged as `:blocking` run when the full suite of tests runs.
Package tests run in various pipelines and we have a few tests tagged as `:smoke`. `:smoke` tests block deployments from going further in case of failure.
Other Package related tests that are not tagged as `:smoke` run when the full suite of tests runs.
### Where are they
......
......@@ -55,7 +55,7 @@ A majority of the testing we will need to do will exist at the `Single Cell` lev
#### Feature
This testing is done as part of the day to day work of development, the unit/integration tests added as part of developing the features. The SET can help advise on edge cases / scenarios that should be considered for testing. We currently have two E2E test suites defined: `smoke` and `blocking` ([the test suite definitions](../test-platform/blocking-tests.md#overview)).
This testing is done as part of the day to day work of development, the unit/integration tests added as part of developing the features. The SET can help advise on edge cases / scenarios that should be considered for testing. We currently have one E2E test suite defined: `smoke`.
#### Single Cell
......
......@@ -66,10 +66,6 @@ You can leave any feedback about this process in the [dedicated issue](https://g
- Define acceptable thresholds for action like quarantining/focus on refactoring
- Step towards unlocking [Merge train](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/195)
### Flaky tests management in end-to-end tests
The end-to-end tests follow a separate process of automatically quarantining flaky tests as [outlined here](/handbook/engineering/infrastructure/test-platform/blocking-tests/#managing-test-failures).
## Additional resources
- [Flaky tests technical documentation](https://docs.gitlab.com/ee/development/testing_guide/flaky_tests.html)
......
......@@ -320,12 +320,6 @@ Please note that the borrow request might not guarantee 100% allocation to the r
The list of all SET borrow requests can be seen [here](https://gitlab.com/gitlab-com/Product/-/issues/?label_name%5B%5D=SET%20Borrow).
### Blocking tests
Blocking tests have met stricter reliability criteria than other tests in our test suite. When a failure is seen in a blocking test, it's less likely to be flakiness and more likely to be a true issue.
For more information, please visit our [blocking tests page](blocking-tests).
### Risk mapping
The Test Platform Sub-Department helps facilitate the risk mapping process. This requires the participation of Product Management, Development, UX, and the Quality team to develop a strategic approach to risk and mitigation planning.
......
---
title: "Blocking end-to-end tests"
description: "This page describes the process and technical documentation around blocking end-to-end tests at GitLab.
Blocking tests are executed as a showstopper step in the release pipelines. Blocking tests are executed as a blocking step in MRs but not the deployments."
---
## Overview
GitLab's end-to-end tests for API and UI, located in `qa/qa/specs/features/`, can be promoted to:
- **Smoke Tests**: Essential for the release process, they are executed and can stop deployments from going further.
- **Blocking Tests**: Key to maintaining code quality in MRs, blocking on failure and mandatory passing for MR approval.
In the longer term, the `:blocking` bucket will be sunset. All the tests will be run as a blocking step in the MRs and
only smoke tests will be run during the release process.
### Defining a Blocking Test
- A `:blocking` test consistently succeeds in the master or nightly pipeline for at least 14 days.
- A blocking test is for MR quality control and not included in deployment pipelines.
## Promotion Processes
### To Blocking Suite
Tests are selected for promotion by a weekly automated script that uses the data produced by reliable test report. The automated script
is run once a week by a schedule named "Weekly reliable, unreliable E2E spec report" in the [quality/toolbox project](https://gitlab.com/gitlab-org/quality/toolbox/-/pipeline_schedules).
- Criteria: 14 consecutive days of success and top 10 in run frequency in master or nightly pipelines
- The process involves generating MRs for the top-performing tests and assigning them for review by counterpart SET for
the DevOps stage of the test as a DRI.
- A test should ideally not be promoted manually without it being identified in the reliable test report. However, if a
test has been identified in the reliable test report did not make it to the top 10 number of runs, it can be promoted
by manually creating an MR.
- Orchestrated tests are also selected for promotion to blocking even though they do not currently run in the blocking GDK jobs (`gdk-qa-blocking`).
This will ensure we have a set of stable orchestrated tests when we make the [orchestrated tests block MRs](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2516)
The flow of promotion to blocking as a decision tree:
```mermaid
flowchart TD
%% nodes
reliable_test_report[Reliable test report\nruns once a week]
end_to_end_test[End to end test]
passed_consistently_for_14_days_check{Passed\nconsistently\nfor 14 days}
master_and_nightly_piplines_check{Ran in master or\nnightly pipline}
surfaces_in_reliable_test_report[Surface in reliable test report]
top_10_run_frequency_check{In top 10 list\nby no of runs}
create_merge_request_for_promotion[Merge request created for promotion]
no_action_performed[No action performed]
%% diagram
reliable_test_report --> |Evaluates| end_to_end_test
end_to_end_test --> master_and_nightly_piplines_check
master_and_nightly_piplines_check --> |Yes |passed_consistently_for_14_days_check
master_and_nightly_piplines_check --> |No |no_action_performed
passed_consistently_for_14_days_check --> |Yes| surfaces_in_reliable_test_report
passed_consistently_for_14_days_check --> |No |no_action_performed
surfaces_in_reliable_test_report --> top_10_run_frequency_check
top_10_run_frequency_check --> |Yes| create_merge_request_for_promotion
top_10_run_frequency_check --> |No |no_action_performed
```
### Weekly Reliable Spec Report
This report plays a crucial role in managing the health of the test suite, highlighting:
- Specs passing consistently for 14 days in different pipelines. These tests are promoted to blocking.
- Blocking specs that failed in last 14 days. These tests are quarantined.
## Managing Test Failures
The flow of test failures as a decision tree:
```mermaid
flowchart TD
%% nodes
test_fails[An E2E test fails]
failure_in_mr_check{Failure is\nin MR?}
author_fixes[MR author fixes the code\nor updates the test]
failure_issue_created[Failure issue created]
test_session_report[Failure issue surfaces on\ntest session report and Slack]
manual_triage_flow[Manual Triage Flow]
blocking_check{Failed test\nis blocking?}
nightly_or_master_check{Failure is\non master\nor nightly\npipeline?}
surface_on_relaible_report[Failure issue surfaces\non reliable test report]
one_percent_failure_check{Test failed\nin more than\n1% of runs?}
failure_issue_open_check{Failure\nissue\nis open?}
test_environment_related_check{Failure is\nenvoronment\nrelated?}
create_mr_for_quarantine[MR auto created for quarantine]
no_action[No action needed]
%% external links
click manual_triage_flow "https://handbook.gitlab.com/handbook/engineering/infrastructure/test-platform/debugging-qa-test-failures/#triage-flow"
%% diagram
test_fails --> failure_in_mr_check
failure_in_mr_check -->|Yes| author_fixes
failure_in_mr_check -->|NO| failure_issue_created
failure_issue_created --> test_session_report
test_session_report --> manual_triage_flow
failure_issue_created --> nightly_or_master_check
nightly_or_master_check -->|Yes| blocking_check
nightly_or_master_check -->|No| manual_triage_flow
blocking_check -->|Yes| surface_on_relaible_report
blocking_check -->|No| manual_triage_flow
surface_on_relaible_report --> one_percent_failure_check
one_percent_failure_check -->|Yes| failure_issue_open_check
one_percent_failure_check -->|No| no_action
failure_issue_open_check -->|Yes| test_environment_related_check
failure_issue_open_check -->|No| no_action
test_environment_related_check -->|No| create_mr_for_quarantine
test_environment_related_check -->|Yes| manual_triage_flow
```
### What happens when a blocking test fails?
Blocking test failures require immediate attention:
- **In MRs**: The author is responsible for addressing any bugs or updating the test.
- **In master or nightly pipeline**: Consistent failures trigger automatic quarantine.
Once a test is quarantined, the counterpart SET will be mentioned in a comment on the failure issue which will already be assigned to them.
It is the responsibility of the counterpart SET to either delegate the issue or fix and de-quarantine the test themselves.
If a test needs to be quarantined sooner than the next reliable test report run,
the [fast quarantine](/handbook/engineering/infrastructure/test-platform/debugging-qa-test-failures/#fast-quarantine)
process must be followed.
The `:blocking` tag cannot be removed from a test. Such a test can only be quarantined and de-quarantined.
## Execution Command
- **Blocking Tests**: `bin/qa Test::Instance::All http://localhost:3000 -- --tag blocking`
## Execution Schedule and Environments
- Executed in MRs and master in `gdk-qa-blocking` jobs for ongoing quality assurance.
- Also executed in `gdk-qa-blocking-ff-inverse` jobs in MRs that include feature flag changes to validate the tests in both feature flag states.
## Future Iterations
- Make reliable orchestrated tests block
- Automate the process of de-quarantining the tests that have been consistently
passing. ([Issue link](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/1918#phase-3-automate-de-quarantining-update-process-and-docs-okr))
MR. ([Issue link](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2516))
- Once most of the tests have been promoted to the blocking, sunset the `:blocking` tag and make all tests block MRs. At
this stage, any left over tests that weren't promoted should be
quarantined. ([Issue link](https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2498))
......@@ -61,7 +61,7 @@ If you have any questions on the status, you can also reach out to the `@release
GitLab `master` has three QA pipelines generated from scheduled pipeline against the default branch:
- [`test-on-omnibus`](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/#using-the-test-on-omnibus-job) runs the `full` suite of end-to-end tests against an omnibus Docker image built from `master`
- [`test-on-gdk`](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/#using-the-test-on-gdk-job) runs both `blocking` and `smoke` suites of end-to-end tests as part of the `blocking` job against a GDK instance from a Docker image built from `master`
- [`test-on-gdk`](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/#using-the-test-on-gdk-job) runs the full suite of end-to-end tests as part of the `gdk-instance` job against a GDK instance from a Docker image built from `master`
If jobs in `test-on-omnibus` failed due to a GitLab Docker image issue, reach out to the [Distribution team](/handbook/engineering/infrastructure/core-platform/systems/distribution/) to see if it's a known problem with the build.
......
......@@ -22,7 +22,7 @@ The test pipelines run on a scheduled basis, and their results are posted to Sla
| GitLab `master` `e2e:test-on-omnibus-ee` | [Pipelines](https://gitlab.com/gitlab-org/gitlab/pipelines) \| [Definition](https://gitlab.com/gitlab-org/gitlab/-/blob/77a5b0b3f8f910d4fe89a3fed46c4da53b1e587e/.gitlab/ci/qa.gitlab-ci.yml#L77-88) | Full | [scheduled pipeline every 2 hours](https://gitlab.com/gitlab-org/gitlab/pipeline_schedules) | [`#e2e-run-master`](https://gitlab.slack.com/archives/CNV2N29DM) | [Master EE](https://gitlab-qa-allure-reports.s3.amazonaws.com/e2e-test-on-omnibus/master/index.html) |
| GitLab `master` `e2e:test-on-omnibus-ce` | [Pipelines](https://gitlab.com/gitlab-org/gitlab/-/pipeline_schedules) \| [Definition](https://gitlab.com/gitlab-org/gitlab/-/blob/77a5b0b3f8f910d4fe89a3fed46c4da53b1e587e/.gitlab/ci/qa.gitlab-ci.yml#L121-135) | Full | [Daily at 4:00am UTC](https://gitlab.com/gitlab-org/gitlab/-/pipeline_schedules) | [`#e2e-run-master`](https://gitlab.slack.com/archives/CNV2N29DM) | [Master CE](https://gitlab-qa-allure-reports.s3.amazonaws.com/e2e-test-on-omnibus-ce/master/index.html) |
| GitLab `master` `e2e:test-on-gdk` | [Pipelines](https://gitlab.com/gitlab-org/gitlab/-/pipelines?ref=master&source=schedule) \| [Definition](https://gitlab.com/gitlab-org/gitlab/-/blob/77a5b0b3f8f910d4fe89a3fed46c4da53b1e587e/.gitlab/ci/qa.gitlab-ci.yml#L151-164) | Full | [scheduled pipeline every 2 hours](https://gitlab.com/gitlab-org/gitlab/pipeline_schedules) | [`#e2e-run-master`](https://gitlab.slack.com/archives/CNV2N29DM) | [Master GDK](https://gitlab-qa-allure-reports.s3.amazonaws.com/e2e-test-on-gdk/master/index.html) |
| GitLab `master` `e2e:test-on-cng` | [Pipelines](https://gitlab.com/gitlab-org/gitlab/-/pipeline_schedules) \| [Definition](https://gitlab.com/gitlab-org/gitlab/-/blob/77a5b0b3f8f910d4fe89a3fed46c4da53b1e587e/.gitlab/ci/qa.gitlab-ci.yml#L166-178) | Smoke, Blocking | [scheduled pipeline every 2 hours](https://gitlab.com/gitlab-org/gitlab/pipeline_schedules) | [`#e2e-run-master`](https://gitlab.slack.com/archives/CNV2N29DM) | [Master CNG](https://gitlab-qa-allure-reports.s3.amazonaws.com/e2e-test-on-cng/master/index.html) |
| GitLab `master` `e2e:test-on-cng` | [Pipelines](https://gitlab.com/gitlab-org/gitlab/-/pipeline_schedules) \| [Definition](https://gitlab.com/gitlab-org/gitlab/-/blob/77a5b0b3f8f910d4fe89a3fed46c4da53b1e587e/.gitlab/ci/qa.gitlab-ci.yml#L166-178) | Smoke | [scheduled pipeline every 2 hours](https://gitlab.com/gitlab-org/gitlab/pipeline_schedules) | [`#e2e-run-master`](https://gitlab.slack.com/archives/CNV2N29DM) | [Master CNG](https://gitlab-qa-allure-reports.s3.amazonaws.com/e2e-test-on-cng/master/index.html) |
| GitLab `master` Nightly | [Pipelines](https://gitlab.com/gitlab-org/gitlab/-/pipeline_schedules) \| [Definition](https://gitlab.com/gitlab-org/gitlab/-/blob/77a5b0b3f8f910d4fe89a3fed46c4da53b1e587e/.gitlab/ci/qa.gitlab-ci.yml#L137-149) | Full | [Daily at 4:00am UTC](https://gitlab.com/gitlab-org/gitlab/-/pipeline_schedules) | [`#e2e-run-master`](https://gitlab.slack.com/archives/CNV2N29DM) | [Master Nightly](https://gitlab-qa-allure-reports.s3.amazonaws.com/nightly/master/index.html) |
NOTE:
......
......@@ -388,7 +388,7 @@ If a test is failing and you have good reason to quarantine it quickly you can [
You should use fast quarantine to unblock deployment pipelines and MRs if the failure is disruptive and you've ruled out bugs as the cause of the failure (e.g., you've identified that the test is stale, or flaky).
> **Note**: Failing `blocking` job in the `e2e: test-on-gdk` child pipeline will block MR pipelines as well as scheduled master pipelines. Failures of those tests are good candidates for fast quarantine.
> **Note**: Failing `gdk-instance` job in the `e2e: test-on-gdk` child pipeline will block MR pipelines as well as scheduled master pipelines. Failures of those tests are good candidates for fast quarantine.
After fast quarantining, please follow the long-term quarantine process below.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment