Skip to content

E2E: Show failure exceptions with job urls in reliable report

Sanad Liaquat requested to merge qa-shl-show-exceptions-in-report into master

What does this MR do and why?

As part of gitlab-org/quality/quality-engineering/team-tasks#1918 (closed), we would like to see failure exceptions in reliable tests. This will help identify common failures related to environmental or non-test/app related factors. Once identified, the failures could be ignored when determining test's reliability.

This MR also skips the following jobs from the reliability report as their execution has unconventional influence that should not affect their ability to run reliably on staging:

  • airgapped
  • instance-image-slow-network
  • nplus1-instance-image

Screenshots or screen recordings

Screenshot_2023-08-28_at_5.27.15_AM

How to set up and validate locally

Change the PROJECT_ID to an id of a project on your local.

From the qa/ directory, run:

GITLAB_ACCESS_TOKEN=<YOUR_PAT> CI_API_V4_URL=http://gdk.test:3000/api/v4/ QA_INFLUXDB_URL=https://influxdb.quality.gitlab.net QA_INFLUXDB_TOKEN=<QA_INFLUXDB_TOKEN_FROM_1P> bundle exec rake "reliable_spec_report[14, true]"

The above command is intentionally missing SLACK_WEBHOOK variable as we do not want to post to Slack for testing. This will cause an error but the issue will be created.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sanad Liaquat

Merge request reports