E2E: Show failure exceptions with job urls in reliable report
What does this MR do and why?
As part of gitlab-org/quality/quality-engineering/team-tasks#1918 (closed), we would like to see failure exceptions in reliable tests. This will help identify common failures related to environmental or non-test/app related factors. Once identified, the failures could be ignored when determining test's reliability.
This MR also skips the following jobs from the reliability report as their execution has unconventional influence that should not affect their ability to run reliably on staging:
airgapped
instance-image-slow-network
nplus1-instance-image
Screenshots or screen recordings
How to set up and validate locally
Change the PROJECT_ID
to an id of a project on your local.
From the qa/
directory, run:
GITLAB_ACCESS_TOKEN=<YOUR_PAT> CI_API_V4_URL=http://gdk.test:3000/api/v4/ QA_INFLUXDB_URL=https://influxdb.quality.gitlab.net QA_INFLUXDB_TOKEN=<QA_INFLUXDB_TOKEN_FROM_1P> bundle exec rake "reliable_spec_report[14, true]"
The above command is intentionally missing SLACK_WEBHOOK
variable as we do not want to post to Slack for testing. This will cause an error but the issue will be created.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.