Skip to content

Makes sanity job check match on expected output instead of string comparison

Sean Gregory requested to merge fix-sanity-check into master

What does this MR do and why?

Currently

When the sanity-framework job runs, GitLab QA expects it to output the number of examples that ran/failed

This causes issues with adding new tests to sanity-framework as the expectation is hard coded

This MR

Changes the expectation to only match the number of failures.

This change should allow the sanity check to expect that a test failed, but doesn't have awareness of how many are passing.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sean Gregory

Merge request reports