Report setup failures in Allure
Problem
When QA jobs fail in setup stage before tests start like this Failure in update_major test (gitlab-org/gitlab#408304 - closed), they are not reported in Allure.
Details
gitlab-org&9201 (comment 1376102356)
it's completely normal that a failure to actually install omnibus did not appear in allure report given it was a failure in setup, so there was no test process. We would have same result with any other test reporter generated by rspec test run.
That being said, https://github.com/allure-framework/allure-ruby does expose a generic "lifecycle" api, so we could plug it in into
gitlab-qa
omnibus installation process and create a "fake" test case with a failure that would show up in allure report in case a migration fails etc.
Decision
After further review it was decided to not implement this feature in favour of Update the pipeline definition to always run a subset of the e2e pipeline that includes the necessary build steps and also the update-major and update-minor tests
This will allow to resolve the root cause. Updating Allure to report building failures is counteractive to its purpose of being test reporting tool as well as such comment or report is not blocking and can be easily missed.
See more details in #1779 (comment 1413796484)