Feedback issue for resolvable comments posted on MR on end-to-end test failures

Background

There was a discussion on Quality retro for 16.0 milestone about bugs leaking into production even though end-to-end tests were run on the MR and had failures. The reason for this is that e2e test are allowed to fail and are run in a child pipeline and a change was made recently to allow blocking merges if trigger-omnibus job fails. The e2e test jobs inside child pipeline are allowed to fail but the trigger job itself isn't (Related Slack thread). This caused any failures in the end-to-end tests to go un-noticed:

Before:

Before

After:

After

As a result, a bug leaked into production.

As a stop gap solution until we have a way to block merges on e2e test failures, we decided to post resolvable comments under Allure test report (MR).

This issue is to gather feedback on this change.

Edited by Sanad Liaquat