E2E: Show Job Stats in Slack and MR Report Summaries for Timed Out Jobs
Problem
Currently, when any E2E test job times out (such as e2e:saas
, e2e:self-managed
and e2e:saas-reliable
), no test result file is generated in the artifacts, and the summary in Slack and merged MRs will not show any information about the job at all.
For example, we can see in this Slack summary for this pipeline that only e2e:saas-reliable
and e2e:self-managed
displays. e2e:saas
had timed out:
This can be confusing and hide actual failures that need to be investigated.
Proposal
If a test result file is not generated, such as when a job times out, we should still show a summary for that job, ex:
JOB | TOTAL | FAILURES | ERRORS | SKIPPED | RESULT
------------------|-------|----------|--------|---------|--------
e2e:saas | N/A | N/A | N/A | N/A | No data
e2e:saas-reliable | 4 | 0 | 0 | 0 | Passed
e2e:self-managed | 31 | 23 | 0 | 4 | Failed
Edited by Valerie Burton