Add execution time and error messages to testResult notifications
Current situation
The parsing services produce notifications such as:
- apiVersion: opentestfactory.org/v1alpha1
kind: Notification
metadata:
attachment_origin:
- 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
name: run11
workflow_id: 8c86fd45-198c-4ef2-ac5a-1dacb6ee1ad9
spec:
testResults:
- attachment_origin: 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
id: e2b21d53-dddb-467d-a1c6-a40f168c9d4c
name: Stock Management To Check Duplicate Execution#Current stock duplicate
status: SUCCESS
- attachment_origin: 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
id: 53f9e52c-cc31-414a-9559-b3699acffaca
name: Stock Management To Check Duplicate Execution#Error in stock duplicate
status: FAILURE
Those notifications do not include rich execution status.
Desired outcome
For each test result, if available, the execution duration and the possible error message should be included.
-
duration
: the test duration, in milliseconds -
failureType
: the failure type (if any) -
failureMessage
: the execution message, as included in the execution report -
failureText
: the body of the<failure>
(or equivalent) tag
If the information is not available or irrelevant (say, for a successful test execution, there is no failure message), the field should not be present.
Assuming the following execution report:
<testcase id="5a614139-fda6-4265-9d8e-c81825c22ebd" name="multFailure#CalculatorMult multFailure" classname="gitlab-demo-job" timestamp="2023-11-23T09:51:19.264264" time="0.123">
<failure message="expected 8 to equal 6" type="AssertionError">AssertionError: expected 8 to equal 6 at Context.eval (webpack:///./cypress/integration/calculator.mult.ko.spec.js:5:21) + expected - actual -8 +6 </failure>
</testcase>
- apiVersion: opentestfactory.org/v1alpha1
kind: Notification
metadata:
attachment_origin:
- 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
name: run11
workflow_id: 8c86fd45-198c-4ef2-ac5a-1dacb6ee1ad9
spec:
testResults:
- attachment_origin: 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
id: e2b21d53-dddb-467d-a1c6-a40f168c9d4c
name: Stock Management To Check Duplicate Execution#Current stock duplicate
status: SUCCESS
duration: 43123
- attachment_origin: 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
id: 5a614139-fda6-4265-9d8e-c81825c22ebd
name: multFailure#CalculatorMult multFailure
status: FAILURE
duration: 123
failureMessage: expected 8 to equal 6
failureType: AssertionError
failureText: |
AssertionError: expected 8 to equal 6 at Context.eval (webpack:///./cypress/integration/calculator.mult.ko.spec.js:5:21) + expected - actual -8 +6
For the generic parsing service, there should be a way to configure where this information is extracted from.
Edited by Martin Lafaix