Skip to content

Add execution time and error messages to testResult notifications

Current situation

The parsing services produce notifications such as:

- apiVersion: opentestfactory.org/v1alpha1
  kind: Notification
  metadata:
    attachment_origin:
    - 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
    name: run11
    workflow_id: 8c86fd45-198c-4ef2-ac5a-1dacb6ee1ad9
  spec:
    testResults:
    - attachment_origin: 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
      id: e2b21d53-dddb-467d-a1c6-a40f168c9d4c
      name: Stock Management To Check Duplicate Execution#Current stock duplicate
      status: SUCCESS
    - attachment_origin: 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
      id: 53f9e52c-cc31-414a-9559-b3699acffaca
      name: Stock Management To Check Duplicate Execution#Error in stock duplicate
      status: FAILURE

Those notifications do not include rich execution status.

Desired outcome

For each test result, if available, the execution duration and the possible error message should be included.

  • duration: the test duration, in milliseconds
  • failureType: the failure type (if any)
  • failureMessage: the execution message, as included in the execution report
  • failureText: the body of the <failure> (or equivalent) tag

If the information is not available or irrelevant (say, for a successful test execution, there is no failure message), the field should not be present.

Assuming the following execution report:

<testcase id="5a614139-fda6-4265-9d8e-c81825c22ebd" name="multFailure#CalculatorMult multFailure" classname="gitlab-demo-job" timestamp="2023-11-23T09:51:19.264264" time="0.123">
<failure message="expected 8 to equal 6" type="AssertionError">AssertionError: expected 8 to equal 6 at Context.eval (webpack:///./cypress/integration/calculator.mult.ko.spec.js:5:21) + expected - actual -8 +6 </failure>
</testcase>
- apiVersion: opentestfactory.org/v1alpha1
  kind: Notification
  metadata:
    attachment_origin:
    - 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
    name: run11
    workflow_id: 8c86fd45-198c-4ef2-ac5a-1dacb6ee1ad9
  spec:
    testResults:
    - attachment_origin: 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
      id: e2b21d53-dddb-467d-a1c6-a40f168c9d4c
      name: Stock Management To Check Duplicate Execution#Current stock duplicate
      status: SUCCESS
      duration: 43123
    - attachment_origin: 49c8bc1e-bb99-48f3-be24-4fc34fcf1dd5
      id: 5a614139-fda6-4265-9d8e-c81825c22ebd
      name: multFailure#CalculatorMult multFailure
      status: FAILURE
      duration: 123
      failureMessage: expected 8 to equal 6
      failureType: AssertionError
      failureText: |
        AssertionError: expected 8 to equal 6 at Context.eval (webpack:///./cypress/integration/calculator.mult.ko.spec.js:5:21) + expected - actual -8 +6 

For the generic parsing service, there should be a way to configure where this information is extracted from.

Edited by Martin Lafaix