Failure Reporting and Reproducibility - Recording videos on failures in test framework
More details here: https://docs.google.com/document/d/1kgrr2Qb3rOsgsJGOFU8qSt17m9z4B42pCR7mZ5yN8eI/edit#bookmark=id.479zno1azczd
Q1:
-
Stage 1: Add functionality to record video with Selenoid. - gitlab-org/gitlab-qa!1153 (merged)
- gitlab-org/gitlab!117182 (merged)
- gitlab-org/gitlab-qa!1166 (merged)
- Example pipeline: https://gitlab.com/gitlab-org/quality/nightly/-/jobs/4158409800 (video will be removed after 4 weeks)
Q2:
-
Stage 2: Edit video container to not start automatically. -
Stage 3: Add triggers before and after each test and only save failed videos. -
Stage 4: Implement video recording on pipelines - First iteration on just two jobs: gitlab-org/gitlab!125041 (merged)
Future iterations:
- Fix timeout issue: gitlab-org/gitlab-qa!1201 (merged)
- Skip video recording for api tests: gitlab-org/gitlab!124342 (merged)
- Add option to save all videos, not just failed: gitlab-org/gitlab!124463 (merged)
- Add failure video to allure report: gitlab-org/gitlab!124349 (merged)
- Close out browser and video recorder containers after suite to be able to run in update-major update-minor jobs.
- Update the handful of failing tests to work with Selenoid: #1865 (closed)
- Implement recording on all package-and-test jobs
- Create our own fork of the custom recorder image
Notes:
The way Selenoid works out of the box is it records the browser container for the duration of it's use. It has no knowledge of the tests running in the browser. The problem with this is the QA framework reuses the browser session for multiple tests so the video would consist of multiple tests. Originally we were going to change to using a separate browser session for each test but found that would add too much time to the overall test runs.
The current solution is to use a fork of the recorder image that doesn't automatically start recording when the browser container is started. This is combined with an API to interact with the browser container and the recorder container to start and stop recording when tests start and stop. This will also only save failed test runs.