Preserve test run artifacts when running gitlab-qa multiple times
Noticed this when trying to run gitlab-qa
multiple times back to back.
The artifacts, screenshot and html of the previous run is over-written on the 1st failure of the next run. This makes it a challenge to compare results from multiple test runs.
Proposing that we make the following improvements to the results artifacts
- Generate a top level folder for each run, time stamped so it is unique.
- Create a sanitized folder name from the test name to differentiate the test in human readable form.
The current folder structure looks like this, it just puts all the results in one place
/private/tmp/gitlab-qa/screenshots ls
screenshot_2018-05-25-21-41-55.916.html
screenshot_2018-05-25-21-42-43.769.html
screenshot_2018-05-25-21-53-39.724.html
screenshot_2018-05-25-21-57-47.683.html
screenshot_2018-05-25-21-41-55.916.png
screenshot_2018-05-25-21-42-43.769.png
screenshot_2018-05-25-21-53-39.724.png
screenshot_2018-05-25-21-57-47.683.png
Proposed structure
- gitlab-qa\results\
- test-run-2018-05-25-201122\
- cicd-pipelines-users-creates-a-new-pipeline\
- screenshot_2018-05-25-21-41-55.916.html
- screenshot_2018-05-25-21-41-55.916.png
- testlogs.txt (REST client & Selenium Capybara logs)
- creates-a-merge-request-user-creates-a-new-merge-request\
- screenshot_2018-05-25-21-57-47.683.html
- screenshot_2018-05-25-21-57-47.683.png
- testlogs.txt (REST client & Selenium Capybara logs)
- test-run-2018-05-27-102530\
- cicd-pipelines-users-creates-a-new-pipeline\
- screenshot_2018-05-27-11-31-51.001.html
- screenshot_2018-05-27-11-31-51.001.png
- testlogs.txt (REST client & Selenium Capybara logs)
In the future we can extend this to contain a summarized html results that can be exported as a zip file for each run.