Aggregate reporting on SauceLabs
Problem to solve
When running Cross-Browser testing against Sauce, we'll need to be able to aggregate test passes / failures and be able to map them.
Further details
SauceLabs has a built-in video / screenshot reporting. We will need to ensure that runs in sauce correlates with runs on night/staging, local.
Proposal
We should be able to utilize {"build": "$JOB_NAME"}
or the like. When a test is run without any tags, it shows as such
By utilizing Build, we should be able to map the test runs with specific repositories and runtimes.
What does success look like, and how can we measure that?
Wherever we run gitlab-qa, we should be able to
- Know if we ran in SauceLabs (as we won't be running Sauce a lot)
- If we "did" run in SauceLabs, perhaps link to the build in Sauce (our SauceLabs results are public domain)