Add MR Dangerbot functionality to report on e2e:package-and-test for developer awareness
Currently, e2e:package-and-test
functionality is not being utilized in many MRs with failures being detected only after the merge. After that, the changes can progress quickly and reach production if the tests are not blocking. It is especially true for orchestrated tests that are not run against live environments. We should try to shift left and detect regressions early in MRs.
Main challenges
- Manual
e2e:package-and-test
is not triggered in an MR - The job is triggered automatically but the results are ignored as it is not blocking
- Results are not easy to find and interpret
Proposed changes
-
Issue a Danger warning to run e2e:package-and-test
when the code changes contain qa selectors. Consider adding a note about testing changes under feature flag to avoid those going undetected. It might also be helpful to link to the current pipeline triage issue to help determine if the failures are known. MR: gitlab-org/gitlab!105978 (merged) -
[Bonus] Detect deprecated .qa-selector
class being re-introduced and issue a Danger warning with a link to documentation. MR: gitlab-org/gitlab!109181 (merged)
Edited by Anastasia McDonald