Broken QA test suite should be equivalent to broken master

Proposal

We should we raising awareness and urgency when QA tests breaks or are about to break. A broken QA suite is also a ~”broken master”.

  • Update to our handbook.
  • Make aware in the team call.

Once in a while a change to the app (usually a frontend change) will result in an end-to-end test failing and needing to be updated.

Ideally, we'd be aware of the change before it's released and could update the tests before they fail in master. It would be even better if we had the E2E tests running fast enough and stably enough to run on every commit. But we're not there yet.

The qa:selectors job can detect when a change affects a selector, but it can't detect when the functionality is changed without touching existing selectors.

How can be best work with engineers to increase their awareness of the impact app changes have on E2E tests, and to increase our awareness of when E2E tests will need to be updated?

Your thoughts @gl-quality?

Edited by Mek Stittri