RSpec tests which produce output in the browser's console should fail
Currently, if the browser console is populated during an RSpec feature test with a user-defined console.log
, or a [Vue warn]:
, or a javascript exception, nothing happens and the output is ignored.
This is bad for several reasons:
- If a test fails due to some javascript exception, we don't see a relevant error message in the test output and it will be more difficult to debug.
- If a javascript error is occurring in which the test still "passes", it will never be brought to anyone's attention.
- Valuable warnings from frontend libraries like Vue about missing props or bad practices will go unnoticed.
We ought to force our tests to fail on any unexpected console output. There are likely a lot of errors which we are seeing in sentry that could be caught automatically while running our integration tests.
After attempting to add this in gitlab-org/gitlab-ce!18355 I immediately noticed dozens of tests which started failing due to unexpected console output (see the pipeline here), so in order to make this change we're going to need to fix each of these first.
I'd like to propose that we tackle this in stages:
-
Stage 1. Produce console output when a test fails. At least this will give some context to devs when a javascript exception happens to coincide with a test failure. -
Stage 2. Fail tests in which console "errors" are encountered. This covers exceptions and console.error()
output from the frontend. All current tests in which console errors are produced will need to be fixed. -
Stage 3. Fail tests in which any unexpected console output is encountered. This covers warnings from libraries, error responses from xhr requests, broken asset links, etc. IMHO if we are writing our tests correctly, these things should not ever occur so it makes sense to fail when encountering them. However this might be a large undertaking as all of our existing tests were written without this constraint in mind.
Edited by Mike Greiling