Draft: Fail feature specs on unexpected browser console errors

Miguel Rinconrequested to merge
feature-specs-browser-console-allowlist into master

What does this MR do and why?

Fails feature specs on unexpected browser console errors, with a baseline of the errors that specs log today.

Before, the config.after(:example, :js) hook in spec/support/capybara.rb called raise_if_unexpected_browser_console_output only when the example had already failed. A passing feature spec never checked the browser console. Vue 2 catches a component render error, logs it with console.error, and renders the component blank. The page still works, the spec passes, and the error is discarded. Vue 3 in the development build did not tolerate the same error and broke the page. So the Rapid Diffs reply placeholder crash (#628791 (closed)) surfaced only after the Vue 3 migration of the page merged, and the migration was reverted.

Changes:

  • spec/support/capybara.rb: the console check runs for every :js example, except when the exception is RSpec::Core::Pending::PendingExampleFixedError. The hook passes the example so the check knows the spec file path.
  • spec/support/helpers/browser_console_helpers.rb, BROWSER_CONSOLE_ERROR_FILTER: ignores messages that come from the test environment, not from the application:
    • Failed to load resource (Chrome logs every 4xx/5xx response as SEVERE; many specs assert on them)
    • Content Security Policy violations and refused connections (no network in CI: Snowplow, Zuora, OneTrust, Sentry envelope, media)
    • ArkoseLabs initialization error, googletagmanager.com
    • [Sentry stub] (the stub forwards captureException to console.error)
    • /-/manifest.json syntax error, GraphQL explorer embedded fonts
    • user_counts 401 after the session ends
    • DOMException: Transition was skipped, blocked beforeunload panel
    • ActionCable WebSocket handshake failure
  • New spec/support/browser_console_allowlist.yml: known application errors that specs may still log. Keys per entry:
    • pattern: regexp matched against the console message (required)
    • specs: spec file paths or globs the entry applies to (required, so an entry never hides the error on other pages)
    • vue: 2 or 3, to scope the entry to one Vue major version (optional)
    • issue: full URL of the issue that tracks the fix (optional)
  • raise_if_unexpected_browser_console_output allows a SEVERE message only when an entry's pattern matches, the current spec file matches one of the entry's specs, and the vue version matches when set. The spec file comes from metadata[:rerun_file_path], so examples from shared example groups are scoped to the spec file that includes them, not to the shared examples file.
  • New validator spec/support_specs/browser_console_allowlist_spec.rb: only allowed keys, patterns compile, issue is a full gitlab.com URL when set, vue is 2 or 3 when set, specs is present and every glob matches at least one existing file (entries die with the specs they reference), no duplicate patterns.

Under Vue 2, [Vue warn] messages use console.error and are SEVERE. Under Vue 3 they use console.warn and are not. So prop-validation warnings fail only the Vue 2 jobs.

Related: !255205 (merged) installs a non-production Vue 3 errorHandler, so Vue 3 recovers from component errors like production and Vue 2 do. With it, Vue 3 errors also become console lines that this check catches.

The baseline

50 entries covering 199 spec files, generated from the traces of the reveal pipelines https://gitlab.com/gitlab-org/gitlab/-/pipelines/2843011566 and https://gitlab.com/gitlab-org/gitlab/-/pipelines/2843079787, and from this MR's first two pipelines https://gitlab.com/gitlab-org/gitlab/-/pipelines/2843143467 and https://gitlab.com/gitlab-org/gitlab/-/pipelines/2843157294. No entry has an issue yet. Each entry is an error to fix right away or to track in an issue; that decision is made per entry. #628901 catalogues every entry with job links, spec files, a likely cause and a suggested fix, and tells the history of how the list came to be.

Largest groups:

  • Apollo Missing field 'humanTimeEstimate' while writing result: boards sidebar, 8 spec files
  • Uncaught ReferenceError: _this12 is not defined: analytics and compliance dashboards, 9 spec files
  • Uncaught Error: No match for (vue: 3): merge request list filtering and sorting, 15 spec files, #628844 (closed)
  • Uncaught ParametersMissing: Route missing required keys: id: Pages settings
  • Uncaught RangeError: Invalid time value: branches pages
  • About 12 [Vue warn]: Invalid prop variants
  • The Rapid Diffs __vnode (vue: 3) and reading 'id' crash, #628791 (closed)

Expected pipeline result: green when the baseline is complete. A red rspec system job means a console error that is not in the baseline: add an entry or fix the error.

References

Screenshots or screen recordings

No UI change.

How to set up and validate locally

  1. Run the validator: bin/rspec spec/support_specs/browser_console_allowlist_spec.rb.
  2. Run a feature spec that has a known console error and an allowlist entry, for example bin/rspec spec/features/projects/branches_spec.rb. It passes.
  3. Remove the Invalid time value entry from spec/support/browser_console_allowlist.yml. Run the spec again. It fails with BrowserConsoleError and prints Uncaught RangeError: Invalid time value.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Miguel Rincon

Merge request reports