Fail feature specs on unexpected browser console errors

Miguel Rinconrequested to merge
feature-specs-browser-console-allowlist-refresh into master

Why

The Vue 3 migration can ship real bugs to production, because our feature specs do not see them.

Vue 2 catches a component error, logs it to the browser console, and renders the component blank. The rest of the page keeps working, so the spec passes. Vue 3 does not recover from the same error in the development build. The page breaks. This is how the Rapid Diffs reply placeholder crash (#628791 (closed)) reached the Vue 3 migration of the page. Every feature spec was green under Vue 2, the migration merged, and it had to be reverted.

Our specs did check the console, but only for examples that had already failed. A passing example discarded its console output. So a whole class of latent bugs stayed invisible: errors that Vue 2 hides today and that Vue 3 will expose page by page as the migration flips feature flags.

What

Feature specs now fail on any unexpected browser console error, in both the Vue 2 and the Vue 3 CI jobs. Known errors are listed in an allowlist so that the check can merge today. Each entry is a bug to fix or to track.

This MR replaces !255207 (closed) with the same code and an allowlist regenerated after the fixes from #628901. The last of those fixes, !255392 (merged), has merged, so this MR sits directly on master.

How

  • spec/support/capybara.rb: the after(:example, :js) hook calls raise_if_unexpected_browser_console_output for every example, not only for failed ones.
  • spec/support/helpers/browser_console_helpers.rb: the noise filter BROWSER_CONSOLE_ERROR_FILTER ignores messages that come from the test environment, not from the application. Examples: failed 4xx/5xx requests that specs trigger on purpose, Content Security Policy blocks for third-party scripts, the Sentry stub, the missing web manifest, the ActionCable handshake.
  • spec/support/browser_console_allowlist.yml: known application errors. An entry has a message, the plain error text, the specs it applies to (required, so an entry never hides the same error on another page), and an optional issue key. No regexps and no Vue version key: the same entry covers the Vue 2 and Vue 3 jobs.
  • An entry matches when its text occurs in the console message. Before the comparison, the check removes Chrome's URL line:column prefix, quotes and backslashes from both sides. So TypeError: Cannot read properties of null (reading 'text') covers the uncaught form, the Vue 2 [Vue warn]: Error in render: "TypeError: …" form and the Vue 3 [gitlab] [Vue warn]: … form of the same error. An empty message allows only a console error with no text, which Chrome logs for an error object without a message.
  • spec/support_specs/browser_console_allowlist_spec.rb: validates the file. Every glob must match an existing spec file, so an entry dies with the spec it covers.

Things a reviewer may stop on:

  • The spec file comes from metadata[:rerun_file_path], not file_path. For shared example groups, file_path points into the shared examples file.
  • Under Vue 2, [Vue warn] uses console.error, so prop validation warnings fail only the Vue 2 jobs. Under Vue 3 they use console.warn and pass. Vue 3 component errors reach the console through the non-production errorHandler from !255205 (merged).
  • The allowlist has no catch-all entry. Every entry names one error with its exact text. Missing field and subscription entries list the exact field and subscription names, split per spec file. The two entries for a bare Object and for an empty message cover the errors logged without text.
  • The check raises only after the session reset and the request draining in the same hook, so a console error never leaks in-flight requests into the next example.

Alternatives ruled out:

  • Grow the global noise filter with application errors. It would hide the same error on every page, which is the problem this MR fixes.
  • Fix every error before merging the check. Each week without the check lets new errors in. The allowlist makes the check land now and shrinks as fixes merge.
  • Per-spec allow: metadata instead of a central file. It spreads the list over 52 files and loses the validation that ties entries to existing specs.

The allowlist

!255207 (closed) (2026-09-12) This MR (2026-09-25)
Entries 50 42
Spec files 199 59
Regexp entries 50 0
Entries with an issue 0 2

Two errors are new since the pipeline of 2026-09-22: Missing field 'project' and Missing field 'job' on the job pages, six spec files, found by pipeline https://gitlab.com/gitlab-org/gitlab/-/pipelines/2881995978 on master of 2026-09-25.

How the list was generated and what is in it
  1. A temporary commit made the console check log every SEVERE message that passed the noise filter, as one JSON line with spec file, example id and Vue version, and never raise. Pipeline https://gitlab.com/gitlab-org/gitlab/-/pipelines/2867610835 ran all 102 rspec system jobs, Vue 2 and Vue 3, green.
  2. A script grouped the 466 messages by pattern and spec file. The temporary commit was dropped.
  3. Cross-check against the reveal MR !255199, pipeline https://gitlab.com/gitlab-org/gitlab/-/pipelines/2867469601, run on master the same day. It found the same errors plus four that the record run missed by timing. They are in the list. Its only extra error is the GlBreadcrumb validator warning that !255392 (merged) fixes.
  4. A second reveal run on 2026-09-22, pipeline https://gitlab.com/gitlab-org/gitlab/-/pipelines/2872215632, after !255392 (merged) merged. It confirmed the list, added the workItemNotes subscription error and two spec files to existing entries, and the observed messages were used to narrow every pattern (Vue version, exact names, anchors). One of its 102 system jobs hung and produced no data; the spec it covers is already in the list.
  5. Entries seen in only one of the three runs, so they are timing dependent: The emoji map is uninitialized, the custom role approval subscription errors and Unauthorized subscription, Missing field 'discussions'. They stay in the list because removing them would make the check flaky.
  6. On 2026-09-25 the regexp entries were rewritten as plain messages, the Vue version key was dropped, and the failures of pipeline https://gitlab.com/gitlab-org/gitlab/-/pipelines/2881995978 on fresh master added the two Missing field entries for the job pages.
  7. Local check on 2026-09-25: all 59 spec files ran in GDK with a recorder that logged every allowed console message. 23 of the 42 entries fired locally. The rest are Vue 3 job errors, which GDK cannot reproduce with its Vue 2 build, or Vue 2 prop warnings that the Vite dev build does not emit. The GDK run also logged errors that CI never does, such as a toJSON prop warning from the dev build; those were not added. CI stays the source of truth for the list.

Groups:

  • @gitlab/at.js reading 'text': 13 spec files, #628981
  • GitLab UI GlBreadcrumb reading 'clientWidth' after unmount: 3 service account specs, gitlab-org/gitlab-services/design.gitlab.com#3650
  • Vue 2 warnings: computed value is readonly (7 incident and abuse report specs), Invalid prop "text" on GlBreadcrumbItem and GlTruncate (7 specs), "placeholder" (Irker integration), "noteId" (Rapid Diffs batch comments)
  • API fuzzing configuration page, one spec, 5 entries: the apiFuzzingCiConfiguration query returns 500 and the form renders without data
  • Homepage, new since the old baseline: reading 'assigned', reading 'recentlyViewedItems', Cannot destructure property 'id', The emoji map is uninitialized
  • Other uncaught errors, one spec each: reading 'hide' (admin runners modal), reading 'close' (pipeline editor), reading 'push' (dashboard issues), reading 'getAttribute' (EE wiki), reading 'projectRepositoryRegistries' (Geo), reading 'namespace' (filtered search dropdowns), a permission error on group work items
  • Data and API: Missing field cache writes (job pages, pipelines, quick actions), runners query errors and a bare Object (pipeline editor, custom role approval, epic deletion), subscription errors and Unauthorized subscription (custom role approval), workItemNotes subscription errors (epic deletion), subscriptionPermissions (usage quotas), Zoekt and Elasticsearch query errors, an empty message (cycle analytics charts)

Gone since the old baseline, because the fixes merged: humanTimeEstimate, _this12 is not defined, ParametersMissing, Invalid time value, No match for, __vnode, and the section B and C batches of the work item.

How to verify

  1. Run the validator: bin/rspec spec/support_specs/browser_console_allowlist_spec.rb.
  2. Run a feature spec that has a known console error and an allowlist entry, for example bin/rspec spec/features/projects/service_accounts_spec.rb. It passes.
  3. Remove the clientWidth entry from spec/support/browser_console_allowlist.yml. Run the spec again. It fails with BrowserConsoleError.

Expected pipeline result: green. A red rspec system job means a console error that is not in the allowlist: fix the error or add an entry.

No UI change.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

References

🤖 Generated with Claude Code

Edited by Miguel Rincon

Merge request reports