Skip to content

Test errors on GitLab shared macOS runner

Summary

Running tests on GitLab shared macOS runners produces the following errors (all others passed, latest log is in this job):

 × tests/pageanrc.test.js > pageanrc lint > should lint the all errors pageanrc and return all JSON errors to stdout with -j option 251ms
   → Snapshot `pageanrc lint > should lint the all errors pageanrc and return all JSON errors to stdout with -j option > stdout 1` mismatched
× tests/tests.test.js > renderedHtmlTest > should not execute test if isEnabled setting is false 30565ms
   → Navigation timeout of 30000 ms exceeded
× tests/tests.test.js > renderedHtmlTest > should log a passed test result for page that has valid HTML 43162ms
   → Navigation timeout of 30000 ms exceeded
× tests/tests.test.js > renderedHtmlTest > should not log data with HTML errors for a passed test 30076ms
   → Navigation timeout of 30000 ms exceeded
 × tests/index.test.js > integration tests > should execute all tests and report failure with the correct tests failed 120007ms
   → Test timed out in 120000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
 × tests/index.test.js > integration tests > should pass reporter config when saving reports 61244ms
   → expected [ 'cli', 'html', 'json' ] to strictly equal [ 'cli', 'json' ]

Another example is this job:

× tests/pageanrc.test.js > pageanrc lint > should lint the all errors pageanrc and return all JSON errors to stdout with -j option 161ms
   → Snapshot `pageanrc lint > should lint the all errors pageanrc and return all JSON errors to stdout with -j option > stdout 1` mismatched
× tests/tests.test.js > renderedHtmlTest > should not execute test if isEnabled setting is false 30606ms
   → Navigation timeout of 30000 ms exceeded
× tests/tests.test.js > renderedHtmlTest > should log a passed test result for page that has valid HTML 44763ms
   → Navigation timeout of 30000 ms exceeded
× tests/tests.test.js > renderedHtmlTest > should not log data with HTML errors for a passed test 30273ms
   → Navigation timeout of 30000 ms exceeded
× tests/index.test.js > integration tests > should execute all tests and report failure with the correct tests failed 120012ms
   → Test timed out in 120000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
 × tests/index.test.js > integration tests > should pass reporter config when saving reports 60456ms
   → expected [ 'cli', 'html', 'json' ] to strictly equal [ 'cli', 'json' ]

This was with the following updates:

.gitlab-ci.yml:

# Included from Npm-Package-Base.gitlab-ci.yml
# Extend timeout due to slow test execution.
node_lts_test_macos:
  tags:
    - saas-macos-large-m2pro
  timeout: 10m

vitest.config.ts

const timeout = 120_000;

The jobs are also timing out at 10 minutes. Given the frequency of jobs failing to initialize properly on macOS runners, increasing to a longer time is not advised.

This shows that puppeteer is able to launch a browser as numerous browser-based tests pass, but there is some other problem. It does appear the same tests are failing, which points towards a specific issue.

Edited by Aaron Goldenthal
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information