Test errors on GitLab shared macOS runner
Summary
Running tests on GitLab shared macOS runners produces the following errors (all others passed, latest log is in this job):
× tests/pageanrc.test.js > pageanrc lint > should lint the all errors pageanrc and return all JSON errors to stdout with -j option 251ms
→ Snapshot `pageanrc lint > should lint the all errors pageanrc and return all JSON errors to stdout with -j option > stdout 1` mismatched
× tests/tests.test.js > renderedHtmlTest > should not execute test if isEnabled setting is false 30565ms
→ Navigation timeout of 30000 ms exceeded
× tests/tests.test.js > renderedHtmlTest > should log a passed test result for page that has valid HTML 43162ms
→ Navigation timeout of 30000 ms exceeded
× tests/tests.test.js > renderedHtmlTest > should not log data with HTML errors for a passed test 30076ms
→ Navigation timeout of 30000 ms exceeded
× tests/index.test.js > integration tests > should execute all tests and report failure with the correct tests failed 120007ms
→ Test timed out in 120000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
× tests/index.test.js > integration tests > should pass reporter config when saving reports 61244ms
→ expected [ 'cli', 'html', 'json' ] to strictly equal [ 'cli', 'json' ]
Another example is this job:
× tests/pageanrc.test.js > pageanrc lint > should lint the all errors pageanrc and return all JSON errors to stdout with -j option 161ms
→ Snapshot `pageanrc lint > should lint the all errors pageanrc and return all JSON errors to stdout with -j option > stdout 1` mismatched
× tests/tests.test.js > renderedHtmlTest > should not execute test if isEnabled setting is false 30606ms
→ Navigation timeout of 30000 ms exceeded
× tests/tests.test.js > renderedHtmlTest > should log a passed test result for page that has valid HTML 44763ms
→ Navigation timeout of 30000 ms exceeded
× tests/tests.test.js > renderedHtmlTest > should not log data with HTML errors for a passed test 30273ms
→ Navigation timeout of 30000 ms exceeded
× tests/index.test.js > integration tests > should execute all tests and report failure with the correct tests failed 120012ms
→ Test timed out in 120000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
× tests/index.test.js > integration tests > should pass reporter config when saving reports 60456ms
→ expected [ 'cli', 'html', 'json' ] to strictly equal [ 'cli', 'json' ]
This was with the following updates:
.gitlab-ci.yml
:
# Included from Npm-Package-Base.gitlab-ci.yml
# Extend timeout due to slow test execution.
node_lts_test_macos:
tags:
- saas-macos-large-m2pro
timeout: 10m
vitest.config.ts
const timeout = 120_000;
The jobs are also timing out at 10 minutes. Given the frequency of jobs failing to initialize properly on macOS runners, increasing to a longer time is not advised.
This shows that puppeteer
is able to launch a browser as numerous browser-based tests pass, but there is some other problem. It does appear the same tests are failing, which points towards a specific issue.
Edited by Aaron Goldenthal