Karma's Progress Reporter Makes Failing Tests Unclear
The default reporter for karma, progress
, shows one line of output for each test. Ideally, these lines are overwritten, resulting in one total line of output; however, adding additional logs breaks this functionality, greatly increasing the number of lines output.
Line per test:
Chrome: Executed 5457 of 5457 (1 FAILED) (1 min 14.886 secs / 1 min 13.196 secs)
Eventual output:
Chrome: Executed 5200 of 5457 (1 FAILED) (0 secs / 1 min 8.691 secs)
Chrome: Executed 5201 of 5457 (1 FAILED) (0 secs / 1 min 8.707 secs)
Chrome: Executed 5202 of 5457 (1 FAILED) (0 secs / 1 min 8.723 secs)
Chrome: Executed 5203 of 5457 (1 FAILED) (0 secs / 1 min 8.74 secs)
Chrome: Executed 5204 of 5457 (1 FAILED) (0 secs / 1 min 8.749 secs)
Chrome: Executed 5205 of 5457 (1 FAILED) (0 secs / 1 min 8.758 secs)
Chrome: Executed 5206 of 5457 (1 FAILED) (0 secs / 1 min 8.785 secs)
Chrome: Executed 5207 of 5457 (1 FAILED) (0 secs / 1 min 8.801 secs)
Chrome: Executed 5208 of 5457 (1 FAILED) (0 secs / 1 min 8.818 secs)
Chrome: Executed 5222 of 5457 (1 FAILED) (0 secs / 1 min 8.933 secs)
Chrome: Executed 5223 of 5457 (1 FAILED) (0 secs / 1 min 8.948 secs)
Chrome: Executed 5224 of 5457 (1 FAILED) (0 secs / 1 min 8.963 secs)
Chrome: Executed 5226 of 5457 (1 FAILED) (0 secs / 1 min 8.993 secs)
Chrome: Executed 5227 of 5457 (1 FAILED) (0 secs / 1 min 9.011 secs)
Chrome: Executed 5228 of 5457 (1 FAILED) (0 secs / 1 min 9.028 secs)
Chrome: Executed 5229 of 5457 (1 FAILED) (0 secs / 1 min 9.045 secs)
Chrome: Executed 5371 of 5457 (1 FAILED) (0 secs / 1 min 10.777 secs)
Chrome: Executed 5372 of 5457 (1 FAILED) (0 secs / 1 min 10.79 secs)
Chrome: Executed 5373 of 5457 (1 FAILED) (0 secs / 1 min 10.804 secs)
Chrome: Executed 5374 of 5457 (1 FAILED) (0 secs / 1 min 10.812 secs)
Chrome: Executed 5375 of 5457 (1 FAILED) (0 secs / 1 min 10.816 secs)
Chrome: Executed 5376 of 5457 (1 FAILED) (0 secs / 1 min 10.821 secs)
Chrome: Executed 5377 of 5457 (1 FAILED) (0 secs / 1 min 10.825 secs)
Chrome: Executed 5378 of 5457 (1 FAILED) (0 secs / 1 min 10.837 secs)
Chrome: Executed 5379 of 5457 (1 FAILED) (0 secs / 1 min 10.841 secs)
Chrome: Executed 5386 of 5457 (1 FAILED) (0 secs / 1 min 10.918 secs)
Chrome: Executed 5387 of 5457 (1 FAILED) (0 secs / 1 min 10.932 secs)
Chrome: Executed 5388 of 5457 (1 FAILED) (0 secs / 1 min 10.937 secs)
Chrome: Executed 5389 of 5457 (1 FAILED) (0 secs / 1 min 10.948 secs)
Chrome: Executed 5392 of 5457 (1 FAILED) (0 secs / 1 min 10.969 secs)
Chrome: Executed 5393 of 5457 (1 FAILED) (0 secs / 1 min 10.98 secs)
Chrome: Executed 5394 of 5457 (1 FAILED) (0 secs / 1 min 10.984 secs)
Chrome: Executed 5446 of 5457 (1 FAILED) (0 secs / 1 min 12.015 secs)
Chrome: Executed 5449 of 5457 (1 FAILED) (0 secs / 1 min 12.872 secs)
Chrome: Executed 5452 of 5457 (1 FAILED) (0 secs / 1 min 13.096 secs)
Chrome: Executed 5453 of 5457 (1 FAILED) (0 secs / 1 min 13.178 secs)
Chrome: Executed 5457 of 5457 (1 FAILED) (1 min 14.886 secs / 1 min 13.196 secs)
TOTAL: 1 FAILED, 5456 SUCCESS
Scrolling through all that makes finding that 1 failed test very challenging. I propose we use a different reporter locally.
Some options are:
-
We use this in CI for full pretty output of specs and failures, also diffs certain failures.
test errors ✔ has no unhandled Promise rejections ✔ has no Vue warnings ✔ has no Vue error ✔ restores axios adapter after mocking Finished in 1 min 14.415 secs / 1 min 12.769 secs @ 14:55:47 GMT+0000 (Coordinated Universal Time) SUMMARY: ✔ 5455 tests completed ✖ 2 tests failed FAILED TESTS: DiffsStoreActions setInlineDiffViewType ✖ should set diff view type to inline and also set the cookie properly Chrome Uncaught TypeError: Cannot read property 'reduce' of undefined thrown MergeRequest task lists tasklist ✖ shows an error notification when tasklist update failed Chrome Expected 'An error occurred while fetching the job log.' to be 'Someone edited this merge request at the same time you did. Please refresh the page to see changes.'. actual expected AnSomeone erroredited occurredthis whilemerge fetchingrequest at the jobsame logtime you did. Please refresh the page to see changes. at webpack:///Users/andrew/Repos/gdk-ce/gitlab/spec/javascripts/merge_request_spec.js:81 <- spec/javascripts/test_bundle.bundle.js:762981:91
-
Similar to
summary
in that very little is printed, but a summary of test failures is printed after the run is complete. Browser logs are also printed with the summary, which can be noisy. They have been disabled below.5457 total 5454 passed 3 failed 0 skipped Failed Tests: DiffsStoreActions setInlineDiffViewType should set diff view type to inline and also set the cookie properly Chrome 1) Uncaught TypeError: Cannot read property 'reduce' of undefined thrown Confirm Rollback Modal Component should show "Rollback" when isLastDeployment is false Chrome 2) Expected 'This action will run the job defined by for commit abc0123 putting the environment in a previous version. You can revert it by re-deploying the latest version of your application. Are you sure you w ant to continue?' not to contain 'commit abc0123'. at UserContext.<anonymous> (webpack:///Users/andrew/Repos/gdk-ce/gitlab/spec/javascripts/environme nts/confirm_rollback_modal_spec.js:37 <- spec/javascripts/test_bundle.bundle.js:672099:30) MergeRequest task lists tasklist shows an error notification when tasklist update failed Chrome 3) Expected 'An error occurred while fetching the job log.' to be 'Someone edited this merge re quest at the same time you did. Please refresh the page to see changes.'. actual expected AnSomeone erroredited occurredthis whilemerge fetchingrequest at the jobsame logtime you did . Please refresh the page to see changes. at webpack:///Users/andrew/Repos/gdk-ce/gitlab/spec/javascripts/merge_request_spec.js:81 <- spec/javascripts/test_bundle.bundle.js:762981:91
-
Shows only failed tests by default, but can be configured. Designed more for running tests across multiple browsers, but looks nice regardless. Leaves browser logs where they are. Doesn't show why a test failed.
SUMMARY 0: Chrome: Executed 5457 of 5457 (2 FAILED) (1 min 13.314 secs / 1 min 11.645 secs) all 0 DiffsStoreActions setInlineDiffViewType should set diff view type to inline and a... ✖ ✖ Confirm Rollback Modal Component should show "Rollback" when isLastDeploymen... ✖ ✖ 5439 more test cases successful in all browsers
BONUS ROUND: Summary + Mocha
Using the recommended configuration from the summary reporter, we can get test failures AND their details
SUMMARY
0: Chrome: Executed 5457 of 5457 (2 FAILED) (1 min 15.653 secs / 1 min 14.096 secs)
0
DiffsStoreActions
setInlineDiffViewType
should set diff view type to inline and a... ✖
Confirm Rollback Modal Component
should show "Rollback" when isLastDeploymen... ✖
5439 more test cases successful in all browsers
Finished in 1 min 15.653 secs / 1 min 14.096 secs @ 15:34:48 GMT+0000 (Coordinated Universal Time)
SUMMARY:
✔ 5455 tests completed
✖ 2 tests failed
FAILED TESTS:
DiffsStoreActions
setInlineDiffViewType
✖ should set diff view type to inline and also set the cookie properly
Chrome
Uncaught TypeError: Cannot read property 'reduce' of undefined thrown
Confirm Rollback Modal Component
✖ should show "Rollback" when isLastDeployment is false
Chrome
Expected 'This action will run the job defined by for commit abc0123 putting the environment in a previou
s version. You can revert it by re-deploying the latest version of your application. Are you sure you want to
continue?' not to contain 'commit abc0123'.
at UserContext.<anonymous> (webpack:///Users/andrew/Repos/gdk-ce/gitlab/spec/javascripts/environments/
confirm_rollback_modal_spec.js:37 <- spec/javascripts/test_bundle.bundle.js:672099:30)
Honorable Mention: helpful.
The output on helpful is pretty slim, and has a nice progress bar, but it collects all browser logs and puts them after the summary. It provides some configuration to slim that output down, but I could not get it to work in my <5 minutes of playing around. This mean that there were several [Bootstrap-Vue warn]
and [vuex]
messages between my cursor and the test results.
I'm open to pretty much any option. mocha
configured with minimal
output would be great as well! I have no idea how this impacts jest
in any way though.