Handle k6 crashes better with clear error reporting
If k6 crashes during a test run it will quietly fail and GPT moves onto the next test:
Quiet k6 OOM crash
running (0m40.0s), 100/100 VUs, 1 complete and 0 interrupted iterations
default [ 67% ] 100/100 VUs 0m40.0s/1m00.0s
running (0m41.0s), 100/100 VUs, 1 complete and 0 interrupted iterations
default [ 68% ] 100/100 VUs 0m41.0s/1m00.0s
running (0m42.0s), 100/100 VUs, 3 complete and 0 interrupted iterations
default [ 70% ] 100/100 VUs 0m42.0s/1m00.0s
running (0m43.0s), 100/100 VUs, 4 complete and 0 interrupted iterations
default [ 72% ] 100/100 VUs 0m43.0s/1m00.0s
running (0m44.0s), 100/100 VUs, 4 complete and 0 interrupted iterations
default [ 73% ] 100/100 VUs 0m44.0s/1m00.0s
Running k6 test 'git_pull' against environment '50k_chromium_experimental'...
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: k6/tests/git/git_pull.js
output: -
scenarios: (100.00%) 1 scenario, 100 max VUs, 1m30s max duration (incl. graceful stop):
* default: Up to 100 looping VUs for 1m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)
time="2024-02-15T16:28:55Z" level=info source=console
time="2024-02-15T16:28:55Z" level=info msg="Git Protocol RPS: 100" source=console
time="2024-02-15T16:28:55Z" level=info msg="RPS Threshold: 80.00/s (4800)" source=console
time="2024-02-15T16:28:55Z" level=info msg="TTFB P90 Threshold: 400ms" source=console
time="2024-02-15T16:28:55Z" level=info msg="Success Rate Threshold: 99%" source=console
time="2024-02-15T16:28:55Z" level=info msg="Commit #43e984808d1bef7debc935dded0c5aba21861ffc exists" source=console
time="2024-02-15T16:28:55Z" level=info msg="Commit #dc7563588833ccad6482a3aa5848108c4387e7e5 exists" source=console
running (0m01.0s), 005/100 VUs, 0 complete and 0 interrupted iterations
default [ 0% ] 005/100 VUs 0m00.2s/1m00.0s
Obviously this can lead to easy misses although the GPT output does show the test was attempted. It would be better for the crashed to be captured and if k6 threw and error for that to be reported by GPT who in turn errors out accordingly. Task is to explore.
Edited by Grant Young