Improving speed of test suite
This issue is for discussion on how to improve the speed of the test suite.
The following test have been made on a Ryzen 2600 3.4 GHz with /usr/bin/unbuffer /bin/bash -o pipefail -c "make check -j1"|ts
.
List of slow tests
Name | Duration | Reason |
---|---|---|
tls13/key_update | 15s | |
dtls-rehandshake-anon | 61s | Sleep 60 |
mini-record | 6s | |
record-retvals | 5s | |
dtls-rehandshake-cert | 61s | Sleep 60 |
mini-dtls-hello-verify-48 | 30s | |
rng-sigint | 5s | |
rng-op-nonce | 8s | |
rng-op-random | 8s | |
rng-op-key | 8s | |
dtls/dtls | 75s | |
starttls-smtp.sh | 43s | |
starttls-pop3.sh | 41s | |
starttls-sieve.sh | 5s | |
ocsp-tests/ocsp-tls-connection | 5s | |
ocsp-tests/ocsp-must-staple-connection | 14s | |
testpkcs11.sh | 9s | |
cert-tests/provable-dh | 8s | |
cert-tests/provable-privkey-dsa2048 | 7s | |
cert-tests/provable-dh-default | 5s | |
cert-tests/dsa | 5s | |
slow/test-hash-large.sh | 38s | |
suite/chain.sh | 12s | |
suite/testcompat-openssl.sh | 50s | |
suite/testrandom.sh | 16s | |
suite/tls-fuzzer/tls-fuzzer-nocert.sh | 109s | |
suite/tls-fuzzer/tls-fuzzer-nocert-tls13.sh | 127s | |
suite/testcompat-tls13-openssl.sh | 39s | |
suite/testdane.sh | 15s | |
suite/prime-check | 7s |
Maybe that list helps to separate tests into directories/groups that could by run in parallel on different runners, decreasing the total time of our CI. We could use either configure flags or make check -C ...
to chose the groups to test.
There also might be some tests with optimization potential. At least there are some tests idling / waiting without much CPU consumption.