Reduce sleeping in tests
Moving discussion from this MR: !558 (closed)
Some POC code can be found in the mentioned MR. Original description is also below:
I'd like to trigger a discussion (in PR, just to show some code) - whether we shall focus (or not) on the problem and how can we improve our tests.
I've done a simple PoC of an event callback that maintains event queue and allows to perform assertions on received events. When applied to dev_intr_event
testcase, execution time went down from ~11s to ~2s.
Pros:
- test suite execution time reduction -> faster CI feedback
- simpler test code (calback's state is reduced), no need to track
cb_executed
, etc. - we can safely increase timeout value for cases where current sleep period is sometimes too low (unstable testcases) -> CI stability improvement
Cons:
- may hide some bugs when blindly applied to all sleep statements (see discussion in #524 (closed))
It's C++11 only, but I think with enough patience it can be rewritten to pthreads (lambdas will be a problem). Maybe we can start introducing C++11 first in test code (as there is little end-user impact).
Long running testcases:
-
100.09 CXX::cxx_reconnection_zmq -
55.07 CXX::cxx_stateless_subscription -
49.28 CXX::cxx_old_poll -
44.21 CXX::cxx_poll_admin -
40.12 event::multi_dev_event -
38.66 old_tests::lock -
38.18 CXX::cxx_fwd_att -
32.08 asyn::asyn_write_cb -
32.07 asyn::asyn_attr_cb -
31.21 event::change_event -
30.10 asyn::asyn_attr -
30.07 asyn::asyn_write_attr -
29.17 event::per_event -
29.07 asyn::asyn_write_attr_multi -
29.07 asyn::asyn_cb_cmd -
29.07 asyn::asyn_attr_multi -
28.26 event::att_type_event -
27.07 asyn::asyn_cmd -
26.12 event::archive_event -
24.03 asyn::auto_asyn_cmd -
23.13 event::change_event_buffer -
23.12 event::multi_event -
21.09 CXX::cxx_server_event -
20.05 CXX::cxx_asyn_reconnection -
13.06 asyn::asyn_cb2 -
13.03 asyn::asyn_cb -
12.29 event::att_conf_event_buffer -
12.16 old_tests::att_conf -
11.31 old_tests::ring_depth -
11.11 event::state_event -
11.11 event::data_ready_event_buffer -
11.10 event::dev_intr_event -
11.09 event::change_event64 -
10.20 CXX::cxx_attr_misc -
9.79 old_tests::state_attr -
9.07 CXX::cxx_dserver_misc -
8.07 CXX::cxx_enum_att -
8.07 CXX::cxx_blackbox -
8.04 CXX::cxx_signal -
7.07 event::pipe_event -
6.07 event::event_lock -
6.04 CXX::cxx_attr_write -
5.20 old_tests::rds -
5.07 old_tests::attr_proxy -
4.34 old_tests::attr_conf_test -
4.05 CXX::cxx_class_dev_signal -
4.04 CXX::cxx_class_signal -
4.03 old_tests::sub_dev -
3.55 old_tests::mem_att -
3.08 CXX::cxx_misc -
3.05 CXX::cxx_poll -
2.07 event::user_event -
2.06 old_tests::ConfEventBugClient -
2.06 event::data_ready_event -
2.03 event::att_conf_event -
1.07 CXX::cxx_pipe_conf