Skip to content

Draft: Reduce sleeping in tests

Michal Liszcz requested to merge github/fork/mliszcz/reduce-sleeping-in-tests into main

Two weeks ago we discussed with @bourtemb the problem of sleeps/waits in test suites.

I'd like to trigger a discussion (in PR, just to show some code) - whether we shall focus (or not) on the problem and how can we improve our tests.

I've done a simple PoC of an event callback that maintains event queue and allows to perform assertions on received events. When applied to dev_intr_event testcase, execution time went down from ~11s to ~2s.

Pros:

  • test suite execution time reduction -> faster CI feedback
  • simpler test code (calback's state is reduced), no need to track cb_executed, etc.
  • we can safely increase timeout value for cases where current sleep period is sometimes too low (unstable testcases) -> CI stability improvement

Cons:

  • may hide some bugs when blindly applied to all sleep statements (see discussion in #524 (closed))

It's C++11 only, but I think with enough patience it can be rewritten to pthreads (lambdas will be a problem). Maybe we can start introducing C++11 first in test code (as there is little end-user impact).

Long running testcases:

  • 100.09 CXX::cxx_reconnection_zmq
  • 55.07 CXX::cxx_stateless_subscription
  • 49.28 CXX::cxx_old_poll
  • 44.21 CXX::cxx_poll_admin
  • 40.12 event::multi_dev_event
  • 38.66 old_tests::lock
  • 38.18 CXX::cxx_fwd_att
  • 32.08 asyn::asyn_write_cb
  • 32.07 asyn::asyn_attr_cb
  • 31.21 event::change_event
  • 30.10 asyn::asyn_attr
  • 30.07 asyn::asyn_write_attr
  • 29.17 event::per_event
  • 29.07 asyn::asyn_write_attr_multi
  • 29.07 asyn::asyn_cb_cmd
  • 29.07 asyn::asyn_attr_multi
  • 28.26 event::att_type_event
  • 27.07 asyn::asyn_cmd
  • 26.12 event::archive_event
  • 24.03 asyn::auto_asyn_cmd
  • 23.13 event::change_event_buffer
  • 23.12 event::multi_event
  • 21.09 CXX::cxx_server_event
  • 20.05 CXX::cxx_asyn_reconnection
  • 13.06 asyn::asyn_cb2
  • 13.03 asyn::asyn_cb
  • 12.29 event::att_conf_event_buffer
  • 12.16 old_tests::att_conf
  • 11.31 old_tests::ring_depth
  • 11.11 event::state_event
  • 11.11 event::data_ready_event_buffer
  • 11.10 event::dev_intr_event
  • 11.09 event::change_event64
  • 10.20 CXX::cxx_attr_misc
  • 9.79 old_tests::state_attr
  • 9.07 CXX::cxx_dserver_misc
  • 8.07 CXX::cxx_enum_att
  • 8.07 CXX::cxx_blackbox
  • 8.04 CXX::cxx_signal
  • 7.07 event::pipe_event
  • 6.07 event::event_lock
  • 6.04 CXX::cxx_attr_write
  • 5.20 old_tests::rds
  • 5.07 old_tests::attr_proxy
  • 4.34 old_tests::attr_conf_test
  • 4.05 CXX::cxx_class_dev_signal
  • 4.04 CXX::cxx_class_signal
  • 4.03 old_tests::sub_dev
  • 3.55 old_tests::mem_att
  • 3.08 CXX::cxx_misc
  • 3.05 CXX::cxx_poll
  • 2.07 event::user_event
  • 2.06 old_tests::ConfEventBugClient
  • 2.06 event::data_ready_event
  • 2.03 event::att_conf_event
  • 1.07 CXX::cxx_pipe_conf
Edited by Tomasz Madej

Merge request reports