Loading
test(pubsub): xfail flaky blocking subscription test and log diagnostics
Summary
The test test_subscriptions_blocking fails at random in CI with this error: TimeoutError: Timeout while waiting for message. (#415).
In the last month, the test failed in 17 of approximately 1,660 test jobs on all branches. On main, it failed in 4 of approximately 290 jobs. The failures occur on all Python versions (3.10 to 3.14).
We do not know the root cause yet. Thus, this MR does not try to fix the test. It marks the test as xfail and records data when the test fails. When we have more failures, we will use this data to find the cause.
Changes
- The test has a non-strict
xfailmarker (strict=False, raises=AssertionError). A random failure does not stop the pipeline. Other errors continue to stop the pipeline. - When the test does not receive an event, it writes a
FlakyTestDiagnosticswarning. The warning shows in the warnings summary of the CI log. The test then fails with the same text. The warning gives this data:- The event that the test did not receive, and its position in the sequence.
- The number of
Connectedprobes, and the time to connect. - The subscriptions that the publisher received but nobody read.
- The result of a second publish of the same event. If the event arrives, the subscription was late. If the event does not arrive, the connection is broken.
test_subscriptions_asyncandtest_subscriptions_blockingno longer change the shared test data (events += ...). Before this change, the blocking test sent theDoneevent two times.
Notes
- One failure occurred in the case
[without_topic_incorrect]. In this case, the test expects only one message: theDoneevent. Thus, the publisher did not sendDone. This can be a race between theDonesubscription and theConnectedprobe. We did not confirm this. - To find new failures, search the logs of the test jobs for
FlakyTestDiagnostics.
Related to #415
Edited by Koushik Kumaran