test(pubsub): xfail flaky blocking subscription test and log diagnostics

Summary

The test test_subscriptions_blocking fails at random in CI with this error: TimeoutError: Timeout while waiting for message. (#415).

In the last month, the test failed in 17 of approximately 1,660 test jobs on all branches. On main, it failed in 4 of approximately 290 jobs. The failures occur on all Python versions (3.10 to 3.14).

We do not know the root cause yet. Thus, this MR does not try to fix the test. It marks the test as xfail and records data when the test fails. When we have more failures, we will use this data to find the cause.

Changes

  • The test has a non-strict xfail marker (strict=False, raises=AssertionError). A random failure does not stop the pipeline. Other errors continue to stop the pipeline.
  • When the test does not receive an event, it writes a FlakyTestDiagnostics warning. The warning shows in the warnings summary of the CI log. The test then fails with the same text. The warning gives this data:
    • The event that the test did not receive, and its position in the sequence.
    • The number of Connected probes, and the time to connect.
    • The subscriptions that the publisher received but nobody read.
    • The result of a second publish of the same event. If the event arrives, the subscription was late. If the event does not arrive, the connection is broken.
  • test_subscriptions_async and test_subscriptions_blocking no longer change the shared test data (events += ...). Before this change, the blocking test sent the Done event two times.

Notes

  • One failure occurred in the case [without_topic_incorrect]. In this case, the test expects only one message: the Done event. Thus, the publisher did not send Done. This can be a race between the Done subscription and the Connected probe. We did not confirm this.
  • To find new failures, search the logs of the test jobs for FlakyTestDiagnostics.

Related to #415

Edited by Koushik Kumaran

Merge request reports

Loading
Loading