Skip to content

Speed up test execution by re-using a moto server

Cal Pratt requested to merge cpratt34/faster-testing into master

The main edit performed in this MR is to eliminate the need for tests to run in isolation from eachother. This PR completely removes xdist isolation from tests.

For local runs, the big blocker was having a Mock AWS server spun up for each of the S3 tests and using xdist to block concurrent access to it, as each test was sharing bucket names. Additionally, with out pytest settings (fork mode), session scoped fixtures get re-created on each test. This added a 2-3 second startup delay on every test using S3.

Now instead, we expect that moto_server is spawned before the tests start. The fixture simply checks that the port is active, which avoids needing to create a subprocess per test. The bucket names in tests are now also prefixed with a guid, such that each test can talk to the same moto server without conflicting.

Currently the bots tests are still running slow. I mitigated this somewhat by allowing the sleep time in the job assigner to be lowered from 1 second. Additionally, buildgrid client retrier was ignoring the max_backoff argument. Now that it respects this value, retry failure tests are much faster.

Overall, on my MacBook with 10 worker threads I can run the full suite in under 1m30s, and the CI time is down from 20 minutes to 10.

Merge request reports