Tests/pytest: balance only selected tests

Context

In !3692 (merged) we introduced automatic balancing of pytests jobs in the CI. It works by reading a file of previous timings of test executions and uses that to split the test to run in separate jobs, using a heuristic attempting to have an evenly distributed execution time for each job.

However, the job selection did not take test selection into account. Assuming we have the previous timings:

test_alpha/test_a.py: 10s
test_alpha/test_b.py: 10s
test_011/test_a.py: 10s
test_011/test_b.py: 10s

and we select all tests batched into two jobs. Then we could have the following balancing:

Job 1: test_alpha/test_a.py, test_alpha/test_b.py
Job 2: test_011/test_a.py, test_011/test_b.py

If we only select the jobs in test_011 (i.e. we run poetry run pytest tests_python/tests_011) then the job selection will produce the same balancing as above. However, since no tests of tests_alpha is selected, Job 1 will be empty.

This is not a problem per se (all tests selected tests will always be run in some job). But it might lead to less ideal balancings (as in the example above) and to empty jobs (which is a waste and might lead to other problems down the line). Furthermore, it means that timings for tests that no longer exists in the tree influence balancing.

This MR fixes this problem by only including selected tests in balancing.

Furthermore, I've added some more information to --job-selection-debug.

Manually testing the MR

To test it, you can run (from inside the tests_python folder):

• poetry run pytest --prev-junit-xml test-results.xml --job=1/25 --collect-only -qq prints only the tests that would be run in job 1 of 25.

You can script a bit and run it for all values of --job and make sure it correspond to what you would find if did no balancing.

From inside the tests_python folder:

$ poetry run pytest --collect-only -qq | grep tests_ | sort > tests-no-balancing.txt
$ { for i in $(seq 1 5); do echo $i >&2; poetry run pytest --prev-junit-xml test-results.xml --job=$i/5 --collect-only -qq | grep tests_; done; } | sort > tests-balanced.txt
$ diff tests-no-balancing.txt tests-balanced.txt

You can also try the --jobs-dry-run option:

$ poetry run pytest --jobs-dry-run --prev-junit-xml test-results.xml --job=1/50

Note that it will list a sequence of "orphaned classes". These are tests present in test-results.xml but which do not correspond to any selected tests. In the previous command, all tests were selected so these orphans should be removed in ``test-results.xml` (see #2058 (closed)).

Checklist

• n/a Document the interface of any function added or modified (see the coding guidelines)
• n/a Document any change to the user interface, including configuration parameters (see node configuration)
• n/a Provide automatic testing (see the testing guide).
• n/a For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, the Development Version section of CHANGES.md for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR