Skip to content

CI: Rebalance integration tests after Kathmandu freeze

Context

Depends on: !7351 (merged)

Tezt and pytest jobs have become quite unbalanced. This is how it looks after the freeze of Kathmandu in !7351 (merged), on which this MR depends:

Tezt: image

Pytest: image

Pytest old protocols: image

This MR:

  1. Changes tezt/records/update.ml so that it fetches jobs from pipelines in batches of 100 instead of 20 (the default). This speeds up the script.
  2. Fixes fetching of artifacts in tezt/records/update.ml. It now fetches artifacts using the API. The old implementation broke due to a change in gitlab (the previous URL now shows an html page that the user has to click through to get the artifact).
  3. updates the tezt records and pytest test-results.xml from pipeline 746431821 in !7351 (merged)
  4. pytest and pytest_old_protocols are split in too many jobs (16 respectively 20). I reduce it to 10 respectively 8.
    • the sequential running time of the pytest_old_protocols is 102 minutes split over 16 jobs. There is roughly one minute of overhead per job, so 80 minutes of actual work. Assuming perfect balancing, we can run 10 jobs instead and still retain a wall-time of (80/10 + 1) ~9 minutes per job.
    • the sequential running time of the pytest is ~80 minutes split over 20 jobs. There is roughly one minute of overhead per job, so 60 minutes of actual work. Assuming perfect balancing, we can run 8 jobs instead and still retain a wall-time of (60/8 + 1) ~9 minutes per job.

To note: we're removing all pytests and I expect that we will be finished quite soon (at least, there are open MRs for roughly half of the LOCs left and the remaings LOCs are simple but repetitive tests). In normal times, it would be future-smart to leave some slack, but in this case I think we can allows ourselves to tighten the belt.

Manually testing the MR

Results afterwards.

Tezt: image

So, actually it's as unbalanced as before :(

Pytest: image

looking good 😄

Forgot to run pytest for old protocols :(

Checklist

  • Document the interface of any function added or modified (see the coding guidelines)
  • Document any change to the user interface, including configuration parameters (see node configuration)
  • Provide automatic testing (see the testing guide).
  • For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
  • Select suitable reviewers using the Reviewers field below.
  • Select as Assignee the next person who should take action on that MR
Edited by Arvid Jakobsson

Merge request reports