Skip to content

Update parallel_runner implementation

Andrejs Cunskis requested to merge andrey-parallel-runner-refactor into master

What does this MR do and why?

Update parallel_runner implementation to use parallel_tests api directly instead of spawning additional process layer. This cleans the initial implementation quite a bit and removes all the additional variable handling that came with spawning additional process layer.

There are quite a few issues to address before it can actually be enabled:

  • Run global hooks once before spawning parallel runs and skip it in the forked processes. Hooks perform some global updates to the instance so we need to minimise chance of collision and just save time
  • Move default report opts set via cli in https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/base.gitlab-ci.yml?ref_type=heads#L127 to runner.rb so it is possible to add pid number to output file name. Otherwise reports will be overwriting each other
  • Do a bit more testing on how many parallel forks omnibus instance can handle. Depending on what tests end up being executed and how many api requests are made, 8 parallel forks can run in to rate limit: https://gitlab.com/gitlab-org/gitlab/-/jobs/7312017767#L2120
  • Make sure no tests are skipped. parallel_run + knapsack open source is rather custom and not really designed to run together, so need to be 100% sure that all the right tests end up being executed
  • Junit report merging when there is a retry report needs to be updated as well. Current implementation operates on assumption that there are only 2 report files

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Andrejs Cunskis

Merge request reports