Automatically split list of tests in parallel
Problem to solve
Currently, we have the parallel which is great and gives control and flexibility to set up parallel tests but this requires a lot of setup from a developer and sometimes the split logic just get duplicated.
Intended users
Further details
For example for GitLab Runner we use the parallel
feature by first creating a test definitions in a txt file and then split that file depending on CI_NODE_TOTAL
& CI_NODE_INDEX
.
Knapsack does something similar and it's possible (I'm not going to say otherwise before I don't have any data on this) others are doing the same thing.
Proposal
Do something similar to what is proposed in https://gitlab.com/gitlab-org/gitlab-ce/issues/3819#note_12287609 and inspired from CricleCI test splitting we can create the following convention:
- The user creates a $JOB_NAME.txt inside of the
.gitlab
dir - When the
parallel
keyword is present for a job we check.gitlab/$JOB_NAME.txt
and split it automatically - Each line will be a test we have to found out what is the best mean to send each test of pkg define in the line to the user, we can do this via env variables or something similar
Convention of the txt file
- It doesn't have to plain text, we can do json if we want
- Each line will be a test/package or something that the user will run
Permissions and Security
Documentation
Giving an example on how to set up something like this for different language/testing frameworks
What does success look like, and how can we measure that?
Developers just have to set up the parallel
keyword and define the command that will run inside of a loop to run the tests.
Links / references
https://gitlab.com/gitlab-org/gitlab-ce/issues/3819#note_12287609