Add documentation for parallel keyword about how to split a file
Problem to solve
The parallel keyword is missing a generic example of how you might split a file into multiple parts, causing some people maybe to not realize what's possible there.
Further details
From the docs today:
parallel allows you to configure how many instances of a job to run in parallel. This value has to be greater than or equal to two (2) and less than or equal to 50.
This creates N instances of the same job that run in parallel. They’re named sequentially from job_name 1/N to job_name N/N.
For every job, CI_NODE_INDEX and CI_NODE_TOTAL environment variables are set.
A simple example:
test: script: rspec parallel: 5
Proposal
Add an actual script example that splits a file into a list of somethings and runs something on each one (this example is not tested):
process_files:
parallel: 5
script:
- find . -name "*.png" >to_process.txt
- split -l$((`wc -l < to_process.txt`/$CI_NODE_TOTAL)) to_process.txt
- cat to_process.txt$CI_NODE_INDEX |xargs do_processing
Who can address the issue
Other links/references
Edited by Jason Yavorska