Generate dashboards in parallel
This patch makes dashboards/generate-dashboards.sh
run in parallel, based on the number of available CPUs.
Currently this is one of the main bottlenecks in CI. And it's also really slow to run locally.
By splitting up the scripts and invoking via xargs -n1 -P "$(nproc)"
we should be able to get it to run much faster.
I tested locally with various configurations:
parallel ./dashboards/generate-dashboards.sh 256.89s user 24.31s system 414% cpu 1:07.92 total
xargs -n1 ./dashboards/generate-dashboards.sh 252.01s user 18.39s system 161% cpu 2:47.64 total
xargs -n1 -P8 ./dashboards/generate-dashboards.sh 251.48s user 21.95s system 423% cpu 1:04.55 total
xargs -n1 -P "$(nproc)" # 10 ./dashboards/generate-dashboards.sh 253.29s user 22.60s system 410% cpu 1:07.28 total
We go from 167 seconds wall clock to 67. That's a speedup factor of 2.5x.