Skip to content

Generate dashboards in parallel

Igor requested to merge generate-dashboards-parallel into master

This patch makes dashboards/generate-dashboards.sh run in parallel, based on the number of available CPUs.

Currently this is one of the main bottlenecks in CI. And it's also really slow to run locally.

By splitting up the scripts and invoking via xargs -n1 -P "$(nproc)" we should be able to get it to run much faster.

I tested locally with various configurations:

parallel                           ./dashboards/generate-dashboards.sh  256.89s user 24.31s system 414% cpu 1:07.92 total

xargs -n1                          ./dashboards/generate-dashboards.sh  252.01s user 18.39s system 161% cpu 2:47.64 total

xargs -n1 -P8                      ./dashboards/generate-dashboards.sh  251.48s user 21.95s system 423% cpu 1:04.55 total

xargs -n1 -P "$(nproc)" # 10       ./dashboards/generate-dashboards.sh  253.29s user 22.60s system 410% cpu 1:07.28 total

We go from 167 seconds wall clock to 67. That's a speedup factor of 2.5x.

Merge request reports