Skip to content

Optimize generate-jsonnet-rules script

Gregorius Marco requested to merge mg-optimize-make-generate into master

Since every rule in rules/ thanos-rules/ and thanos-staging-rules/ are independent of each other, we can parallelize each rule generation by forking each of render_multi_jsonnet call.

From profiling each rule generation in !6593 (comment 1661681851), thanos-rules-jsonnet/service-component-alerts.jsonnet is the slowest, so we can also parallelize this rule by splitting them by services.

Performance comparison on my machine (M1 Mac):

TLDR ./scripts/generate-jsonnet-rules.sh took 59s in this MR compared to 150s.

  • On master branch (without parallelization):
$ /usr/bin/time ./scripts/generate-jsonnet-rules.sh
       151.98 real       230.64 user         4.81 sys
  • On this branch with parallelization with 2 processes
$ /usr/bin/time ./scripts/generate-jsonnet-rules.sh
       92.17 real       244.56 user         7.26 sys

Below is no longer applicable, keeping as reference:

Optimizing `thanos-rules-jsonnet/service-component-alerts.jsonnet` ``` # running /usr/bin/time ./scripts/generate-jsonnet-rules.sh # 1 process: (without splitting service-component-alerts.jsonnet) 95.09 real 232.87 user 5.38 sys

2 processes: (splitting service-component-alerts.jsonnet to 2 files)

   59.92 real       234.86 user         5.24 sys

3 processes:

   55.53 real       241.67 user         5.87 sys

4 processes:

   55.28 real       245.04 user         6.30 sys

The time gain from 2 processes and above is negligible, so this MR splits `service-component-alerts.jsonnet` to `service-component-alerts-1.jsonnet` and `service-component-alerts-2.jsonnet`.~~
</details>
Edited by Gregorius Marco

Merge request reports