Skip to content

Cache assets in gitlab:assets:compile

Rémy Coutable requested to merge speedup-gitlab-assets-compile into master

What does this MR do?

This adds a cache directive to the gitlab:assets:compile and compile-assets jobs.

This improvement is based on omnibus-gitlab#2424 (comment 43074206).

Following are four different caching experiments. The second and fourth tries are mostly the same, and I could track the cache ballooning to https://github.com/rails/sprockets/issues/563.

Fifth try by caching tmp/cache/assets/sprockets and monkey-patching Sprockets with https://github.com/rails/sprockets/compare/3.x...jmreid:no-mtime-for-digest-key

compile-assets

  1. 1st compile-assets run (11 minutes 16 seconds):
    • rake:assets:precompile finished in 72.26437519 seconds
    • tmp/cache/assets/sprockets/ is currently 132M
  2. 2nd compile-assets run (5 minutes 20 seconds => 2.11x faster):
    • rake:assets:precompile finished in 6.456429399 seconds (11.2x faster)
    • tmp/cache/assets/sprockets/ is currently 132M (same size)
  3. 3rd compile-assets run (5 minutes 35 seconds):
    • rake:assets:precompile finished in 6.780493447 seconds
    • tmp/cache/assets/sprockets/ is currently 132M (same size)

Conclusion: With the Sprockets monkey-patch suggested in https://github.com/rails/sprockets/issues/563, the Sprockets assets precompilation speed is increased, with no ballooning effect since the mtime isn't used in the cache key.

gitlab:assets:compile

  1. 1st gitlab:assets:compile run (23 minutes 55 seconds):
    • rake:assets:precompile finished in 178.490804189 seconds
    • tmp/cache/assets/sprockets/ is currently 131M
  2. 2nd gitlab:assets:compile run (15 minutes 9 seconds => 1.57x faster):
    • rake:assets:precompile finished in 8.583158472 seconds (20.8x faster)
    • tmp/cache/assets/sprockets/ is currently 131M (same size)
  3. 3rd gitlab:assets:compile run (14 minutes 0 seconds):
    • rake:assets:precompile finished in 6.766899414 seconds
    • tmp/cache/assets/sprockets/ is currently 131M (same size)

Conclusion: With the Sprockets monkey-patch suggested in https://github.com/rails/sprockets/issues/563, the Sprockets assets precompilation speed is increased, with no ballooning effect since the mtime isn't used in the cache key.

Fourth try by caching tmp/cache/assets/sprockets

### `compile-assets`
  1. 1st compile-assets run:
    • rake:assets:precompile finished in 73.791743339 seconds
    • tmp/cache/assets/sprockets/ is currently 132MB
  2. 2nd compile-assets run
    • rake:assets:precompile finished in 5.64133392 seconds (13.17x faster)
    • tmp/cache/assets/sprockets/ is currently 152MB (+ 20MB for the exact same SHA/assets)
  3. 3rd compile-assets run
    • rake:assets:precompile finished in 5.78727085 seconds (no improvement)
    • tmp/cache/assets/sprockets/ is currently 172MB (+ 20MB for the exact same SHA/assets)

Conclusion: The cache is improving the Sprockets precompilation speed, but it's also ballooning here, even if the assets didn't change!

gitlab:assets:compile

  1. 1st gitlab:assets:compile run:
    • rake:assets:precompile finished in 190.122873861 seconds
    • tmp/cache/assets/sprockets/ is currently 131MB
  2. 2nd gitlab:assets:compile run
    • rake:assets:precompile finished in 6.873365871 seconds (27.67x faster)
    • tmp/cache/assets/sprockets/ is currently 151MB` (+ 20MB for the exact same SHA/assets)

Conclusion: The cache is improving the Sprockets precompilation speed, but it's also ballooning here, even if the assets didn't change!

Third try by caching public/assets minus public/assets/webpack (Sprockets compiled assets)

### `compile-assets`
  1. 1st compile-assets run:
    • rake:assets:precompile finished in 74.44845465 seconds
    • sprockets-assets/ is currently 58MB
  2. 2nd compile-assets run:
    • rake:assets:precompile finished in 69.041837927 seconds (10% faster)
    • sprockets-assets/ is currently 58MB (same size)
  3. 3rd compile-assets run:
    • rake:assets:precompile finished in 69.830209646 seconds (no improvement)
    • sprockets-assets/ is currently 58MB (same size)

Conclusion: Cache doesn't balloon but rake:assets:precompile isn't faster... 🤔

gitlab:assets:compile

  1. 1st gitlab:assets:compile run:
    • rake:assets:precompile finished in 189.936003268 seconds
    • sprockets-assets/ is currently 52MB
  2. 2nd gitlab:assets:compile run:
    • rake:assets:precompile finished in 190.181387912 seconds (no improvement)
    • sprockets-assets/ is currently 52MB (same size)
  3. 3rd gitlab:assets:compile run:
    • rake:assets:precompile finished in 183.757361696 seconds (4% faster)
    • sprockets-assets/ is currently 52MB (same size)

Conclusion: Cache doesn't balloon but rake:assets:precompile isn't faster... 🤔

Second try by caching tmp/cache/assets (Sprockets cache folder)

### `compile-assets`
  1. 1st compile-assets run:
    • rake:assets:precompile finished in 74.55412116 seconds
    • tmp/cache/assets/ is currently 131MB
  2. 2nd compile-assets run:
    • rake:assets:precompile finished in 5.811031225 seconds (14.8x faster)
    • tmp/cache/assets/ is currently 152MB (+ 21MB for the exact same SHA/assets)
  3. 3rd compile-assets run:
    • rake:assets:precompile finished in 6.313982111 seconds (11.7x faster)
    • tmp/cache/assets/ is currently 172MB (+ 20MB for the exact same SHA/assets)
  4. 4th compile-assets run:
    • rake:assets:precompile finished in 6.51208368 seconds (11.3x faster)
    • tmp/cache/assets/ is currently 192MB (+ 20MB for the exact same SHA/assets)

Conclusion: The cache is ballooning here, even if the assets didn't change!

gitlab:assets:compile

  1. 1st gitlab:assets:compile run:
    • rake:assets:precompile finished in 161.426688669 seconds
    • tmp/cache/assets/ is currently 131MB
  2. 2nd gitlab:assets:compile run:
    • rake:assets:precompile finished in 9.140452779 seconds (17.6x faster)
    • tmp/cache/assets/ is currently 151MB (+ 20MB for the exact same SHA/assets)
  3. 3rd gitlab:assets:compile run:
    • rake:assets:precompile finished in 9.759735945 seconds (16.5x faster)
    • tmp/cache/assets/ is currently 171MB (+ 20MB for the exact same SHA/assets)

Conclusion: The cache is ballooning here, even if the assets didn't change!

First try below by caching tmp/cache, and public/assets

### `compile-assets`

  1. First run with empty cache (https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149256635). Total time: 11 minutes:

    • yarn:check finished in 21 seconds
    • gettext:po_to_json finished in 8 seconds
    • rake:assets:precompile finished in 68 seconds
    • webpack:compile finished in 46 seconds

  2. Second run with warm cache (same SHA) (https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149263689). Total time: 9 minutes:

    • yarn:check finished in 26 seconds
    • gettext:po_to_json finished in 7 seconds
    • rake:assets:precompile finished in 3 seconds
    • webpack:compile finished in 33 seconds

gitlab:assets:compile

  1. First run with empty cache (https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/148904529). Total time: 22 minutes:

    • webpack:compile finished in 455 seconds

  2. Second run with warm cache (same SHA) (https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149085002). Total time: 12 minutes.

    • webpack:compile finished in 131 seconds (3.45x faster!) 🚀

  3. Third run with new CompressionPlugin({ cache: true }) (https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149091616). Total time: 15 minutes:

    • webpack:compile finished in 135 seconds => this doesn't seem to be significant.

What are the relevant issue numbers?

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/56538.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/56691, and maybe https://gitlab.com/gitlab-org/gitlab-ce/issues/35305?

Does this MR meet the acceptance criteria?

Edited by Rémy Coutable

Merge request reports