compile-production-assets is on the critical path of Frontend pipelines and has a P50 and P90 duration that ranges from 15 minutes to 28 minutes which indicates that the cache is not rebuilt when needed. Compared to compile-test-assets the P50 to P90 range is 6 to 8 minutes
Is there a more effective way of re-building the cache to close the gap?
Rémy Coutablechanged title from compile-assets-production has large spread in duration percentiles to compile-production-assets has large spread in duration percentiles
changed title from compile-assets-production has large spread in duration percentiles to compile-production-assets has large spread in duration percentiles
Rémy Coutablechanged the descriptionCompare with previous version
Seems like that they have updated a few things related to caching as well. Likely the culprit, if they are doing it in-memory and we are memory constrained...
Ha, it seems like they run prettier on the source no matter if in production or development. Creating an MR to set it based on the environment. Let's see if that speeds things up... !96254 (closed)
Here is the comparison for the updated vue-loader from 15.9.6 to 15.10.0:
vue-loader@15.10.0:
Done in 292.04s.________________________________________________________Executed in 293.08 secs fish external usr time 666.58 secs 0.13 millis 666.58 secs sys time 40.78 secs 5.17 millis 40.78 secs
vue-loader@15.9.6:
✨ Done in 167.69s.________________________________________________________Executed in 168.01 secs fish external usr time 253.74 secs 0.12 millis 253.74 secs sys time 15.37 secs 3.78 millis 15.37 secs
Problem being we probably vue-loader@15.10 for vue 2.7 (if we want to use the new features like <script setup>. UGH. Unfortunately there are no releases and not commits on the repo to easily see what was backported and how.
@vitallium I will create an MR which downgrades vue-loader and see if it still builds...
Some additional tests to confirm the updated vue-loader is the root cause:
vue@2.6.12 and vue-loader@15.10.0:
✨ Done in 192.93s.________________________________________________________Executed in 193.15 secs fish external usr time 487.07 secs 46.00 micros 487.07 secs sys time 29.36 secs 996.00 micros 29.35 secs
vue@2.6.12 and vue-loader@15.9.6:
✨ Done in 147.29s.________________________________________________________Executed in 147.52 secs fish external usr time 212.72 secs 0.05 millis 212.72 secs sys time 15.55 secs 1.11 millis 15.55 secs
@vitallium Looking and comparing to your mretrics above, it seems like there still is a large impact from vue@2.7' compiler though (nearly 200s difference ).
@vitallium Seems like the pipeline for vue@2.7.10 and vue-loader@15.9.6 (only downgrading the loader) is green as well: !96256 (closed). Will have a look at the timings, potentially a good compromise rather than fully downgrading vue as well?
If you do not feel the purpose of this issue matches one of the types, you may apply the typeignore label to exclude it from type tracking metrics and future prompts.
I have also created a test MR to test a matrix of data against each other.
Data matrix expand!
Loader / Vue
ESBUILD_MINIFIER
WEBPACK_REPORT
Run
Cache status
Duration
vue-loader-2.6
FALSE
FALSE
a
Cold
860
vue-loader-2.6
FALSE
FALSE
a
Hot
351
vue-loader-2.6
FALSE
FALSE
b
Cold
827
vue-loader-2.6
FALSE
FALSE
b
Hot
323
vue-loader-2.6
FALSE
FALSE
c
Cold
853
vue-loader-2.6
FALSE
FALSE
c
Hot
351
vue-loader-2.6
FALSE
TRUE
a
Cold
1048
vue-loader-2.6
FALSE
TRUE
a
Hot
554
vue-loader-2.6
FALSE
TRUE
b
Cold
1062
vue-loader-2.6
FALSE
TRUE
b
Hot
559
vue-loader-2.6
FALSE
TRUE
c
Cold
1084
vue-loader-2.6
FALSE
TRUE
c
Hot
560
vue-loader-2.6
TRUE
FALSE
a
Cold
1050
vue-loader-2.6
TRUE
FALSE
a
Hot
371
vue-loader-2.6
TRUE
FALSE
b
Cold
447
vue-loader-2.6
TRUE
FALSE
b
Hot
416
vue-loader-2.6
TRUE
FALSE
c
Cold
1262
vue-loader-2.6
TRUE
FALSE
c
Hot
393
vue-loader-2.6
TRUE
TRUE
a
Cold
670
vue-loader-2.6
TRUE
TRUE
a
Hot
665
vue-loader-2.6
TRUE
TRUE
b
Cold
732
vue-loader-2.6
TRUE
TRUE
b
Hot
851
vue-loader-2.6
TRUE
TRUE
c
Cold
1103
vue-loader-2.6
TRUE
TRUE
c
Hot
619
vue-loader-2.7
FALSE
FALSE
a
Cold
928
vue-loader-2.7
FALSE
FALSE
a
Hot
391
vue-loader-2.7
FALSE
FALSE
b
Cold
991
vue-loader-2.7
FALSE
FALSE
b
Hot
402
vue-loader-2.7
FALSE
FALSE
c
Cold
892
vue-loader-2.7
FALSE
FALSE
c
Hot
362
vue-loader-2.7
FALSE
TRUE
a
Cold
1089
vue-loader-2.7
FALSE
TRUE
a
Hot
553
vue-loader-2.7
FALSE
TRUE
b
Cold
1033
vue-loader-2.7
FALSE
TRUE
b
Hot
529
vue-loader-2.7
FALSE
TRUE
c
Cold
1089
vue-loader-2.7
FALSE
TRUE
c
Hot
562
vue-loader-2.7
TRUE
FALSE
a
Cold
543
vue-loader-2.7
TRUE
FALSE
a
Hot
600
vue-loader-2.7
TRUE
FALSE
b
Cold
529
vue-loader-2.7
TRUE
FALSE
b
Hot
442
vue-loader-2.7
TRUE
FALSE
c
Cold
486
vue-loader-2.7
TRUE
FALSE
c
Hot
581
vue-loader-2.7
TRUE
TRUE
a
Cold
694
vue-loader-2.7
TRUE
TRUE
a
Hot
768
vue-loader-2.7
TRUE
TRUE
b
Cold
712
vue-loader-2.7
TRUE
TRUE
b
Hot
865
vue-loader-2.7
TRUE
TRUE
c
Cold
1396
vue-loader-2.7
TRUE
TRUE
c
Hot
568
and came to the following conclusions:
If you copy the table from the description into e.g. Google Sheets or any kind of table filtering tool, you might be able to follow the following conclusions:
terserreally loves it's cache. If the cache is there, the minification time goes down drastically (e.g. 860 => 351, 1048 => 554, 991 => 402, 1089 => 553). And that's true no matter the other parameters: webpack report (WR), vue version.
esbuild minifier get's wildy variying results:
cold cache time with WR=off,vue@2.6 are 1050, 447, 1262. However if the cache is hot, it's 371, 416 and 393.
cold cache time with WR=off,vue@2.7 are 543, 529, 486. Hot cache takes even longer with 600, 442, 581.
Hard to draw a conclusion here, but maybeesbuild-minifier works better with vue-loader@15.10 because it is emitting the compiled vue components somehow differently? Even seems like we could disable caching here. But one conclusion: definitely more testing before we move to esbuild-minifier.
WEBPACK_REPORTeasily takes up to 200s more: 860 => 1048, 928 => 1089, 351 => 554, 391 => 553. So moving it out of our normal asset_compile makes sense.
vue@2.6 vs vue 2.7: With cold cache and terser it seems to take about 100s longer... Other than that there is no real conclusion to draw. As long as the WEBPACK_REPORT is disabled timings are otherwise comparable. I think we really should inestigte here, especially in a combo with the esbuild minifier.
So far I don't see any problems reported, but I've checked a few random jobs, and it seems to work well for backend and speed up the compile-*-assets jobs for these!
I've disabled the new strategy due to an issue that happens when the new cache package doesn't exist but master cache is present. In that case, it looks like we use the master cache without checking its version. I created #375703 (closed) to track this.
compile-production-assets is on the critical path of Frontend pipelines and has a P50 and P90 duration that ranges from 15 minutes to 28 minutes which indicates that the cache is not rebuilt when needed.