Improve Jest spec sharding/sequencing in CI

Mark Florianrequested to merge
461073-improve-jest-sharding into master

What does this MR do and why?

Improve Jest spec sharding/sequencing in CI

As discussed in !150196 (comment 1880916097), we were distributing Jest specs to parallel jbos in a round-robin fashion.

This means that related test files were spread across separate jobs, and unrelated specs were run in the same job. This probably worsened the resource usage of Jest jobs, both in terms of memory (Jest/V8 leaks) and compute (transpiling common source files).

That was the theory, anyway. In practice it doesn't seem to make much of a difference in terms of resource usage or test duration! If you're interested in the analysis, open the disclosure under Notes and analysis.

Now, we run related specs (determined by path) in the same job (e.g., ee/analytics/devops_reports and ~/analytics/devops_reports in the same job, and ~/vue_shared in another). We also run specs in path sort order, so that related failures are more likely to be close together. This improves predictability about which specs run in which shards.

Addresses #461073 (closed).

Notes and analysis

Click to open

I created the below charts and data using a very rough script, which you can see in eca96377. Warning! If you decide to run it yourself, please read it thoroughly first. I made a lot of assumptions, and it may do something stupid to your data/system/eyes.

Some notes/observations:

  • Here're lists of the specs each strategy would run on the first shard.
  • This relies on Jest to report memory usage via its --logHeapUsage flag. If that's not accurate, none of this is.
  • I ran this locally. To try to imitate CI as closely as possible, I set --maxWorkers=1, because our runners have two cores, and Jest defaults to the number of the cores available on your machine minus one for the main thread.
  • Again, to imitate CI as closely as possible, I used a prepared Jest build cache using the script from !150196 (merged) before each shard/strategy test run.
  • The Related specs per shard, Jest default sorted strategy climbs in memory usage more quickly than the other two. This makes sense, because by default, Jest runs the slowest (largest) specs first. That they're the largest probably means they're importing the most, increasing various caches.
  • The Original round-robin and Related specs per shard, path sorted strategies show roughly similar memory usage growth, but they diverge eventually (one way or the other) as they run different specs in a given shard.
  • Garbage collection is done at different times, but the maximum memory usage is equivalent across all three strategies.
  • Total test duration is equivalent across all three strategies.
Sharding strategy Total Duration Maximum resident set size (min to max across shards)
Original round-robin 3365.84s 3747 to 4131 MB
Related specs per shard, path sorted 3389.26s 3763 to 4195 MB
Related specs per shard, Jest default sorted 3380.29s 3756 to 4253 MB
Shard Memory usage chart
1 of 12 jest-shard-1-of-12-plot.svg
2 of 12 jest-shard-2-of-12-plot.svg
3 of 12 jest-shard-3-of-12-plot.svg
4 of 12 jest-shard-4-of-12-plot.svg
5 of 12 jest-shard-5-of-12-plot.svg
6 of 12 jest-shard-6-of-12-plot.svg
7 of 12 jest-shard-7-of-12-plot.svg
8 of 12 jest-shard-8-of-12-plot.svg
9 of 12 jest-shard-9-of-12-plot.svg
10 of 12 jest-shard-10-of-12-plot.svg
11 of 12 jest-shard-11-of-12-plot.svg
12 of 12 jest-shard-12-of-12-plot.svg

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Related to #461073 (closed)

Edited by Lukas Eipert

Merge request reports