YAML New World Order
** IMPORANT - READ THIS FIRST **
This MR replaces the old YAML handling with a new lower level YAML data structure. As a result, provenance is cheaper to handle, and YAML is, in general, faster to process. Unfortunately the intricacy of the code has increased as a side-effect.
The majority of the changes are in
_yaml.py and it may be easier to review that file
as though it were a new file rather than by reading the diff in the UI. Everything else
should be fairly mechanical reworks.
There are a number of new APIs in Plugin which need checking carefully.
|measurement||master (uncached)||master (cached)||this MR|
|Loading elements (
|Resolving elements (
|Resolving cached state (
|Total runtime of
|Peak memory consumption of that command||6.2G||6.6G||3.9G|
All tests were run on a Lenovo T470p in a VM with two cores of i7 @ 4GHz and 16G of DDR4 RAM.
All tests had a hot disk cache (YAML was in RAM) and outliers were discarded from several runs
Values above are averages, though variance was on the order of ±3s
Some further benchmarks
Below are some benchmarks for the Debian-like project with the 'remote' file source imports.
- Note that master was at: bschubert/optimize-dependencies - 4930e3f0
- shared/yaml-rework (this branch) was rebased on top of this at the time of benchmarking.
- !1067 (merged) was a good place for the initial benchmark as it was from the date of this patch (16/01/2019) that we started focusing strongly on performance
These benchmarks were run on a Lenovo ThinkPad X230, very lightly loaded, 8G RAM, 4 cores, 500GB HDD.
|-----||Time (s)||-----||-----||Max-rss (M)||-----|
|Show with cache||27.86||17.34||N/A||754||470||N/A|
|Show once Built||34.03||23.93||12.20||757||602||471|
|Build||51:42 MINUTES||9:47 MINUTES||9:26 MINUTES||759||470||320|
Closes #591 (closed)