YAML New World Order
Description
** IMPORANT - READ THIS FIRST **
This MR replaces the old YAML handling with a new lower level YAML data structure. As a result, provenance is cheaper to handle, and YAML is, in general, faster to process. Unfortunately the intricacy of the code has increased as a side-effect.
The majority of the changes are in _yaml.py and it may be easier to review that file
as though it were a new file rather than by reading the diff in the UI. Everything else
should be fairly mechanical reworks.
There are a number of new APIs in Plugin which need checking carefully.
Performance numbers:
| measurement | master (uncached) | master (cached) | this MR |
|---|---|---|---|
Loading elements (loader.load()) |
314s | 43s | 31s |
Resolving elements (_new_from_meta) |
65s | 62s | 45s |
Resolving cached state (_update_state) |
45s | 41s | 39s |
Total runtime of bst show debian-stack.bst
|
522s | 247s | 159s |
| Peak memory consumption of that command | 6.2G | 6.6G | 3.9G |
All tests were run on a Lenovo T470p in a VM with two cores of i7 @ 4GHz and 16G of DDR4 RAM.
All tests had a hot disk cache (YAML was in RAM) and outliers were discarded from several runs
Values above are averages, though variance was on the order of ±3s
Some further benchmarks
Below are some benchmarks for the Debian-like project with the 'remote' file source imports.
- Note that master was at: bschubert/optimize-dependencies - 4930e3f0
- shared/yaml-rework (this branch) was rebased on top of this at the time of benchmarking.
- !1067 (merged) was a good place for the initial benchmark as it was from the date of this patch (16/01/2019) that we started focusing strongly on performance
These benchmarks were run on a Lenovo ThinkPad X230, very lightly loaded, 8G RAM, 4 cores, 500GB HDD.
| ----- | Time (s) | ----- | ----- | Max-rss (M) | ----- | |
|---|---|---|---|---|---|---|
| Pre-!1067 | Master | Shared/yaml-rework | Pre-!1067 | Master | Shared/yaml-rework | |
| Show | 51.14 | 42.79 | 11.74 | 729 | 449 | 321 |
| Show with cache | 27.86 | 17.34 | N/A | 754 | 470 | N/A |
| Show once Built | 34.03 | 23.93 | 12.20 | 757 | 602 | 471 |
| Build | 51:42 MINUTES | 9:47 MINUTES | 9:26 MINUTES | 759 | 470 | 320 |
Closes #591 (closed)