Skip to content

Resolve "WAL analysis runs out of memory with very large files"

Jon Jenkins requested to merge 1-lazy-read into main

Closes #1 (closed)

Memory Improvements

Start by profiling:

def self.analyze(arguments)
    require 'memory_profiler'
    MemoryProfiler.start(top:5)
    lines = Parser.parse_file(arguments.input_file)
    mem_report = MemoryProfiler.stop
    ...
    mem_report.pretty_print
end

The objective is to see how much memory lines uses.

cat spec/fixtures/wal_sample.txt | ruby wal_analysis.rb --operation-argument=10

And get:

Total allocated: 21615991 bytes (282836 objects)
Total retained:  2910913 bytes (60383 objects)

Then slowly turn up the heat and see where the memory goes by running:

cat wal_sample_big.txt > temp; cat temp >> wal_sample_big.txt; rm temp # copy the fixture to wal_sample_big.txt first!

When we have approx. 8MB of WAL we get these numbers:

Total allocated: 86537929 bytes (1131275 objects)
Total retained:  11722609 bytes (241499 objects)

From what I can surmise, we have a linear increase in memory usage as the amount of WAL lines increases, which is roughly to be expected.

Since we are loading our dataset entirely into memory before processing, naturally there is a limitation over how much data we can process at once.

A lazy loader would solve this problem.

First, lets get the output and memory usage for 64mb of WAL:

---------------------------------------------------
vulnerability_reads   | 8696.0828 KB/s | 1133.7774 MB/s
All unknown relations | 7961.0169 KB/s | 164.0563 MB/s 
merge_request_diffs   | 6157.864 KB/s  | 106.7364 MB/s 
projects              | 2092.4108 KB/s | 29.8892 MB/s  
users                 | 711.7877 KB/s  | 24.3869 MB/s  
project_statistics    | 190.1178 KB/s  | 4248.2157 KB/s
namespaces            | 309.0416 KB/s  | 1739.6754 KB/s
keys                  | 48.1876 KB/s   | 1523.9567 KB/s
routes                | 173.8645 KB/s  | 1430.9771 KB/s
packages_packages     | 122.7773 KB/s  | 1400.34 KB/s  

Total allocated: 692069233 bytes (9050039 objects)
Total retained:  93558321 bytes (1931915 objects)

It required about 94 megs.

Now, let's add in the lazy parser, as well as profile the entire program:

name                  | wal_rate_base  | wal_rate_total
---------------------------------------------------
vulnerability_reads   | 8696.0828 KB/s | 1133.7774 MB/s
All unknown relations | 7961.0169 KB/s | 164.0563 MB/s 
merge_request_diffs   | 6157.864 KB/s  | 106.7364 MB/s 
projects              | 2092.4108 KB/s | 29.8892 MB/s  
users                 | 711.7877 KB/s  | 24.3869 MB/s  
project_statistics    | 190.1178 KB/s  | 4248.2157 KB/s
namespaces            | 309.0416 KB/s  | 1739.6754 KB/s
keys                  | 48.1876 KB/s   | 1523.9567 KB/s
routes                | 173.8645 KB/s  | 1430.9771 KB/s
packages_packages     | 122.7773 KB/s  | 1400.34 KB/s  
Total allocated: 841766521 bytes (11474765 objects)
Total retained:  6297 bytes (16 objects)

These numbers are much more acceptable, and demonstrate that the data is correct for both LazyParser and regular parser.

Through experimental verification on our benchmarking server, I can see that the process has been running for over two hours and only 90 megabytes of memory has been used!

Edited by Jon Jenkins

Merge request reports