Skip to content

RPM processing: file parsing and checksum calculation improvements

Oleksandr Chernov requested to merge 386567-checksum-calcullation-optimization into master

What does this MR do and why?

To calculate file checksum we must to read all bytes of file, previously it was done by reading whole file into memory. According to RMP docs the file size is too big for allocate all it bytes in memory at once, so in this MR was implemented stream file reading for checksum calculation.

Test file size: 155.4579052734375 MB
Memory Benchmark
Calculated checksum: 469217e83ac923255d3cdb74c10c28010b89fbaa10058fc6ca041b7c6f8f0686
Stream checksum: 4.99609375 MB
Calculated checksum: 469217e83ac923255d3cdb74c10c28010b89fbaa10058fc6ca041b7c6f8f0686
Full read checksum: 151.81640625 MB
CPU Benchmark
---------------------------------------------
Stream      4.748893   2.403375   7.152268 (  7.223631)
Full read   0.804568   0.088684   0.893252 (  0.900256)
------------------------------------ total: 8.045520sec

Related to #386567 (closed)

Merge request reports