Skip to content

[MINOR] Optimize SkyfileReader memory usage

Christopher Schinnerlrequested to merge
chris/optimize-addreadbuffer into master

MERGE REQUEST

MR Guidelines

Overview

A memory profile revealed that the 4MiB peek we do for small uploads makes up quite a bit of the allocated memory and is actually doubled by the AddReadBuffer method. This MR improves this with 2 changes.

  1. AddReadBuffer becomes SetReadBuffer which avoids allocating another 4MiB of memory.
  2. The buffer will be cleared as soon as it is completely read.

That way we don't end up with 8MiB of memory which are used for the duration of each large upload. Instead, after uploading the first 4MiB of the file, the GC should release the 4MiB again.

Checklist

Review and complete the checklist to ensure that the MR is complete before assigned to an approver.

  • All new methods or updated methods have clear docstrings
  • Testing added or updated for new methods
  • Any new packages are added to Makefile and .gitlab-ci.yml
  • API documentation updated for API updates
  • Module README.md updated for changes to workflow
  • Issue added to Sia-UI repo for new supporting features
  • Changelog File Created

Issues Closed

Merge request reports