Skip to content

WIP: Add a cache of parsed and provenanced yaml

Jonathan Mawrequested to merge
jonathan/pickle-yaml into master

Description

This MR is based around adding a cache around loading elements' yaml files. i.e. the yaml parser is quite slow, and it's possible to cache the yaml after provenance decoration as well.

Changes proposed in this merge request:

  • Loader._load_file() in Loader.load() is wrapped with a YamlCache, providing the cache from a single file, and writing it once _load_file() returns.
  • Serialisation is done using python's pickle module, with custom pickler/unpicklers so that I can store the Project object as its name, and look it up on unpickling time.

Outstanding problems:

  • There are no tests yet.
  • It can't actually handle multiple projects at once - currently I restore the entire cache at once, which doesn't work because I can't create all the project objects I need until I've loaded the junction files - My approach here will be to unpickle files on-demand.
  • I need to handle yaml files from junctions, both when in checked-out workspaces, and when they only exist as temporary files.
  • I need to worry about the filename given in the provenance being wrong if projects are moved around / checked out into a temporary dir.

This merge request is part of #466 (closed)

Merge request reports