WIP: _artifact: use _TempTextBuffer instead of tmp file

Summary:

  • _yaml.dump: allow pass-through of file handle
  • utils: add _TempTextBuffer for avoiding tmp files
  • _artifact: use _TempTextBuffer instead of tmp file
  • utils: indent 'with' continuation line
  • cascache: add_object, assert path if link_directly
  • cascache: refactor, rm some unused exception vars

Detail:

When adding public data to CAS, save on file I/O by passing an in-memory buffer to _cas.add_object() instead of a filename to read. This means that we avoid reading back the file from disk when hashing it, we still only write it to disk once.

This also side-steps a win32 compatability issue, the 'name' member of tempfile.NamedTemporaryFile cannot be opened on Windows NT or later, as per the Python docs: https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile 734aa378

During this, I notice that we default to platform-specific encodings and line endings. In later work we may need to fix up to use utf-8 and \n deliberately, to avoid making unnecessarily platform-specific artefacts.

Add a convenience context manager utils._TempTextBuffer for temporary text buffers, backed by StringIO. This can be used instead of temporary files in some cases, saving on file I/O and compatibility concerns.

The underlying _yaml.roundtrip_dump() for _yaml.dump() allows for the 'file' parameter to be a filename or a file handle. Offer this flexibility with _yaml.dump() too.

Also copy the slightly better documentation for the roundtrip_dump() parameters to dump().

Alternatives considered:

  • Try to re-use the existing open file handle for I/O apart from linking

    • Open the temp file as binary, and use an io.TextIOWrapper to write the yaml. Turn off buffering or be sure to flush().
    • Add support to add_object() for a file-like object. This may make it too complicated.
  • Introduce a new tempfile wrapper, which doesn't open the file, it just returns the filename.

    • Preserve the existing behaviour of write file, read file, link file.

In both alternatives, it feels odd to go memory->disk->memory, caching aside.

Edited by Angelos Evripiotis

Merge request reports

Loading