WIP: _artifact: use _TempTextBuffer instead of tmp file
Summary:
- _yaml.dump: allow pass-through of file handle
- utils: add _TempTextBuffer for avoiding tmp files
- _artifact: use _TempTextBuffer instead of tmp file
- utils: indent 'with' continuation line
- cascache: add_object, assert path if link_directly
- cascache: refactor, rm some unused exception vars
Detail:
When adding public data to CAS, save on file I/O by passing an in-memory buffer to _cas.add_object() instead of a filename to read. This means that we avoid reading back the file from disk when hashing it, we still only write it to disk once.
This also side-steps a win32 compatability issue, the 'name' member of tempfile.NamedTemporaryFile cannot be opened on Windows NT or later, as per the Python docs: https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile 734aa378
During this, I notice that we default to platform-specific encodings and line endings.
In later work we may need to fix up to use utf-8 and \n deliberately,
to avoid making unnecessarily platform-specific artefacts.
Add a convenience context manager utils._TempTextBuffer for temporary text buffers, backed by
StringIO. This can be used instead of temporary files in some cases,
saving on file I/O and compatibility concerns.
The underlying _yaml.roundtrip_dump() for _yaml.dump() allows for the 'file' parameter to be a
filename or a file handle. Offer this flexibility with _yaml.dump() too.
Also copy the slightly better documentation for the roundtrip_dump() parameters to dump().
Alternatives considered:
-
Try to re-use the existing open file handle for I/O apart from linking
- Open the temp file as binary, and use an
io.TextIOWrapperto write the yaml. Turn off buffering or be sure toflush(). - Add support to
add_object()for a file-like object. This may make it too complicated.
- Open the temp file as binary, and use an
-
Introduce a new tempfile wrapper, which doesn't open the file, it just returns the filename.
- Preserve the existing behaviour of write file, read file, link file.
In both alternatives, it feels odd to go memory->disk->memory, caching aside.