Docs: Add remote execution architecture documentation.

See merge request !923

Avoid hanging artifact cache tests

See merge request !964

This eliminates unneeded copies of the project directory for junctions
with a single local source.

Optimization for local junctions

See merge request !290

This improves readability.

This adds the batch() context manager.

This allows the sandbox to output messages using the plugin id.

This allows batching of all sandbox commands aross prepare() and
assemble().

Execute batched commands in a single shell script.

Command batching

Closes #675

See merge request !915

There is nothing in ArtifactCacheSpec that's actually specific to
artifacts, so I've made it a CAS class so we can use the same
spec for remote execution.

Since the artifact cache and remote execution share the same
local CAS store, they should share the same CASCache object. Moving
this into context allows us to do this.

Uses the new config options introduced earlier to specify remote
exec storage and execution service.

Tests that we get a sensible error message when we specify incomplete
or wrong information in the configuration for remote execution.

The port number must come after the hostname and before the path.
The path is ignored by CASCache anyway, so including it is misleading.

Split remote execution from artifact cache

Closes #750

See merge request !946

setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52

See merge request !975

f_bfree space might not be usable. In practice we see failures in big
disks because f_bfree is over 2GB and f_bavail is 0. We get ENOSPC if
writing on disk then.

We plan to make cache incomplete. That is some blobs are missing.  For
most of cases we will delete references when requested if they are
incomplete. But there will be corner cases where objects are removed
after the reference is requested.

The file is already a temporary file and does not need copy.  ENOSPC
is thrown during that copy in issue #609.

Fixes #678.

This locks the temporary object file so that cleanup does not need to
be done for every write.

This also remove references when some objects are missing. This is in
preparation for the move from reference to object garbage collection.

When there is less than 2GB left, it cleans up have 10GB available.
These values are configurable.

Cleaning up in parallel might slow down the cleaning process

Fix cleanup of cache in server when disk is full

Closes #678

See merge request !830

push_message and push_directory will both raise assertion failures if
they fail to send the digest to the server. Checking the digest
returned by each only tests that the content was hashed locally, which
cannot reasonably fail.

_sandboxremote.py: Remove unnecessary tests.

Closes #786

See merge request !976

Fixes #723.

Add support for .netrc in remote/tar/zip plugins

Closes #723

See merge request !908

_loader/loader.py: Added Suffix check and warning for named elements and add helper function

completions.py: Add test for fail on invalid suffix
Added required files for testing
integration/source-determinism.py: renamed test elements to end with .bst