Bump version of fd.o SDK and bst_external in overnight tests

See merge request !962

Docs: Add remote execution architecture documentation.

See merge request !923

Avoid hanging artifact cache tests

See merge request !964

This eliminates unneeded copies of the project directory for junctions
with a single local source.

Optimization for local junctions

See merge request !290

This improves readability.

This adds the batch() context manager.

This allows the sandbox to output messages using the plugin id.

This allows batching of all sandbox commands aross prepare() and
assemble().

Execute batched commands in a single shell script.

Command batching

Closes #675

See merge request !915

There is nothing in ArtifactCacheSpec that's actually specific to
artifacts, so I've made it a CAS class so we can use the same
spec for remote execution.

Since the artifact cache and remote execution share the same
local CAS store, they should share the same CASCache object. Moving
this into context allows us to do this.

Uses the new config options introduced earlier to specify remote
exec storage and execution service.

Tests that we get a sensible error message when we specify incomplete
or wrong information in the configuration for remote execution.

The port number must come after the hostname and before the path.
The path is ignored by CASCache anyway, so including it is misleading.

Split remote execution from artifact cache

Closes #750

See merge request !946

tmp_path was actually introduced in the 3.9.x series,
but it slipped by when the pytest version was bumped before
by pip opting to fetch a version which was newer than necessary.

This commit introduces BST_STAGE_INTEGRATES as a compatibility flag,
so Elements can set it to false
to signal that integrate_dependency_artifacts should be called.

This will be needed to implement staging multiple elements,
since the integration commands for all elements need to be called
after all dependencies are staged.

This ensures Element.integrate_dependency_artifacts is called after stage
when BST_STAGE_INTEGRATES hasn't been set to False.

This takes the liberty of splitting the layout handling code paths
into separate methods to reduce the amount of nested conditionals
with the aim of improving readability.

This is mostly just marking which elements work as expected
from changes to their base class.

Junction and Filter elements expect overriding stage to be sufficient
to prevent it doing anything in those contexts,
and since we're not intending to deprecate BST_STAGE_INTEGRATES elements
there's no harm in them continuing to do so
when they aren't expected to work in contexts we require
BST_STAGE_INTEGRATES to be False.

There's redundancy in pre-staging sandbox creation logic
with Element._sandbox doing something similar,
but hopefully not too much.