Docs: Add remote execution architecture documentation.

Remote execution

================

Remote execution allows builds to take place on different machines from the machine *bst* is run on, allowing faster builds, shared resources and different build capabilities, for example different machine architectures.

Sandbox extension

~~~~~~~~~~~~~~~~~

The previous section :ref:`sandboxing` describes the two forms of local sandbox: the *bubblewrap* sandbox and the less common *chroot* sandbox (which is used on non-Linux POSIX environments). Remote execution uses a third type, the *remote sandbox*, which functions similarly to the local sandbox types, but is responsible for causing the build to occur on a remote system. Remote sandboxes should produce the same result as local sandboxes.

Artifact caches and other caches

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

BuildStream can transmit the results of local builds to remote artifact caches and retrieve them later. The same mechanism is used for remote execution: in this case, the filesystem staged before building is stored in the local artifact cache, and the command to execute is also stored as an object in this cache. The artifact push mechanism is used to send these objects to the remote execution cache, and the artifact pull method will be used to retrieve the finished filesystem after the build. The remote execution service uses the same communication protocol as artifact caches, and may use the same internal storage, but may not implement the extensions used by BuildStream to store full artifacts.

.. image:: images/arch-remote-execution.svg

   :align: center

After sending the prerequisite file system and command to the remote execution cache, BuildStream uses the *Remote Execution API (REAPI)* [1]_ to signal to the build server that it should perform a build. How the build server does this is not BuildStream's concern, but typically it will start a remote worker on a machine under its control which will access the remote execution cache using a FUSE layer. The remote execution server is entitled to return a cached result, if the filesystem and command have already been executed.

BuildStream will continue to poll the remote execution server until the build is completed or lost. If it's completed (successfully or otherwise) the resulting objects (typically the finished file system and logs of stdout and stderr) will be pulled to the local cache. BuildStream will retry jobs that are lost by the remote build server or which complete with certain error types.

*sandboxremote.py* contains all the communication with the remote execution API.

After a successful build, BuildStream will push the completed artifact to the remote artifact servers as if it had built it locally.

Use of sandboxes outside builds

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Because staging requires *integration-commands* to be run, a sandbox is necessary for the *bst checkout* command to run.

A sandbox is also required for the *bst shell* command. Because the REAPI does not provide any mechanism for interactive use, *bst shell* will always use a local sandbox.

.. [1] See https://github.com/bazelbuild/remote-apis.

   arch_scheduler

   arch_cachekeys

   arch_sandboxing

   arch_remote_execution