Split remote execution from artifact cache

Description

Currently remote execution can only specify one URL, which is used for the REAPI server (usually BuildGrid), and remote storage has to be the same as the artifact cache. This MR splits those up, so you can specify another CAS remote, and separately specify the endpoint for the execution service.

You can still specify the same server for your artifact cache and remote execution store if you want.

At the moment, local RE CAS storage is still in the same location as artifact cache storage, it's just the remotes that differ.

Proposed changes

Changes proposed in this merge request:

• Change config for remote execution so it can accept separate configuration for the execution server and storage service.
• Create our own CASCache objects in SandboxRemote instead of using the artifact cache's objects.
• Add documentation for the new config options
• Add tests for new config options
• Remove the methods in ArtifactCache which are only necessary for remote execution.

I'm not proposing to add a full CI remote execution test, but I will add some tests which check the new config option behaves sensibly.

This merge request, when approved, will close: #750 (closed)

---

Create our own CASCache objects in SandboxRemote instead of using the artifact cache's objects.

I'm wondering, as the same local cache directory is used, does it really make sense to use a separate CASCache instance? This might be problematic e.g. if we keep track of the size of the local cache directory in an instance variable of CASCache (although, that would also be an issue with multiple bst instances).

We'd have to change ownership of the CASCache from ArtifactCache to probably Context, however, that's likely straight forward.

Copying comment from !922 (merged) so that we don't forget about it: The following methods recently added to ArtifactCache can likely be removed when SandboxRemote is modified to use a separately configured CAS remote for remote execution instead of the configured artifact servers: pull_tree(), push_directory(), push_message(), verify_digest_on_remote().

changed the description

mentioned in merge request !952 (merged)

added 74 commits

• 3264148d...daca0c01 - 70 commits from branch master
• 64993e89 - artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
• bd144058 - _sandboxremote: Remove references to artifact cache.
• 36ca2b4b - Documentation: Update to show new config options
• c7f8bca7 - tests/sandboxes/remote-exec-config.py: New file.

Compare with previous version

added 2 commits

• 0db712ec - Documentation: Update to show new config options
• 1b2f602a - tests/sandboxes/remote-exec-config.py: New file.

Compare with previous version

marked the checklist item Add tests for new config options as completed

added 12 commits

• 1b2f602a...4d6e2cd3 - 6 commits from branch master
• af1f35f4 - artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
• cff1826b - artifactcache.py, _context.py: Move CASCache object into context
• 2cd02840 - _sandboxremote: Remove references to artifact cache.
• 1a11faf7 - Documentation: Update to show new config options
• 006a121b - tests/sandboxes/remote-exec-config.py: New file.
• 6b2ceaac - artifactcache.py: Remove functions previously used by remote execution.

Compare with previous version

marked the checklist item Remove the methods in ArtifactCache which are only necessary for remote execution. as completed

resolved all discussions

assigned to @jmacarthur

unmarked as a Work In Progress

changed the description

approved this merge request

mentioned in merge request !950 (closed)

added 8 commits

• 6b2ceaac...abef70fe - 2 commits from branch master
• f280dbe7 - artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
• d619f105 - artifactcache.py, _context.py: Move CASCache object into context
• 75352808 - _sandboxremote: Remove references to artifact cache.
• 125e3bf0 - Documentation: Update to show new config options
• 5edecc4f - tests/sandboxes/remote-exec-config.py: New file.
• a234b224 - artifactcache.py: Remove unused verify_digest_pushed

Compare with previous version

mentioned in issue #781

added 7 commits

• b42defc0 - artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
• 1abea743 - artifactcache.py, _context.py: Move CASCache object into context
• 094d3185 - _sandboxremote: Remove references to artifact cache.
• 7208078a - Documentation: Update to show new config options
• b13754f9 - tests/sandboxes/remote-exec-config.py: New file.
• 835f403a - artifactcache.py: Remove unused verify_digest_pushed
• 494fcf7b - _sandboxremote.py: Remove print statements and correct some mistakes

Compare with previous version

added 4 commits

• cdcfc0d2 - tests/sandboxes/remote-exec-config.py: New test.
• 46aa0237 - artifactcache.py: Remove unused verify_digest_pushed
• 20cfc39b - _sandboxremote.py: Remove print statements and correct some mistakes
• d980a5c3 - _sandboxremote: pep8

Compare with previous version

added 4 commits

• 85ff6cff - _sandboxremote: Remove references to artifact cache.
• 9abbbae8 - Documentation: Update to show new config options
• 045c385c - tests/sandboxes/remote-exec-config.py: New test.
• 0d4d709a - artifactcache.py: Remove unused verify_digest_pushed

Compare with previous version

mentioned in issue #785

added 3 commits

• ce98e3f6 - Documentation: Update to show new config options
• 392c2803 - tests/sandboxes/remote-exec-config.py: New test.
• 26c3e97f - artifactcache.py: Remove unused verify_digest_pushed

Compare with previous version

added 11 commits

• 26c3e97f...d09a18b8 - 4 commits from branch master
• a4dc6786 - artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
• 0325afba - artifactcache.py, _context.py: Move CASCache object into context
• 183587fb - _sandboxremote: Remove references to artifact cache.
• 3bce4ed0 - Documentation: Update to show new config options
• 1bb4510d - tests/sandboxes/remote-exec-config.py: New test.
• 94f82693 - artifactcache.py: Remove unused verify_digest_pushed
• 8afcfb9d - Docs: Correct URLs in artifact cache example

Compare with previous version

resolved all discussions

added 28 commits

• 8afcfb9d...127d332f - 21 commits from branch master
• d5b5538d - artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
• f224d4cb - artifactcache.py, _context.py: Move CASCache object into context
• 926c1446 - _sandboxremote: Remove references to artifact cache.
• d7cdb03e - Documentation: Update to show new config options
• 95d8244f - tests/sandboxes/remote-exec-config.py: New test.
• 7f2f62eb - artifactcache.py: Remove unused verify_digest_pushed
• c55ca3a5 - Docs: Correct URLs in artifact cache example

Compare with previous version

approved this merge request

merged

mentioned in commit 2a8a3b19

mentioned in commit BuildGrid/buildgrid@a2952208

mentioned in commit BuildGrid/buildgrid@f978e30e

mentioned in merge request BuildGrid/buildgrid!127 (merged)

mentioned in commit BuildGrid/buildgrid@535e62a0

mentioned in commit BuildGrid/buildgrid@70431fd4

mentioned in commit BuildGrid/buildgrid@c246d3af

mentioned in commit BuildGrid/buildgrid@a1ee2373