Remote Execution: build command failures hard to diagnose (always 'Directory not found')

Summary

When deliberately inducing a build failure by invoking 'false', I find that the error message is a red herring:

Directory '/buildstream-install' was not found inside the sandbox, unable to collect artifact contents

Steps to reproduce

• Follow the excellent running commands tutorial to setup a vanilla build project.
• Follow the BuildStream instructions in the readme of this buildgrid.hub.docker.com merge-request to get BuildGrid running locally.

Note that you probably want to put the remote config in your project.conf:

artifacts:
    url: http://localhost:50052
    push: true

remote-execution:
    execution-service:
        url: http://localhost:50051
    action-cache-service:
        url: http://localhost:50052
    storage-service:
        url: http://localhost:50052
        client-key: ''
        client-cert: ''
        server-cert: ''

First, verify that you can bst build hello.bst without any errors, watch for lines like this to know it was remote:

    [--:--:--][31d226f1][build:hello.bst                     ] START   Waiting for the remote build to complete
    [00:00:02][31d226f1][build:hello.bst                     ] SUCCESS Waiting for the remote build to complete

Next, apply a patch to make the build fail:

diff --git a/elements/hello.bst b/elements/hello.bst
index 4eb17d8..163948b 100644
--- a/elements/hello.bst
+++ b/elements/hello.bst
@@ -16,6 +16,8 @@ sources:
 config:

   build-commands:
+  - mkdir -p "%{install-root}"
+  - 'false'
   - make PREFIX="%{prefix}"

   install-commands:

What is the current bug behavior?

The error message is unrelated to the error, furthermore it seems to be complaining about a directory that does exist not existing.

    [--:--:--] START  : Waiting for the remote build to complete
    [00:00:03] SUCCESS: Waiting for the remote build to complete
    [--:--:--] START   hello.bst: Caching artifact
    [00:00:00] FAILURE hello.bst: Caching artifact
    [00:00:23] FAILURE hello.bst: Directory '/buildstream-install' was not found inside the sandbox, unable to collect artifact contents

What is the expected correct behavior?

Something more like this:

    [--:--:--] START   hello.bst: Running build-commands
    [--:--:--] STATUS  hello.bst: Running command

        false
    [00:00:00] FAILURE hello.bst: Running build-commands
    [--:--:--] START   hello.bst: Caching artifact
    [00:00:00] SUCCESS hello.bst: Caching artifact
    [00:00:03] FAILURE hello.bst: Command failed with exitcode 1

        false

or at least something that indicates a command failed.

Relevant logs and/or screenshots

Possible fixes

Perhaps this MR from @jmacarthur will help? Note that in the repro we deliberately create "%{install-root}" though, perhaps it is already too late.

• "sandboxremote.py: Create marked directories before running"

Other relevant information

This seems related to me, but not the same:

• "Remote Execution: Don't ignore stdout and stderr", this would also help.

---