Skip to content

bst shell --build is using a buildtree that is cached after a staging failure

Summary

I have a source plugin that fails during staging. While it needs to be fixed, I found bst build and bst shell -b are also behaving interesting when staging of the plugin fails.

It is caching a buildtree that failed at Staging and implicitly using it in a following bst shell -b and 2nd bst build with (r)etry - Retry this job

Steps to reproduce

Have a source plugin failing at stage(), use that in A.bst.

$ bst build A.bst
..
[--:--:--][5002f532][build:A.bst                 ] START   project/A.log
[--:--:--][5002f532][build:A.bst                 ] START   Staging dependencies
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Staging dependencies
[--:--:--][5002f532][build:A.bst                 ] START   Integrating sandbox
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Integrating sandbox
[--:--:--][5002f532][build:A.bst                 ] START   Staging sources
[00:00:00][5002f532][build:A.bst                 ] FAILURE Staging XXYYZZ
[00:00:00][5002f532][build:A.bst                 ] FAILURE Staging sources # Ok, plugin failed
[--:--:--][5002f532][build:A.bst                 ] START   Caching artifact # But why cache??
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Caching artifact

# when asked, pick (t)erminate.

The problem starts here (caused by implicit preference of cached buildtree, captured here #740 (closed))

$ bst shell -b A.bst
[--:--:--][][] START   Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START   Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][5002f532][build:A.bst                 ] START   Staging dependencies
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Staging dependencies
[--:--:--][5002f532][build:A.bst                 ] START   Integrating sandbox
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Integrating sandbox
[--:--:--][5002f532][build:A.bst                 ] START   Staging sources
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Staging sources # SHOULD HAVE FAILED?
[--:--:--][5002f532][build:A.bst                 ] STATUS  Running command

    sh -i

Similar behaviour can also be seen here, in a second bst build attempt, but with (r)etry option.

$ bst build A.bst
[--:--:--][][] START   Build
[--:--:--][][] START   Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START   Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][][] START   Checking sources
[00:00:00][][] SUCCESS Checking sources

BuildStream Version 1.3.0+882.g327b19dd
  Session Start: Tuesday, 27-11-2018 at 11:50:24

..
..

Pipeline
      failed 5002f5324a330e6584005fc35bb7ae1aada7b16d04ef9491188517eb7f4726dd A.bst
===============================================================================
[00:00:00][5002f532][build:A.bst                 ] FAILURE XXYYZZ source at A.bst [line18 column 2]: Failed to checkout git ref A_REF_THAT_DOES_NOT_EXIST_IN_LOCAL_MIRROR

Build failure on element: A.bst

# It is actually ok so far.

Choose one of the following options:
  (c)ontinue  - Continue queueing jobs as much as possible
  (q)uit      - Exit after all ongoing jobs complete
  (t)erminate - Terminate any ongoing jobs and exit
  (r)etry     - Retry this job

Pressing ^C will terminate jobs and exit

Choice: [continue]: r

Retrying failed job

[--:--:--][5002f532][build:A.bst                 ] START   A.log
[--:--:--][5002f532][build:A.bst                 ] START   Staging dependencies
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Staging dependencies
[--:--:--][5002f532][build:A.bst                 ] START   Integrating sandbox
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Integrating sandbox
[--:--:--][5002f532][build:A.bst                 ] START   Staging sources
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Staging sources # SHOULD HAVE FAILED?
[--:--:--][5002f532][build:A.bst                 ] START   Running build-commands
[--:--:--][5002f532][build:A.bst                 ] STATUS  Running build-commands

    some_shell_script_to_run.sh

[00:00:00][5002f532][build:A.bst                 ] FAILURE Running build-commands
[--:--:--][5002f532][build:A.bst                 ] START   Caching artifact
[00:00:00][5002f532][build:A.bst                 ] SUCCESS Caching artifact
[00:00:00][5002f532][build:A.bst                 ] FAILURE Staged artifacts do not provide command
 'sh' # Don't mind this error. I just omitted a platform.bst.

What is the current bug behavior?

It is caching a buildtree from a build attempt if it failed at staging and implicitly prefers to use it during bst shell -b and bst build /w (r)etry.

What is the expected correct behavior?

It shouldn't cache if it failed at staging by default.

In case it caches (i.e. we want to keep that behaviour), it shouldn't use it by default unless user specifically asks for it (this is already captured by #740 (closed))

Relevant logs and/or screenshots

Possible fixes

This seems to happen at Element._assemble()

More specifically, when self.stage(sandbox) fails in the try .. except block, a buildtree-caching gets triggered.

We briefly discussed with @juergbi and @willsalmon and it seems like

                    # Step 1 - Configure
                    self.__configure_sandbox(sandbox)
                    # Step 2 - Stage
                    self.stage(sandbox)

should live out of that try .. except.

Other relevant information

  • BuildStream version affected: BuildStream Version 1.3.0+882.g327b19dd
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information