bst shell --build is using a buildtree that is cached after a staging failure
Summary
I have a source plugin that fails during staging. While it needs to be fixed, I found bst build
and bst shell -b
are also behaving interesting when staging
of the plugin fails.
It is caching a buildtree that failed at Staging
and implicitly using it in a following bst shell -b
and 2nd bst build
with (r)etry - Retry this job
Steps to reproduce
Have a source plugin failing at stage()
, use that in A.bst
.
$ bst build A.bst
..
[--:--:--][5002f532][build:A.bst ] START project/A.log
[--:--:--][5002f532][build:A.bst ] START Staging dependencies
[00:00:00][5002f532][build:A.bst ] SUCCESS Staging dependencies
[--:--:--][5002f532][build:A.bst ] START Integrating sandbox
[00:00:00][5002f532][build:A.bst ] SUCCESS Integrating sandbox
[--:--:--][5002f532][build:A.bst ] START Staging sources
[00:00:00][5002f532][build:A.bst ] FAILURE Staging XXYYZZ
[00:00:00][5002f532][build:A.bst ] FAILURE Staging sources # Ok, plugin failed
[--:--:--][5002f532][build:A.bst ] START Caching artifact # But why cache??
[00:00:00][5002f532][build:A.bst ] SUCCESS Caching artifact
# when asked, pick (t)erminate.
The problem starts here (caused by implicit preference of cached buildtree, captured here #740 (closed))
$ bst shell -b A.bst
[--:--:--][][] START Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][5002f532][build:A.bst ] START Staging dependencies
[00:00:00][5002f532][build:A.bst ] SUCCESS Staging dependencies
[--:--:--][5002f532][build:A.bst ] START Integrating sandbox
[00:00:00][5002f532][build:A.bst ] SUCCESS Integrating sandbox
[--:--:--][5002f532][build:A.bst ] START Staging sources
[00:00:00][5002f532][build:A.bst ] SUCCESS Staging sources # SHOULD HAVE FAILED?
[--:--:--][5002f532][build:A.bst ] STATUS Running command
sh -i
Similar behaviour can also be seen here, in a second bst build
attempt, but with (r)etry
option.
$ bst build A.bst
[--:--:--][][] START Build
[--:--:--][][] START Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][][] START Checking sources
[00:00:00][][] SUCCESS Checking sources
BuildStream Version 1.3.0+882.g327b19dd
Session Start: Tuesday, 27-11-2018 at 11:50:24
..
..
Pipeline
failed 5002f5324a330e6584005fc35bb7ae1aada7b16d04ef9491188517eb7f4726dd A.bst
===============================================================================
[00:00:00][5002f532][build:A.bst ] FAILURE XXYYZZ source at A.bst [line18 column 2]: Failed to checkout git ref A_REF_THAT_DOES_NOT_EXIST_IN_LOCAL_MIRROR
Build failure on element: A.bst
# It is actually ok so far.
Choose one of the following options:
(c)ontinue - Continue queueing jobs as much as possible
(q)uit - Exit after all ongoing jobs complete
(t)erminate - Terminate any ongoing jobs and exit
(r)etry - Retry this job
Pressing ^C will terminate jobs and exit
Choice: [continue]: r
Retrying failed job
[--:--:--][5002f532][build:A.bst ] START A.log
[--:--:--][5002f532][build:A.bst ] START Staging dependencies
[00:00:00][5002f532][build:A.bst ] SUCCESS Staging dependencies
[--:--:--][5002f532][build:A.bst ] START Integrating sandbox
[00:00:00][5002f532][build:A.bst ] SUCCESS Integrating sandbox
[--:--:--][5002f532][build:A.bst ] START Staging sources
[00:00:00][5002f532][build:A.bst ] SUCCESS Staging sources # SHOULD HAVE FAILED?
[--:--:--][5002f532][build:A.bst ] START Running build-commands
[--:--:--][5002f532][build:A.bst ] STATUS Running build-commands
some_shell_script_to_run.sh
[00:00:00][5002f532][build:A.bst ] FAILURE Running build-commands
[--:--:--][5002f532][build:A.bst ] START Caching artifact
[00:00:00][5002f532][build:A.bst ] SUCCESS Caching artifact
[00:00:00][5002f532][build:A.bst ] FAILURE Staged artifacts do not provide command
'sh' # Don't mind this error. I just omitted a platform.bst.
What is the current bug behavior?
It is caching a buildtree from a build attempt if it failed at staging and implicitly prefers to use it during bst shell -b
and bst build /w (r)etry
.
What is the expected correct behavior?
It shouldn't cache if it failed at staging by default.
In case it caches (i.e. we want to keep that behaviour), it shouldn't use it by default unless user specifically asks for it (this is already captured by #740 (closed))
Relevant logs and/or screenshots
Possible fixes
This seems to happen at Element._assemble()
More specifically, when self.stage(sandbox)
fails in the try .. except
block, a buildtree-caching gets triggered.
We briefly discussed with @juergbi and @willsalmon and it seems like
# Step 1 - Configure
self.__configure_sandbox(sandbox)
# Step 2 - Stage
self.stage(sandbox)
should live out of that try .. except
.
Other relevant information
- BuildStream version affected: BuildStream Version 1.3.0+882.g327b19dd