## Summary I have a source plugin that fails during staging. While it needs to be fixed, I found `bst build` and `bst shell -b` are also behaving interesting when `staging` of the plugin fails. __It is caching a buildtree that failed at `Staging` and implicitly using it in a following `bst shell -b` and 2nd `bst build` with `(r)etry - Retry this job`__ ## Steps to reproduce Have a source plugin failing at `stage()`, use that in `A.bst`. ```bash $ bst build A.bst .. [--:--:--][5002f532][build:A.bst ] START project/A.log [--:--:--][5002f532][build:A.bst ] START Staging dependencies [00:00:00][5002f532][build:A.bst ] SUCCESS Staging dependencies [--:--:--][5002f532][build:A.bst ] START Integrating sandbox [00:00:00][5002f532][build:A.bst ] SUCCESS Integrating sandbox [--:--:--][5002f532][build:A.bst ] START Staging sources [00:00:00][5002f532][build:A.bst ] FAILURE Staging XXYYZZ [00:00:00][5002f532][build:A.bst ] FAILURE Staging sources # Ok, plugin failed [--:--:--][5002f532][build:A.bst ] START Caching artifact # But why cache?? [00:00:00][5002f532][build:A.bst ] SUCCESS Caching artifact # when asked, pick (t)erminate. ``` The problem starts here (caused by implicit preference of cached buildtree, captured here https://gitlab.com/BuildStream/buildstream/issues/740) ```bash $ bst shell -b A.bst [--:--:--][][] START Loading elements [00:00:00][][] SUCCESS Loading elements [--:--:--][][] START Resolving elements [00:00:00][][] SUCCESS Resolving elements [--:--:--][][] START Resolving cached state [00:00:00][][] SUCCESS Resolving cached state [--:--:--][5002f532][build:A.bst ] START Staging dependencies [00:00:00][5002f532][build:A.bst ] SUCCESS Staging dependencies [--:--:--][5002f532][build:A.bst ] START Integrating sandbox [00:00:00][5002f532][build:A.bst ] SUCCESS Integrating sandbox [--:--:--][5002f532][build:A.bst ] START Staging sources [00:00:00][5002f532][build:A.bst ] SUCCESS Staging sources # SHOULD HAVE FAILED? [--:--:--][5002f532][build:A.bst ] STATUS Running command sh -i ``` Similar behaviour can also be seen here, in a second `bst build` attempt, but with `(r)etry` option. ```bash $ bst build A.bst [--:--:--][][] START Build [--:--:--][][] START Loading elements [00:00:00][][] SUCCESS Loading elements [--:--:--][][] START Resolving elements [00:00:00][][] SUCCESS Resolving elements [--:--:--][][] START Resolving cached state [00:00:00][][] SUCCESS Resolving cached state [--:--:--][][] START Checking sources [00:00:00][][] SUCCESS Checking sources BuildStream Version 1.3.0+882.g327b19dd Session Start: Tuesday, 27-11-2018 at 11:50:24 .. .. Pipeline failed 5002f5324a330e6584005fc35bb7ae1aada7b16d04ef9491188517eb7f4726dd A.bst =============================================================================== [00:00:00][5002f532][build:A.bst ] FAILURE XXYYZZ source at A.bst [line18 column 2]: Failed to checkout git ref A_REF_THAT_DOES_NOT_EXIST_IN_LOCAL_MIRROR Build failure on element: A.bst # It is actually ok so far. Choose one of the following options: (c)ontinue - Continue queueing jobs as much as possible (q)uit - Exit after all ongoing jobs complete (t)erminate - Terminate any ongoing jobs and exit (r)etry - Retry this job Pressing ^C will terminate jobs and exit Choice: [continue]: r Retrying failed job [--:--:--][5002f532][build:A.bst ] START A.log [--:--:--][5002f532][build:A.bst ] START Staging dependencies [00:00:00][5002f532][build:A.bst ] SUCCESS Staging dependencies [--:--:--][5002f532][build:A.bst ] START Integrating sandbox [00:00:00][5002f532][build:A.bst ] SUCCESS Integrating sandbox [--:--:--][5002f532][build:A.bst ] START Staging sources [00:00:00][5002f532][build:A.bst ] SUCCESS Staging sources # SHOULD HAVE FAILED? [--:--:--][5002f532][build:A.bst ] START Running build-commands [--:--:--][5002f532][build:A.bst ] STATUS Running build-commands some_shell_script_to_run.sh [00:00:00][5002f532][build:A.bst ] FAILURE Running build-commands [--:--:--][5002f532][build:A.bst ] START Caching artifact [00:00:00][5002f532][build:A.bst ] SUCCESS Caching artifact [00:00:00][5002f532][build:A.bst ] FAILURE Staged artifacts do not provide command 'sh' # Don't mind this error. I just omitted a platform.bst. ``` ## What is the current bug behavior? It is caching a buildtree from a build attempt if it failed at __staging__ and implicitly prefers to use it during `bst shell -b` and `bst build /w (r)etry`. ## What is the expected correct behavior? It shouldn't cache if it failed at __staging__ by default. In case it caches (i.e. we want to keep that behaviour), it shouldn't __use__ it by default unless user specifically asks for it (this is already captured by https://gitlab.com/BuildStream/buildstream/issues/740) ## Relevant logs and/or screenshots [//]: # (Paste any relevant logs - please use code blocks ``` to format console output, logs, and code as it's hard to read otherwise. You can also add a Snippet and link it here. Check the markdown giude on Gitlab for further tips) ## Possible fixes This seems to happen at [`Element._assemble()`](https://gitlab.com/BuildStream/buildstream/blob/7827ac5039f7d7cd80e3937d0b3c81e462524c78/buildstream/element.py#L1551) More specifically, when [`self.stage(sandbox)`](https://gitlab.com/BuildStream/buildstream/blob/7827ac5039f7d7cd80e3937d0b3c81e462524c78/buildstream/element.py#L1584) fails in the `try .. except` block, a buildtree-caching gets [triggered](https://gitlab.com/BuildStream/buildstream/blob/7827ac5039f7d7cd80e3937d0b3c81e462524c78/buildstream/element.py#L1622). We briefly discussed with @juergbi and @willsalmon and it seems like ```python # Step 1 - Configure self.__configure_sandbox(sandbox) # Step 2 - Stage self.stage(sandbox) ``` should live out of that `try .. except`. ## Other relevant information [//]: # (Add the BuildStream version. Substitute below the "x" by "1", "2" or the right version. Add more than one version if necessary) * BuildStream version affected: BuildStream Version 1.3.0+882.g327b19dd ----