As per PEP 0479 (https://www.python.org/dev/peps/pep-0479/),
StopIteration thrown in context managers are not valid starting from
Python 3.7.

source.py: don't let StopIteration propagate to silence() contextmanager

See merge request !945

Phillip Smyth authored Nov 07, 2018 and Jürg Billeter committed Nov 12, 2018

_project.py: Added validate_nodes() helper function to prevent duplicate lists

element-path was being used before node validation resulting in uncaught errors

Element path not validated before use

See merge request !937

Valentin David authored Nov 12, 2018 and Valentin David committed Nov 14, 2018

Fixes #755

Run tests on aarch64

Closes #369 and #755

See merge request !948

It's inconvenient to have to create a Linux platform to parse the bwrap version
and we want to get the version in a consistent manner.

If `bwrap` fails to set up the sandbox and start the payload command
it won't write an exit-code in --json-status-fd,
so we can report if it was a sandboxing failure if we don't get exit-code status
and a payload command failure if we do and it's non-zero.

Closes #286

Distinguish between bubblewrap sandboxing failure and command failure

Closes #286

See merge request !868

Angelos Evripiotis authored Nov 08, 2018 and Jürg Billeter committed Nov 15, 2018

Decrease uncertainty around whether unit tests are welcome in the
project or not.

contributing: add guidance on unit tests

See merge request !943

Tom Pollard authored Nov 05, 2018 and Jürg Billeter committed Nov 17, 2018

_context.py: Add pull_buildtrees global user context, the default
of which is set to False via the addition of pull-buildtrees to
userconfig.yaml cache group.

_frontend/app.py & cli.py: Add --pull-buildtrees as a bst main
option, which when passed will override the default or user defined
context for pull_buildtrees.

tests/completions/completions.py: Update for the added flag.

Tom Pollard authored Nov 05, 2018 and Jürg Billeter committed Nov 17, 2018

The addition of cached build trees being included in element
artifacts has led to mostly redundant download overheads when
pulling from a remote artifact server. As such the default behaviour
of pull shouldn't fetch the build tree object if available.

element.py: extend relevant pull logic for specified subdir
consideration and ensure push logic does not lead to partial
artifact pushes. Change assumption that the buildtree can be
extracted if artifact is cached. __cached_buildtree() and
__pull_directories() helpers added.

_artifactcache/: artifactcache.py & cascache.py inclusion of
helper functions for subdir artifact checking & fetching, fetch
logic extended to only pull required artifact directories.
extract & checkout updated to handle full/partial operation.

tests/: addition of integration test pullbuildtrees.py,
buildtrees.py adapted cli options, testutils/artifactshare.py
has_artifact changed to return artifact digest on true condition.

Don't pull artifact buildtrees by default

See merge request !786

In some virtual environments, we might not have the binary with the
exact python version, which would lead the current pip plugin to
fail to find a working pip version.

This adds "python" at the start of the list of valid python versions

plugins/sources/pip.py: also look for python version named "python"

Closes #758

See merge request !942

Benjamin Schubert authored Nov 12, 2018 and richardmaw-codethink committed Nov 19, 2018

Previously the result of the first call in test_build_track was not
checked to be successful, which would lead to potential erros later on
that were hard to debug.

Let's check it everytime

Benjamin Schubert authored Nov 07, 2018 and richardmaw-codethink committed Nov 19, 2018

According to the documentation
(https://www.unix.com/man-page/POSIX/3posix/rename/), when the directory
already is there, either EEXIST or ENOTEMPTY could be thrown.

Previously only ENOTEMPTY was checked.

Done:
  - Separated the move into its own function
  - Check for both errors
  - Create unit tests for it, covering most test cases

Benjamin Schubert authored Nov 09, 2018 and richardmaw-codethink committed Nov 19, 2018

Moving atomically a file/directory can be tricky since different
errors might be raised for the same underlying problem.

Having a utility function to reduce this discrepancies will help
in ensuring we have correct behavior

Benjamin Schubert authored Nov 09, 2018 and richardmaw-codethink committed Nov 19, 2018

This uses move_atomic insteand of the manual os.rename and manual error
checking and throws a SourceError for consistency with other modules.

Fix os.rename in git source element to correctly handle error codes

See merge request !938

Chandan Singh authored Oct 23, 2018 and Chandan Singh committed Nov 19, 2018

Currently, `Pipeline.assert_consistent()` prints an error message like
the one below when certain elements have inconsistent sources:

```
...
      Element: hello.bst is inconsistent
        Source tar source at hello.bst [line 16 column 2] is missing ref
...
```

Drop the word "Source" from the beginning of the message as
"Source <source-kind> source ..." is kind of awkward to read.

Chandan Singh authored Sep 22, 2018 and Chandan Singh committed Nov 19, 2018

As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `bst source-checkout` command. This will allow users to checkout
sources for a given target.

* _frontend/cli.py: Add source-checkout command
* _pipeline.py: Add assert_sources_cached() method
* _stream.py: Add source_checkout method, abstract out
  __check_location_writable() method that used to part of checkout()

Add `bst source-checkout` command

See merge request !820

Abderrahim Kitouni authored Oct 28, 2018 and Valentin David committed Nov 19, 2018

cmake sometimes misinterprets relative paths as relative to the current directory
if this is not specified. See freedesktop-sdk/freedesktop-sdk#431

adjust tests/format/variables.py accordingly.

plugins/elements/cmake.yaml: always specify variable types

Closes #761

See merge request !947

Sandbox errors (like missing host tools) are dependent on the host
system and rarely on what is actually done.

It is therefore better to not cache them as they are subject to
change between two runs.

Also add test to ensure sandbox failure are not cached

Reverse the way we were handling caching by only caching
when we know it's a problem with the build or the build is a
success in order not to cache transient errors.

Don't cache sandbox failures

Closes #727

See merge request !895

`bst source-checkout` command was recently added in
!820. Add a
NEWS entry for the new command.

NEWS: Add entry for the new source-checkout command

See merge request !960

richardmaw-codethink authored Nov 14, 2018 and Jürg Billeter committed Nov 19, 2018

The Context was only used to obtain a reference to the CASCache
and set the unused cas_directory field.

The fake context did not set the cache quota, triggering an error. With
CASCache now separate from ArtifactCache, we can instantiate a CASCache
without context.

The default values are in userconfig.yaml, together with the
documentation. The default values should not be duplicated in
_context.py.

_context.py: Drop duplicated default values for user configuration

See merge request !953

Enable this option of 'terminate', which is mentioned in userconfig.yaml
and handled in _frontend/app.py:_handle_failure(). It appears to have
been left out of the valid_actions as an oversight.

Originally introduced in
2622d5da

Use a new helper function to simplify working with nodes that can only
accept certain strings. This will be used when adding the prompt.*
config options.

In later work we can see if this function would be useful elsewhere, and
could be added to '_yaml.py'.

Provide an option in buildstream.conf to disable the 'Would you like to
...' prompt when we cannot resolve a project.

Some users prefer not to be interrupted by such prompts, so pave the way
to creating options to disable all those that might get in the way.

Follow the example of the advice.* options 'git-config', and create a
namespace for these UI options grouped by behaviour, rather than an
over-reaching 'ui.*' namespace. In later work perhaps we'll also add
'advice.*' options.

Add a NEWS item for this.

Provide options in project.conf to disable the 'Are you sure ...'
prompts when making destructive changes:

- Add prompt.really-workspace-close-remove-dir
- Add prompt.really-workspace-reset-hard

Add a NEWS item for these.

Add 'prompt.*' config options to buildstream.conf

See merge request !887

_sandboxremote.py: Add sigterm handler that sends CancelOperation

Closes #725

See merge request !900

James Ennis authored Nov 16, 2018 and Jürg Billeter committed Nov 20, 2018

This patch ensures that we receive an appropriate error
message if we specify an absolute path that leads within
the project.

Fix warning when using an absolute path which exists inside the project

See merge request !956

This doesn't change the test behavior and enable for better
composability of tests

This will allow for further de-duplication of work.

The unix-logs is always empty, there is therefore no reasons of keeping
it there

This makes all tests use the same template, which makes reasoning
about them simpler

This allows us to make sure that tests behave nicely while missing brwap
and ostree.

This removes the need of having it synchronized in multiple places

Add tests for fedora with no bubblewrap available

See merge request !949

This is to be coherent with the rest of the codebase

Followup on MR 938, addressing additional comments

See merge request !958

create_pipeline was passing no config parameter to context.load,
leading to _context.py loading in the host's buildstream user conf
when running tests locally, potentially leading to unwanted errors.

tests/plugin/pipeline.py: Avoid using host user conf

See merge request !961

William Salmon authored Nov 02, 2018 and William Salmon committed Nov 21, 2018

The Refactor now makes it possible to test for the forth coming multi
element workspace open commands and to test for failure.

William Salmon authored Nov 02, 2018 and William Salmon committed Nov 21, 2018

This is to update the workspace CLI to as agreed on the mailing list
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00046.html

This patch also introduces the default workspace directory.

Updated Workspace CLI

See merge request !897

Bump version of fd.o SDK and bst_external in overnight tests

See merge request !962

_frontend/cli.py: Added --all flag to `def show()`
_frontend/cli.py: Added helper function for walking through dirs to return files