Chandan Singh authored Jan 29, 2019 and Chandan Singh committed Jan 31, 2019

We use output of `git --version` to determine if we can run some tests
that rely on features from newer git versions. Usually, we expect the
output to be like:

    git version 2.17.2

On some platforms, like MacOS, there could be a suffix after the version
string, so that it looks something like:

    git version 2.17.2 (Apple Git-113)

This causes things to fail like so:

    ValueError: invalid literal for int() with base 10: '2 (Apple Git-113)\n'

Fix logic around `HAVE_OLD_GIT` such that we split the output of
`git --version` without limit on how many times we split. Previously we
used to split only twice so the suffixes like `(Apple Git-113)` are not
part of the parsed version.

testutils/site.py: Support parsing more exotic git versions

See merge request !1118

This removes the need for the 'Dependency' list to then be matched
with the corresponding LoadElement and will allow iterating the
graph more easily

This simplifies the loading

Cleanup loader by linking LoadElements sooner

See merge request !1122

tests/cachekey: Test cache keys are independent of target elements

See merge request !1123

Improve our filter documentation

Closes #278

See merge request !1112

This patch also uncovered the fact that our test_filter_deps_ok() test
has been inaccurate. Thus the element built in this test
(deps-permitted.bst) has been modified so that it build depends on the
input.bst element, as it should.

tests/filter.py: Ensure deps_ok test passes

Fail when we explictly try to include/exclude non-existent domains in a filter element

See merge request !1117

Providing a remote will limit build's pull/push remote actions to
the given remote specifically, ignoring those defined via user or
project configuration.

Add --remote, -r option to bst build, inline with pull & push

See merge request !1119

In order to support use of roaring bitmaps in the loader, we need to
depend on it here.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The dependency cache in LoadElement causes the peak RAM consumption of
the loader to be exceedingly large, upwards of 25GB for a test of a
Debian stack.  By switching from the old dict cache to a roaring bitmap
based cache, we reduce the cost of the loader cache in RAM terms to
around 5.5GB for the same stack, which makes it plausible for the short
term.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Switch to roaring bitmaps for the loader dependency caches

See merge request !1128

The unix specification for unix sockets only allocates ~100 characters
for the length of the path. This may be longer than the path for a CI
runner, or the path to a user's buildstream directory.

Add pre-merge tests that use a WSL runner

Closes #852

See merge request !1108

Daniel Silverstone authored Feb 06, 2019 and Benjamin Schubert committed Feb 06, 2019

The licence for roaringbitmap was not compatible.  pyroaring is
a similar project which is MIT licensed and is slightly better on
memory too.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

loadelement.py: Switch from roaringbitmap to pyroaring

Closes #899

See merge request !1133

The 'wallclock-us' is a option that we thought we already had.

Add more log formatting options

See merge request !1125

We want external plugins to be able to make use of the core testing utils.
This commit exposes the basic utilities which are currently in use in
bst-external plugins. If necessary, more utilities could be exposed in the
future.

Moves the following files from tests/testutils/ to
buildstream/plugintestingutils/:
o runcli.py
o integration.py

As part of this, this commit makes the following changes to runcli.py
and integration.py:
o runcli.py: Fix linting errors
o runcli.py: Add user facing documentation
o Integration.py: Add user facing documentation

Expose basic api for testing external plugins.

Closes #847

See merge request !1075

Abderrahim Kitouni authored Jan 27, 2019 and Javier Jardón committed Feb 08, 2019

the code for initializing remotes added the project specific remote
caches to the global list instead of making a copy.

Fixes #618

Abderrahim Kitouni authored Jan 28, 2019 and Javier Jardón committed Feb 08, 2019

This makes a junction use the artifact cache of the parent project
before the ones defined for the junction

Fixes #401

Use artifact cache specs from the parent project before those defined in junctions

Closes #618 and #401

See merge request !1113

isdir() follows symlinks on the host, resulting in potential host
contamination. This change reorders the file checks to avoid this issue.

This is required to ensure symlinks are not resolved on the host.

list_dirs was always True in the BuildStream code base. There was also a
bug in the list_dirs=False code path as it did not return symlinks in
`dirnames`.

This is an API break, however, there are no known external callers.

os.walk() resolves symlinks to check whether they point to a directory
even when followlinks is set to False. We already work around that
broken behavior by extracting symlinks from `dirnames`. However, the
sort order was still incorrect as we returned symlinks in dirnames
before files and other symlinks. This change fixes this, sorting all
files and symlinks in a single list.

This matches the change in utils.list_relative_paths() that now sorts
all symlinks as files, instead of following the broken behavior of
os.walk().