We want external plugins to be able to make use of the core testing utils.
This commit exposes the basic utilities which are currently in use in
bst-external plugins. If necessary, more utilities could be exposed in the
future.

Moves the following files from tests/testutils/ to
buildstream/plugintestingutils/:
o runcli.py
o integration.py

As part of this, this commit makes the following changes to runcli.py
and integration.py:
o runcli.py: Fix linting errors
o runcli.py: Add user facing documentation
o Integration.py: Add user facing documentation

Expose basic api for testing external plugins.

Closes #847

See merge request !1075

Abderrahim Kitouni authored Jan 27, 2019 and Javier Jardón committed Feb 08, 2019

the code for initializing remotes added the project specific remote
caches to the global list instead of making a copy.

Fixes #618

Abderrahim Kitouni authored Jan 28, 2019 and Javier Jardón committed Feb 08, 2019

This makes a junction use the artifact cache of the parent project
before the ones defined for the junction

Fixes #401

Use artifact cache specs from the parent project before those defined in junctions

Closes #618 and #401

See merge request !1113

Javier Jardón authored Feb 08, 2019 and Jürg Billeter committed Feb 09, 2019

Fedora 27 is EOL since 2018-11-30

See https://fedoraproject.org/wiki/End_of_life

.gitlab-ci.yml: Remove testing on Fedora 27

See merge request !1136

Fixes #662.

import.py: Validate config node

Closes #662

See merge request !1141

isdir() follows symlinks on the host, resulting in potential host
contamination. This change reorders the file checks to avoid this issue.

This is required to ensure symlinks are not resolved on the host.

list_dirs was always True in the BuildStream code base. There was also a
bug in the list_dirs=False code path as it did not return symlinks in
`dirnames`.

This is an API break, however, there are no known external callers.

os.walk() resolves symlinks to check whether they point to a directory
even when followlinks is set to False. We already work around that
broken behavior by extracting symlinks from `dirnames`. However, the
sort order was still incorrect as we returned symlinks in dirnames
before files and other symlinks. This change fixes this, sorting all
files and symlinks in a single list.

This matches the change in utils.list_relative_paths() that now sorts
all symlinks as files, instead of following the broken behavior of
os.walk().

Symlink fixes

See merge request !1138

Angelos Evripiotis authored Feb 05, 2019 and Jürg Billeter committed Feb 11, 2019

Replace the instructions for pyflame+flamegraph with simpler ones for
snakeviz. For our general use-case this seems to be easier and better.
Usage of this tool was demonstrated at the 2019 BuildStream Gathering in
January by Daniel Silverstone, when presenting the aggregate results of
profiling on many target environments.

Here is the relevant mailing list thread:

"Profiling before the gathering"
https://mail.gnome.org/archives/buildstream-list/2019-January/msg00057.html

contributing: snakeviz replaces pyflame+flamegraph

See merge request !1129

Angelos Evripiotis authored Oct 23, 2018 and Jürg Billeter committed Feb 11, 2019

Use the provenance of the include block, instead of the whole node.

Angelos Evripiotis authored Oct 23, 2018 and Jürg Billeter committed Feb 11, 2019

Previously, a missing include would result in an error like this:

    Could not find file at not-a-file.include

Note that the file containing the include was not mentioned.

Now we get an error like this instead:

    element.bst [line 7 column 5]: Include block references a file that
    could not be found: 'not-a-file.include'.

Angelos Evripiotis authored Oct 23, 2018 and Jürg Billeter committed Feb 11, 2019

Previously, include a directory result in an error like this:

    mydir is a directory. bst command expects a .bst file.

Note that the file containing the include was not mentioned.

Now we get an error like this instead:

    element.bst [line 12 column 0]: Include block references a
    directory instead of a file: 'mydir'.

Angelos Evripiotis authored Oct 23, 2018 and Jürg Billeter committed Feb 11, 2019

Avoid an unnecessary call to os.path.join().

Angelos Evripiotis authored Feb 06, 2019 and Jürg Billeter committed Feb 11, 2019

Don't create and remove temp dirs unnecessarily when they are not used,
looks like these were just copy-pastes without intended side-effects.

More user-friendly reporting on include errors

See merge request !891

.gitlab-ci.yml: allow "test-wsl" to fail until runner problems are sorted out

See merge request !1145

Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.

Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.

Increase read buffer size to improve performance

See merge request !1142

At present, there isn't an easy way to print anything from `bst show`
that would give the users an idea of what the dependency graph looks
like. One could use things like `--deps build`, but that will just print
a list, without any information about the dependency edges.

Add `%{deps}`, `%{build-deps}` and `%{runtime-deps}` format strings to
`bst show` that would simply print the list of all dependencies, build
dependencies and runtime dependencies respectively.

Summary of changes:

  * buildstream/_frontend/cli.py: Add help for new format symbols.
  * buildstream/_frontend/widget.py: Add support for new format symbols
                                     for dependencies.
  * tests/frontend/show.py: Add tests for new format symbols.

_frontend: Allow printing dependencies using `bst show`

Closes #890

See merge request !1121

Fedora 29 includes python 3.7 so this is not needed anymore

.gitlab-ci.yml: Test with current fedora release: 29

See merge request !1137

We were indiscriminately retrieving the node's provenance data in the
`node_get()` function which was accounting for approximately a third of
the total runtime of `node_get()` which dominates pre-scheduler time in
`bst build`.  This change ameliorates that situation by only retrieving
the provenance data when it's actually needed.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

We know that nodes are typically one of:
str, list, dict, bool, tuple, NoneType or our ChainMap
Of these, dict and ChainMap are Mapping, only list is list
and the rest are returned unchanged.  We can reduce/defer our use
of isinstance here, dramatically, improving performance.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

A non-trivial proportion of time pre-scheduler in `bst build` is spent
copying (or chain-copying) nodes.  Approximately a quarter of the time
spent in that effort is in `isinstance()`.  This removes that CPU load.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

By re-using the isinstance replacements from earlier commits and
using a tuple of the string constants for checking for composition
markers, we reduce the cost of node_final_assertions by two thirds
in basic testing.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

While get_strict() doesn't look expensive per-se, it is called so many
times that it is valuable to cache the result once computed.  Since I
don't think it can change once it is computable, cache it immediately
that becomes possible and we save 20s in my test case.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Further optimisations

See merge request !1131

Since junction check-outs are now persistent, there is no need to clean
up temporary directories once finished.
Now, junction checkouts are stored within the top-level project dir and
will be cleaned-up when that directory is removed.

This involves changes in:
* loader.py: Remove cleanup logic and passing-around of tmpdir.
* _project.py: Remove passing around tmpdir and calling loader's
               cleanup.