_classify_artifacts() no longer filters glob expressions by first
obtaining a list of ALL refs locally cached. We now only obtain refs
specified by the glob expression.

Furthermore, the Project.element_path is used to start searching for
globbed elements as opposed to the Project.directory.

This commit ensures that CASCache.list_refs(), and
ArtifactCache.list_artifacts(), can both handle glob expressions.

This method has been moved to Stream, where is it used there exclusively.

The loading of elements and the handling of artifacts does not belong
in this module. Such logic should be invoked using the Stream API

A CasBasedDirectory object of an artifacts logs can be obtained
with ArtifactCache.get_artifacts_log(). This ultimately calls
CASCache.get_top_level_dir() to obtain a CasBasedDirectory
of an artifact's subdirectory (or subdirectories).

* There is no need for this method to use a cas object.
* Search for artifact globs in the project's element path
* An artifact key is always 64 chars long

This object should be used when we want to handle artifact refs directly
from the command line.
An ArtifactElementError has also been added to _exceptions.py

This commit removes the method ArtifactCache.get_artifact_fullname()
and replaces it with Element.get_artifact_name()

Given a key, we are now able to construct the full name of any of an
element's artifacts.

buildstream/_cas/cascache.py: Set 0644 rights to pulled files

See merge request !1144

Valentin David authored 6 years ago and Valentin David committed 6 years ago

This was broken by 5ef19a0b.

Stage junctions into .bst instead of a tmpdir

Closes #895

See merge request !1134

Since junction check-outs are now persistent, there is no need to clean
up temporary directories once finished.
Now, junction checkouts are stored within the top-level project dir and
will be cleaned-up when that directory is removed.

This involves changes in:
* loader.py: Remove cleanup logic and passing-around of tmpdir.
* _project.py: Remove passing around tmpdir and calling loader's
               cleanup.

Further optimisations

See merge request !1131

While get_strict() doesn't look expensive per-se, it is called so many
times that it is valuable to cache the result once computed.  Since I
don't think it can change once it is computable, cache it immediately
that becomes possible and we save 20s in my test case.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

By re-using the isinstance replacements from earlier commits and
using a tuple of the string constants for checking for composition
markers, we reduce the cost of node_final_assertions by two thirds
in basic testing.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

A non-trivial proportion of time pre-scheduler in `bst build` is spent
copying (or chain-copying) nodes.  Approximately a quarter of the time
spent in that effort is in `isinstance()`.  This removes that CPU load.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

We know that nodes are typically one of:
str, list, dict, bool, tuple, NoneType or our ChainMap
Of these, dict and ChainMap are Mapping, only list is list
and the rest are returned unchanged.  We can reduce/defer our use
of isinstance here, dramatically, improving performance.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

We were indiscriminately retrieving the node's provenance data in the
`node_get()` function which was accounting for approximately a third of
the total runtime of `node_get()` which dominates pre-scheduler time in
`bst build`.  This change ameliorates that situation by only retrieving
the provenance data when it's actually needed.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

.gitlab-ci.yml: Test with current fedora release: 29

See merge request !1137

Fedora 29 includes python 3.7 so this is not needed anymore

_frontend: Allow printing dependencies using `bst show`

Closes #890

See merge request !1121

At present, there isn't an easy way to print anything from `bst show`
that would give the users an idea of what the dependency graph looks
like. One could use things like `--deps build`, but that will just print
a list, without any information about the dependency edges.

Add `%{deps}`, `%{build-deps}` and `%{runtime-deps}` format strings to
`bst show` that would simply print the list of all dependencies, build
dependencies and runtime dependencies respectively.

Summary of changes:

  * buildstream/_frontend/cli.py: Add help for new format symbols.
  * buildstream/_frontend/widget.py: Add support for new format symbols
                                     for dependencies.
  * tests/frontend/show.py: Add tests for new format symbols.

Increase read buffer size to improve performance

See merge request !1142

Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.

Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.

.gitlab-ci.yml: allow "test-wsl" to fail until runner problems are sorted out

See merge request !1145

More user-friendly reporting on include errors

See merge request !891

Angelos Evripiotis authored 6 years ago and Jürg Billeter committed 6 years ago

Don't create and remove temp dirs unnecessarily when they are not used,
looks like these were just copy-pastes without intended side-effects.