Since junction check-outs are now persistent, there is no need to clean
up temporary directories once finished.
Now, junction checkouts are stored within the top-level project dir and
will be cleaned-up when that directory is removed.

This involves changes in:
* loader.py: Remove cleanup logic and passing-around of tmpdir.
* _project.py: Remove passing around tmpdir and calling loader's
               cleanup.

Stage junctions into .bst instead of a tmpdir

Closes #895

See merge request !1134

Valentin David authored Feb 11, 2019 and Valentin David committed Feb 12, 2019

This was broken by 5ef19a0b.

buildstream/_cas/cascache.py: Set 0644 rights to pulled files

See merge request !1144

This commit removes the method ArtifactCache.get_artifact_fullname()
and replaces it with Element.get_artifact_name()

Given a key, we are now able to construct the full name of any of an
element's artifacts.

This object should be used when we want to handle artifact refs directly
from the command line.
An ArtifactElementError has also been added to _exceptions.py

* There is no need for this method to use a cas object.
* Search for artifact globs in the project's element path
* An artifact key is always 64 chars long

A CasBasedDirectory object of an artifacts logs can be obtained
with ArtifactCache.get_artifacts_log(). This ultimately calls
CASCache.get_top_level_dir() to obtain a CasBasedDirectory
of an artifact's subdirectory (or subdirectories).

The loading of elements and the handling of artifacts does not belong
in this module. Such logic should be invoked using the Stream API

This method has been moved to Stream, where is it used there exclusively.

This commit ensures that CASCache.list_refs(), and
ArtifactCache.list_artifacts(), can both handle glob expressions.

_classify_artifacts() no longer filters glob expressions by first
obtaining a list of ALL refs locally cached. We now only obtain refs
specified by the glob expression.

Furthermore, the Project.element_path is used to start searching for
globbed elements as opposed to the Project.directory.

Refactor artifact log command

See merge request !1101

Most split rules already included the relevant directories themselves in
addition to the directory contents.  Add the missing bin, sbin, and
libexec directories.

This is required to fix tests with the following commit that changes
list_relative_paths() to return all directories.

This is required to fix tests with the following commit that changes
list_relative_paths() to return all directories.

Returning only empty directories leads to inconsistencies when computing
a manifest by combining results from multiple list_relative_paths()
calls as done by the compose plugin.

I.e., the same directory may be empty in one dependency and non-empty in
another dependency.  The merged file list will still contain that
directory even though it's no longer empty.

This inconsistency causes problems when calculating differences between
manifests. Returning all directories fixes these inconsistencies.

This is a change in API behavior.

This matches the change in utils.list_relative_paths().

Return all directories in list_relative_paths()

See merge request !1139

Resolving symlinks during staging causes various issues:

* Split rules may not work properly as the resolved paths will differ
  depending on whether another artifact with a directory symlink has
  been staged in the same root directory or not, e.g., as part of
  compose.

* The order of symlinks in file lists is difficult to get right to
  guarantee consistent and predictable behavior as paths in a file list
  might rely on symlinks in the same file list. See #647 and #817.

* Staging order differences can lead to surprising results. See #390.

* Difficult to properly support absolute symlinks. Absolute symlinks are
  currently converted to relative symlinks, however, this doesn't always
  work. See #606 and #830.

This will require changes in projects that rely on the current behavior.
However, the changes are expected to be small and are often a sign of
buggy element files. E.g., elements that don't fully obey `bindir` or
`sbindir` variables.

This matches the change in utils._process_list().

This also removes the _Resolver class as it is now unused. We may want
to support controlled symlink resolution in the future, in which case
the _Resolver class can be resurrected from this commit.

Copy symlinks as they are, absolute or relative. We no longer resolve
symlinks when copying files, which makes this safe.