The CAS uses a temp directory while manipulating the cache, temporary
files can be added and removed while adding artifacts to the cache here.

Since calculation of the cache size happens in parallel to artifact
cache additions, this causes race conditions in the size calculation
job, as we end up calling `stat` on temporary files which are being
removed in parallel.

Handle this by moving the temporary directory out of the way, and avoid
considering the tmp directory when calculating the cache size

Fix race condition when calculating disk usage

See merge request !600

When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
must be readable. Unfortunately O_RDWR was not passed which made
read fail with EBADF and mmap to signal SIGBUS.

This issue happened with man-db for example.

Fixes #143.

Keep original flags for create in SafeHardlinks.

Closes #143

See merge request !624

Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
systems where not all CPU can be used (in particular, Docker, CI etc).

As suggested in the multiprocessing docs
(http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
using len(os.sched_getaffinity(0)) might be better.

even if the machine have mores cores available

Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227

buildstream/_project.py: Restrict max-jobs

See merge request !620

Pin ruamel.yaml version to <= 0.15

See merge request !630

This fixes #191

A note has been added to NEWS explaining backwards
compatibility issues

Patch for issue #191 support relative workspaces

Closes #191

See merge request !504

Valentin David authored Jul 25, 2018 and Valentin David committed Aug 10, 2018

Fixes #498

Set environment in bwrap command line instead of its environment

Closes #498

See merge request !565

In some cases, such as when working inside a virtual environment, it can
be desirable to install all dependencies for running tests using `pip`.
This is currently not possible since setuptools does not support
installing these dependencies in a virtual environment (by design).
(See https://stackoverflow.com/a/21003259.)

To circumvent this issue, move such requirements to
`dev-requirements.txt` file that can be used easily with
`pip install -r`. This also enables tests to be run directly using
`pytest`, which can be more convenient than `-addopts` approach when one
needs to add multiple options.

This will also be useful in creating better testuite images, and fix
some of the issues noticed in
buildstream-docker-images!56.

Move development reqirements to dev-requirements.txt

See merge request !637

Fix typo in dev-requirements.txt

See merge request !641

.gitlab-ci-yml: Add ubuntu 18.04 test

See merge request !523

This also fixes an error with pylint in setup.py:
setup.py:226:19: R1718: Consider using a set comprehension (consider-using-set-comprehension)

Add job to run tests in fedora 28

See merge request !643

Valentin David authored Aug 06, 2018 and Tristan Van Berkom committed Aug 12, 2018

Instead of copying metadata on files staged by local, we manually set
mode to 0755 or 0644 depending on whether user execution was enabled
on source file.

This makes file modes deterministic independently on the way source
was distributed.

Non-deterministic mode copying all metadata can still be enabled by
disable 'deterministic' Boolean configuration on the plugin.

Fixes #527.

Valentin David authored Aug 06, 2018 and Tristan Van Berkom committed Aug 12, 2018

This fix is applied to plugins bzr, git, patch.

Fixes #543 #544 #555.

Valentin David authored Aug 07, 2018 and Tristan Van Berkom committed Aug 12, 2018

Staging could end-up with file with different rights depending on the
umask. The extracted files need to get their access rights fixed.

Deterministic staging

Closes #543, #544, #555, and #527

See merge request !616

Chandan Singh authored Aug 12, 2018 and Javier Jardón committed Aug 12, 2018

`python3-arpy` package is not available on Fedora so it has to be
installed using `pip`. Update instructions accordingly.

For reference, here is a pipeline that failed when trying to install
`arpy` using current instructions:
https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/88553245.

doc: Fix install instructions for Fedora

See merge request !640

Add cachekey tests for the recently added `remote` source plugin to
ensure that future changes do not break API compatibility.

remote source: Add cachekey test

See merge request !636

Valentin David authored Aug 08, 2018 and Tristan Van Berkom committed Aug 13, 2018

Stream._load() now returns early without resolving build pipeline when
only tracking. Resolving track pipelines does not require to fully
load project configurations when when elements to track are only
junctions.

However build pipelines require to fully load project
configurations. This might not be possible in the case a project
configuration includes a file from a junction that yet needs to be
tracked.

Fixes #565.

Fix tracking of junctions used in project.conf

Closes #565

See merge request !619

Valentin David authored Aug 13, 2018 and Valentin David committed Aug 13, 2018

This allows to use version 0.15.51 rather than 0.15.0 which is
required for Python 3.

Fixes #571.

Set version of ruaml.yaml to at least 0.15.41 but strictly less than 0.15.52.

Closes #571

See merge request !646

Caching failed builds and relative workspaces were not part of the 1.1.5
developer snapshot.

Format version raised because of a change to the core element format.

Resolve "Make dependency type default to build"

Closes #463

See merge request !633

Valentin David authored Aug 14, 2018 and Valentin David committed Aug 14, 2018

We now delay refreshing submodules until we have a clone, that is after
we have fetched the main fetcher.

Fixes #537

Delay refreshing git submodule until we have a clone

Closes #537

See merge request !656

Qinusty authored Aug 15, 2018 and Qinusty committed Aug 15, 2018

This should avoid current issues accessing the necessary sources which
are blocking CI currently.

Fix CI - ftp.gnu.org unreachable

See merge request !665

Qinusty authored Aug 03, 2018 and Qinusty committed Aug 15, 2018

This allows for users to configure fatal-warnings to be either a list of
warnings.

This commit deprecates the use of fail-on-overlap within project.conf,
this will now use the fatal-warnings configuration item.

element.py: Cache key calculation now takes into account all of the
fatal-warnings

tests: This modifys the tests/frontend/overlaps.py tests to support the
new fatal-warnings configuration. Backwards compatibility is also
tested for `fail-on-overlap`

_versions.py: BST_FORMAT_VERSION bumped to 15 for fatal-warnings
	      BST_CORE_ARTIFACT_VERSION bumpted to 5 for fatal-warnings

Fixes: #526

Qinusty authored Aug 09, 2018 and Qinusty committed Aug 15, 2018

This follows the implementation of configurable warnings.

Qinusty authored Aug 08, 2018 and Qinusty committed Aug 15, 2018

This includes detailing the use of `fatal-warnings` within project.conf

Qinusty authored Aug 09, 2018 and Qinusty committed Aug 15, 2018

This adds multiple tests for custom plugin warnings and core warnings,
providing checks for both cases which should cause warnings and errors
when configured as fatal.

Also adds tests for cache key calculations.

Configurable Warnings

Closes #526

See merge request !627

Continue to accept requests from non-conforming BuildStream clients for
the time being to ease migration.

This requires an updated server.

Fixes #572.

Replace assertions with gRPC error responses.

CAS: Fix resource_name format for blobs

Closes #572

See merge request !660

CAS-backed virtual directory implementation

See merge request !481

Source plugin implementations can now specify that they need access to
previously staged sources by specifying
`BST_REQUIRES_PREVIOUS_SOURCES_TRACK` and/or
`BST_REQUIRES_PREVIOUS_SOURCES_FETCH`, corresponding to access at `track`
and `fetch` times respectively.

Fixes #381.

Replaces !505. For relevant discussion, see this discussion:
!505 (comment 83780747)

`pip` source plugin can stage python packages that are either specified
directly in the element definition or picked up from `requirements.txt`
from previous sources. In order to support the latter use-case
(which is also the primary motivation for this plugin), this plugin
requires access to previous sources and hence is an example of a
Source Transform source.

Also, bump `BST_FORMAT_VERSION` as this patch adds a new core plugin.

Allow source plugins to access previous sources

Closes #381

See merge request !568

Valentin David authored Aug 14, 2018 and Tristan Van Berkom committed Aug 16, 2018

Ostree mirrors were not sharing the same local repository, so it was
impossible the request refs from the right local repository when data
was fetched from a mirror rather than upstream.

Instead of having several repository with one remote each, we now
have one repository with several remotes.

This fixes #538.

Fix ostree repository mirroring

Closes #538

See merge request !658

This patch mitigates the granularity of the mtimes used on the default
gitlab runners, Allowing the test suite to pass on these runners.

Trying to mitigate a file system issue

See merge request !595

https seems broken on the Debian image:

START   autotools/hello/42930621-fetch.499.log
START   Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
FAILURE Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
FAILURE tar source at hello.bst [line 16 column 2]: Error mirroring https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:720)>

Use http instead of https for the gnu ftp mirror

See merge request !676

os.errno does no longer work with Python 3.7

Closes #577.

cascache: use errno module

Closes #577

See merge request !674

Tristan Van Berkom authored Aug 16, 2018 and Tristan Van Berkom committed Aug 17, 2018

Use the optionally implemented desktop notification method
to notify when a session completes.

Previously, this used to only notify when an interactive prompt
appears.

This is an attempt to fix #385

_frontend/app.py: Notify session completions

Closes #385

See merge request !672

Raise a error at configure time if the track and ref properties are
not present in the sources.

This is to address #471
that documented unhelpful behaviour when tracking git sources. However
the issue was also identified in ostree.

Updating the no track or ref fix to match the buildstream style

See merge request !628

Qinusty authored Aug 14, 2018 and Qinusty committed Aug 17, 2018

Fixes #531: Jobs were retrying when terminated, this lead to the process
being spawned again and starting up fresh.

Prevent jobs retrying on terminate

Closes #531

See merge request !662

This just adds one option, "remote-execution/url". Affects multiple files.