Add test to ensure gracefull handling of exception thrown while loading
a pipeline due the python's max recursion depth being exceeded.

This is part of the work for issue #203

Because the RecursionError exception was introduced in Python 3.5, until we
drop support for for Python 3.4, we must use RuntimeError.

Phil/203 BuildStream crashes when dependency tree too deep

See merge request !512

tests/frontend/workspace.py: Added tests

_stream.py: Added functionality for workspace open -f

See merge request !549

Add a plugin that supports downloading files verbatim from a source with
an optional overridable filename and destination directory. Bumps bst
format version to 10.

Fixes #163

When the user provides a path for the filename parameter, provide a reason

Add remote source plugin

Closes #163

See merge request !541

William Salmon authored Jul 26, 2018 and Javier Jardón committed Jul 26, 2018

Daniel went through the Doc's etc and made a list of suggestions.
These are the result of the most easily patched suggestions and
one or two thing that I noticed as well.

Documentation typos and fixes

See merge request !569

job.py: Changes to the logic surrounding retry attempts and child process return codes
element.py, source.py: ElementError and SourceError also implement this change.

These exceptions now have an optional parameter of temporary which defaults to false. This will potentially break
backwards compatibility where exceptions were previously raised and a retry was intended.

To trigger a retry, one must now raise their SourceError or ElementError with temporary=True.

This aims to fix #397.

This follows the change in 67ecd97a05279a3b7570ad59f05bf0a5973ef04c.

This is following 5d6418a06a3a0262df5f6b9e6e1a36578515fd4a where
DownloadableFileSource was modified to raise 'temporary'
errors when relating to file downloading.

Further work needs to be done for the current grpc exceptions which are reraised.

Plugin.call() now takes fail_temporarily as an optional parameter,
when supplied it will cause subsequent failures to trigger temporary
errors as opposed to permanent errors.

This also extends Plugin.check_output() which makes use of Plugin.call()

The git plugin will now make use of the fail_temporarily parameter
to Plugin.call(), allowing failures to trigger a retry.

Temorary failures

Closes #397

See merge request !574

Since we backported the temporary/permanent failures patch for #397
into the `bst-1.2` branch, we need to adjust the since versions in master
down to 1.2.

This aims to fix issue #502.

cascache.py: Raise ArtifactError on grpc error

Closes #502

See merge request !576

James Ennis authored Jul 27, 2018 and Javier Jardón committed Jul 27, 2018

  This patch also removes displaying the projects that live within
  the buildstream-examples repository, as we wish to deprecate this
  repo and migrate the two examples.
  - #438 and #439

  In addition to this, both examples do not 'build' in their current
  state.

Refer readers to our tutorial before referring them to existing bst projects

See merge request !578

The separator is useful in source files other than _project.py

This is part of a later plan to implement mirroring without forcing
everyone to update their sources. We use the expected calls to
Source.translate_url() when running Source.configure() to extract the
aliases from the URL. Multiple aliases must be extracted because
sources exist that may fetch from multiple aliases (for example, git
submodules)

Later, we want to substitute another URI where the alias normally reads
from the project - We accomplish this by re-instantiating the Source
with the alias overrides passed as an argument to the constructor.

**KLUDGE WARNING**: This involves making the source store its "meta"
object so that it's possible to create a copy of the source inside the
fetch queue, instead of back when the pipeline was being loaded.

This adds the SourceFetcher class, which is intended for sources that
fetch from multiple URLs (e.g. the git source and its submodules)

Fix when fetching

In user config (buildstream.conf), it is set with the "default-mirror"
field.

On the command-line, it is set with "--default-mirror"

This fixes:
* Bzr repositories pulling from the branch they were created with.
* Bzr's _ensure_mirror() not actually checking that it successfully
  mirrored the ref.

When implementing fetching from mirrors, I encountered some problems
with the git source:

1. The mirror URL was using translate_url()'s output, so if a different
   alias was used, then fetching from the mirror would go to a different
   directory, and be inaccessible.
2. After fixing that, fetching was unable to pull from a URL other than
   the one used at repository creation, meaning it wouldn't actually
   pull from the mirror.

This is helpful if you want to test what happens when you have one repo
that has diverged from another. By copying the repo you're sure they
start with shared history.

This is especially useful when mirroring.

Resolve "Support for downloading sources from mirrors"

Closes #328

See merge request !404

Qinusty authored Jul 23, 2018 and Qinusty committed Jul 27, 2018

This address issue #491.

When attempting to run buildstream with a configuration specifying
a cache quota larger than your available disk space, buildstream
will alert the user and exit.

Note:
This takes into consideration your current cache usage and
therefore restricts the overall size of your artifact cache folder.

Cache quota is now restricted to available disk space

See merge request !563

Source plugin implementations can now specify that they need access to
previously staged sources by specifying
`requires_previous_sources_track` and/or
`requires_previous_sources_fetch`, corresponding to access at `track`
and `fetch` times respectively.

Fixes #381.

Replaces !505. For relevant discussion, see this discussion:
!505 (comment 83780747)