The link explaining python's requirements files was formatted
incorrectly (missing underscore at the end). This resulted in the text
being rendered literally, as opposed to being converted into a
hyperlink.

CONTRIBUTING.rst: Fix formatting of link to pip docs

See merge request !1094

This commit also ensures that if we try to use the 'old' commands,
BuildStream will fail and instruct the user to use the new command.

'artifact checkout' has slightly different behaviour from 'checkout',
that is, either '--directory' or '--tar' are now required options.
This is a step towards allowing checkout to take multiple args.

This commit marks 'bst checkout' as a 'hidden' command. If used,
the user will be prompted to use the new 'bst artifact checkout'
command.

All tests which used 'bst checkout' have been modified to use
the new artifact sub-command.

This partially solves #822.

Due to an upstream click_man issue:
https://github.com/click-contrib/click-man/issues/10
generating the man pages with our current setup.py only generates
a man page for bst-artifact-server, our first entry point.
I then had to remove this entry point from setup.py and regenerate to
obtain man pages for the commands in cli.py

Move push/pull/checkout to the artifact subcommand group

See merge request !1045

Click 7.0 allows us to declare commands as 'hidden'.
However, sphinx-click and click-man still generate these hidden
commands in the documentation and man pages, respectively.

This is a stop gap solution until the upstream issues (tracked in
issues #879 and #881) have been addressed.

Add an obsoletion note to our deprecated commands.

See merge request !1097

This fixes a recent regression introduced in c2fc2a5e

_scheduler: Fix dont display a failure for terminated jobs

See merge request !1096

This test has one test case which is marked as an integration test,
and the other is not an integration test, but was using the integration
cli. The integration cli does not work correctly if not run in
integration mode.

This was causing an error locally in conftest.py when trying to
create a tmpdir inside a nonexisting integration cache directory.

tests/integration/pullbuildtrees.py: Fix the non-integration case.

See merge request !1098

Chandan Singh authored Jan 21, 2019 and Chandan Singh committed Jan 23, 2019

https://github.com/rtfd/sphinx_rtd_theme/pull/672 has been fixed
upstream, and the newer versions of `sphinx_rtd_theme` do not break
search functionality with Sphinx >= 1.8.

tox.ini: Unpin sphinx dependency

See merge request !1093

Instead of sitting mysteriously alone in internals/utils.py

This is not an error related to loading data, like a parse error
in the quota specification is, but a problem raised by the artifact
cache - this allows us to assert more specific machine readable
errors in test cases (instead of checking the string in stderr, which
this patch also fixes).

This also removes a typo from the error message in the said error.

  * tests/artifactcache/cache_size.py

    Updated test case to expect the artifact error, which consequently
    changes the test case to properly assert a machine readable error
    instead of asserting text in the stderr (which is the real, secret
    motivation behind this patch).

  * tests/artifactcache/expiry.py: Reworked test_invalid_cache_quota()

    Now expect the artifact error for the tests which check configurations
    which create caches too large to fit on the disk.

Tristan/insufficient storage error

See merge request !1102

Instead of a single target, we can always provide a single target
in a list.

Reduce number of calls to `bst show` in tests

See merge request !1103

Tristan Van Berkom authored Jan 24, 2019 and Tristan Van Berkom committed Jan 24, 2019

This tests how BuildStream reacts when it fails to write the tracking
results to the element files or project.refs file, which is an operation
that plugins do not play a part in.

As such, removing the per repo kind parameterization from this test
as multiple runs are redundant here.

test_track_error_cannot_write_file() fixup

See merge request !1104

Jonathan Maw authored Jan 04, 2019 and Jürg Billeter committed Jan 24, 2019

Previously, it merely tested that buildstream did not fall other,
rather than whether it did anything useful.

Jonathan Maw authored Jan 07, 2019 and Jürg Billeter committed Jan 24, 2019

Previously, there was no way of detecting whether fetching happened, as
an element with an open workspace will not be fetched.

Add tests to cover reinstated support for guessing targets

See merge request !1042

There is no reason to disallow guess_element() for bst build --all.

Phillip Smyth authored Dec 12, 2018 and Jürg Billeter committed Jan 24, 2019

_versions.py: Bump format version

_frontend/cli.py: Use new methods.

Based on patches by Phillip Smyth.

This filters out junctions from the list of targets.

Junctions cannot be built, pulled, or pushed. Specifying a junction on
the command line for these commands will result in an error. However,
junctions may be in the list of default targets, so they need to be
ignored for build, pull, and push commands.

Test that `bst build` does not fail in a project where the list of
default targets includes a junction (junctions cannot be built).

Add support for default targets

See merge request !925

Don't require the interest to be registered, just discard any interest,
this function just sets a bit in a mask, and is not intended to maintain
a balance like the reserve() function is.

When running any session that has Queues which require Resource.CACHE,
check if our loaded estimated size exceeds the quota, and if so;
lock the Resource.CACHE resource exclusively right away and run
an exclusive initial cache size job.

This ensures we cleanup first before doing anything which might
add to the cache at startup time, if deemed needed.

This is a partial fix for issue #737

In order to test when core activities occur by parsing the stderr
in tests, we should make the messages conform more.

At the same time, this restores alignment of columns in core
messages with the element processing related messages.

Also, _scheduler/scheduler.py is updated to make it's activity names
conform to the (current) 5 character limit for the sake of alignment.

The tests/frontend/logging.py test gets it's regexes updated for
the log lines it checks for in stderr.

Cache management fixes

See merge request !1091

The artifact cache emits messages, and we want to allow that in preflight.

We can streamline this call to os.statvfs() in a few places.

This will benefit from a better UtilError being raised, and
and turns the artifact cache's local function into a one liner.

The loop which finds the first existing directory in the
given path has been removed, being meaningless due to the
call to os.makedirs() in ArtifactCache.__init__().

The local function was renamed to _get_cache_volume_size() and
no longer takes any arguments, which is more suitable for the
function as it serves as a testing override surface for
unittest.mock().

The following test cases which use the function to override
the ArtifactCache behavior have been updated to use the new
overridable function name:

   tests/artifactcache/cache_size.py
   tests/artifactcache/expiry.py

A simple object which creates a snapshot of current
usage statistics for easy reporting in the frontend.

A frontend facing API for obtaining usage statistics.

I would have put this on Stream instead, but the Context
seems to be the de facto place for looking up the artifact cache
in general so let's put it here.

This also adds some comments around the main status bar heading
rendering function.

Added some useful status messages when:

  * Calculating a new artifact cache usage size

  * Starting a cleanup

  * Finishing a cleanup

Also enhanced messaging about what was cleaned up so far when
aborting a cleanup.

Updates the known cache size in the main process while the cleanup
process is ongoing, so that the status indicators update live
while the cleanup happens.

This seems to have been copy/pasted from cascache, and
documents the function to possibly return None if defer_prune
was specified, but this function does not expose defer_prune.

Cache management logging enhancements

See merge request !1105

profile_start() and profile_end() calls have been incorporated into
Scheduler.run()

profile_start() and profile_end() calls have been added to
Stream.load_selection()

Add new 'scheduler' and 'load-selection' profiling topics

See merge request !1088

Instead only rely on the headroom to be enough to protect against
out of space conditions. The headroom can become configurable as
a separate step is required.

The changes to achieve this are:

  * Rename ArtifactCache.has_quota_exceeded() to ArtifactCache.full().

  * ArtifactCache.full() now also reports True if the available
    space on the artifact cache volume is smaller than the headroom.

    This ensures jobs get triggered to cleanup the cache when
    reaching the end of the disk.

  * When loading the artifact quota, it is now only an error if
    the quota exceeds the overall disk space, not if it does not
    fit in the available space.

    It is still a warning if the quota does not fit in the
    available space on the artifact cache volume.

  * Updated scheduler.py and buildqueue.py for the API rename

  * tests: Updated the artifactcache/expiry.py test for its
    expectations in this regard.

    Added a new test to test an error when quota was specified to
    exceed total disk space, and adjusted the existing tests to
    expect a warning when the quota does not fit in the available
    space.

This fixes issue #733 and #869.

_artifactcache.py: Don't require the quota to be available on disk.

Closes #869 and #733

See merge request !1106

Chandan Singh authored Jan 24, 2019 and Chandan Singh committed Jan 25, 2019

Previously, one had to manually install `click-man` package, and
remember to run the correct command. Now, we can simply run `tox -e man`
to update the man pages.

Chandan Singh authored Jan 24, 2019 and Chandan Singh committed Jan 25, 2019

Notable changes:

* New `source` and `artifact` command groups
* Man pages corresponding to obsolete commands, that were marked as
  hidden in Click, have now been removed.

Fixes #881.

Chandan Singh authored Jan 24, 2019 and Chandan Singh committed Jan 25, 2019

Simplify our docs, by requesting users to run `tox -e man` to update man
pages, instead of manually installing `click-man` and running the
command manually.

Chandan Singh authored Jan 25, 2019 and Chandan Singh committed Jan 25, 2019

If the `man` directory is empty, then it won't be copied in the source
distribution, and `list_man_pages()` will throw an exception when trying
to list files in a non-existent directory. This prevents us from
installing the BuildStream package when the man pages are not there.

The most common use-case for this is when we want to re-generate the man
pages but want to install the package before re-generating them.

Generate man pages using tox & update them

Closes #880 and #881

See merge request !1107

Now that 'bst track' is obsolete, change guidance to refer to the
replacement 'bst source track' instead.

Now that 'bst fetch' is obsolete, change guidance to refer to the
replacement 'bst source fetch' instead.

Fixup refs to 'bst track' and 'bst fetch'

See merge request !1086

use_artifact_config added as an optional default arg, allowing
for loading of given elements artifact remote config.

Provide bst shell --use-buildtree the ability to attempt to
acquire missing buildtrees, given respective option, user
pull-buildtree context and remote availability.

_frontend/cli.py: Refactor logic for determining --use-buildtree
option with given opportunity to attempt pulling a non-local
buildtree. Element loaded with artifact_config to allow remote
querying.

_stream.py: With given user option and element state, construct
PullQueue to fetch remote buildtree. Continue or Error without
buildtree if cannot be attained.

tests/integration/build-tree.py: Update to support new usecases

Download buildtrees on demand for bst shell --use-buildtree

Closes #829

See merge request !1050

This is needed since 0f2bc375

This is needed since 629a6e52, and was lost in the conversion to requirements.in

requirements/requirements.in: update minimum versions

Closes #884

See merge request !1114

Valentin David authored Jan 22, 2019 and Jürg Billeter committed Jan 28, 2019

Job can complete before we return from `Job.span()` to
`Scheduler._spawn_job()`, so that `_active_jobs` would not yet contain
the job.

This would print a stack on the console and try to run a second time
the job which can have unexpected effects.

In order to reproduce the issue, in
`buildstream/_scheduler/jobs/job.py`, in `Job.spawn`,
add a call to `time.sleep()` right before call to
`asyncio.get_child_watcher()`.

This fixes issue #857.

Fix crash when spawned job completes very fast

Closes #857

See merge request !1095

Valentin David authored Jan 23, 2019 and Jürg Billeter committed Jan 29, 2019

Fixes #882.

Fix type of error codes in CAS server

Closes #882

See merge request !1099

Valentin David authored Jan 21, 2019 and Jürg Billeter committed Jan 29, 2019

Fixes #873

Make sure testing cache directory exists

Closes #873

See merge request !1092