This changes the deepest callback from when a Job completes to
propagate a JobStatus value instead of a simple boolean, and updates
all of the effected code paths which used to receive a boolean
to now handle the JobStatus values.

This further improves the situation for issue #753, as now we avoid
queueing cache size jobs for pull jobs which are skipped.

When queuing the special cache management related cleanup and
cache size jobs, we now treat these jobs as special and do the
following:

  * Avoid queueing a cleanup/cache_size job if one is already queued

    We just drop redundantly queued jobs here.

  * Ensure that jobs of this type only run one at a time

    This could have been done with the Resources mechanics,
    however as these special jobs have the same properties and
    are basically owned by the Scheduler, it seemed more straight
    forward to handle the behaviors of these special jobs together.

This fixes issue #753

This is not used anywhere outside of the Scheduler, currently
only the Scheduler itself is allowed to queue a job at this level.

If the highlevel business logic for automatic queueing of auxiliary
jobs moves to another location, we can make this public again.

Handle NoSuchProcess errors transparently inside the function, instead
of only from outside of it where Job uses it.

_frontend/app.py: Use buildtree for interactive shell on build failure

See merge request !1039

Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")

.gitlab-ci.yml: Remove prepare stage

See merge request !1037

As we now run tests using `tox`, we don't need to worry about manually
packing and unpacking BuildStream. So, we can remove the preapre stage
entirely.

Update `coverage` and nightly jobs to appropriately cope with this
change. Both these jobs now install all runtime dependencies from
requirements files.

Add Makefile to update requirements files

See merge request !1035

Chandan Singh authored 6 years ago and Tristan Van Berkom committed 6 years ago

Split the "The MANIFEST.in and setup.py" section in two:
"Managing data files" and "Updating BuildStream's Python dependencies".

Briefly explain the layout of `requirements` directory and add
instructions to use the Makefile added in the last commit.

Chandan Singh authored 6 years ago and Tristan Van Berkom committed 6 years ago

This seems like a better name for the directory, as it more closely
describes the purpose of its contents.

_frontend/cli.py: Reinstate support for guessing targets

See merge request !1036

Moving fetch and track to the source command group accidentally dropped
the support for guessing targets for these commands when invoked from a
workspace directory. This brings it back.

Use collections.abc for Mapping, Iterable

Closes #831

See merge request !1020

Tom Coldrick authored 6 years ago and Tristan Van Berkom committed 6 years ago

In _yaml.py and _frontend/complete.py we were getting pylint warnings
for using collections.Mapping and collections.Iterable, which are
abstract classes now provided from collections.abc. This patch just
uses the classes from the right place.

Run the linter separately from the tests

See merge request !1033

We no longer run the linters through pytest, so we don't need these
plugins anymore.

Apparently you need to spell Jinja2 with a capital J, otherwise
we find it listed twice in the resulting requirements.txt files.

This should save us some cycles in CI, and also allow the
developer to more conveniently lint separately from testing.

tests/frontend/buildcheckout.py: Fix bad filename issue for Windows

Closes #842

See merge request !1032

Chandan Singh authored 6 years ago and Chandan Singh committed 6 years ago

In !1028, we added a test specifically to test
that BuildStream correctly raises an warning when the name of an element
contains characters that are invalid on Windows. Unfortunately, we
didn't see it coming that it would make it impossible to checkout this
branch on Windows.

Fix it by generating this file, only if we are not running on Windows.

* tests/testutils/site.py: Add `IS_WINDOWS` check
* tests/frontend/buildcheckout.py: Generate file with invalid filename
  on the fly
* Remove tests/frontend/project/elements/invalid-chars.

Fixes #842.

Note that this may still cause issues on WSL when running tests on a
shared filesystem, but that seems to be a generic issue on WSL with
`os.rename`.

Use tox as a frontend for running tests

See merge request !1027

When we update our dependencies, developers will eventually need to
recreate their `tox` environments to get correct results. This happens
because `tox` isn't particularly good at recognizing changes in the
requirements files.

Currently the CI and the docs both have to duplicate the same inforation
about how to gather dependencies etc, and have to use hacky ways to run
them.

Add a new `docs` environment to our tox setup so that building docs is
as simple as running `tox -e docs`.

These new `.in` and `.txt` are making the repository look very
cluttered. Move them to a separate `tools` directory to make it look a
bit cleaner.

Since we don't allow use of site packages by default in our `tox`
configuration, people will need to install non-python build dependencies
of some of our dependencies that do not provide pre-built wheels. We
have two such packages at the moment:

* psuitl: requires python C headers, compiler
* pygobject: requires python C headers, pkg-config, compiler, cairo
  headers, and gobject libraries

Instead of invoking tests throung `setup.py`, use `tox` as a frontend in
the CI pipelines.

Add `tox.ini` file that will enable us to use
[tox](https://tox.readthedocs.io/) as a frontend for running tests.
Since we share the config via `setup.cfg` and requirements via
requirements files, commands like `python3 setup.py test` will continue
to work.

Add `.in` and `.txt` requirements files for BuildStream's pure python
dependencies. For each pair, the `.in` file is supposed to capture the
loose version requirements, and the corresponding `.txt` file is
supposed to have frozen requirements. We have 3 such sets:

* `requirements`: BuildStream's runtime dependencies
* `dev-requirements`: Dependencies for running tests
* `plugin-requirements`: Dependencies for core plugins

Note that the frozen requirements files will only be used for testing
purposes, and `setup.py` will continue to read loose requirements.

Use pycodestyle instead pep8 python module

See merge request !638

Javier Jardón authored 6 years ago and Tristan Maat committed 6 years ago

pep8 package was renamed to pycodestyle to reduce confusion.

Disabled warnings:
W504 - Line break after binary operator (seems people like this)
W605 - Invalid escape sequence (some of our regexes use these)

We need to bump the testsuite image versions to use the new
pycodestyle update that allows excluding file paths, so that we can
exclude generated python code.

doc/Makefile: Update comment about sphinx entrypoint

See merge request !1026

Chandan Singh authored 6 years ago and Chandan Singh committed 6 years ago

https://github.com/sphinx-doc/sphinx/issues/4375 has been closed as
`wontfix`. Update the comment in doc/Makefile that mentions it, based on
the discussion on the issue.