Chandan Singh authored Dec 29, 2018 and Chandan Singh committed Jan 02, 2019

https://github.com/sphinx-doc/sphinx/issues/4375 has been closed as
`wontfix`. Update the comment in doc/Makefile that mentions it, based on
the discussion on the issue.

doc/Makefile: Update comment about sphinx entrypoint

See merge request !1026

We need to bump the testsuite image versions to use the new
pycodestyle update that allows excluding file paths, so that we can
exclude generated python code.

Javier Jardón authored Aug 10, 2018 and Tristan Maat committed Jan 02, 2019

pep8 package was renamed to pycodestyle to reduce confusion.

Disabled warnings:
W504 - Line break after binary operator (seems people like this)
W605 - Invalid escape sequence (some of our regexes use these)

Use pycodestyle instead pep8 python module

See merge request !638

Add `.in` and `.txt` requirements files for BuildStream's pure python
dependencies. For each pair, the `.in` file is supposed to capture the
loose version requirements, and the corresponding `.txt` file is
supposed to have frozen requirements. We have 3 such sets:

* `requirements`: BuildStream's runtime dependencies
* `dev-requirements`: Dependencies for running tests
* `plugin-requirements`: Dependencies for core plugins

Note that the frozen requirements files will only be used for testing
purposes, and `setup.py` will continue to read loose requirements.

Add `tox.ini` file that will enable us to use
[tox](https://tox.readthedocs.io/) as a frontend for running tests.
Since we share the config via `setup.cfg` and requirements via
requirements files, commands like `python3 setup.py test` will continue
to work.

Instead of invoking tests throung `setup.py`, use `tox` as a frontend in
the CI pipelines.

Since we don't allow use of site packages by default in our `tox`
configuration, people will need to install non-python build dependencies
of some of our dependencies that do not provide pre-built wheels. We
have two such packages at the moment:

* psuitl: requires python C headers, compiler
* pygobject: requires python C headers, pkg-config, compiler, cairo
  headers, and gobject libraries

These new `.in` and `.txt` are making the repository look very
cluttered. Move them to a separate `tools` directory to make it look a
bit cleaner.

Currently the CI and the docs both have to duplicate the same inforation
about how to gather dependencies etc, and have to use hacky ways to run
them.

Add a new `docs` environment to our tox setup so that building docs is
as simple as running `tox -e docs`.

When we update our dependencies, developers will eventually need to
recreate their `tox` environments to get correct results. This happens
because `tox` isn't particularly good at recognizing changes in the
requirements files.

Use tox as a frontend for running tests

See merge request !1027

Chandan Singh authored Jan 03, 2019 and Chandan Singh committed Jan 03, 2019

In !1028, we added a test specifically to test
that BuildStream correctly raises an warning when the name of an element
contains characters that are invalid on Windows. Unfortunately, we
didn't see it coming that it would make it impossible to checkout this
branch on Windows.

Fix it by generating this file, only if we are not running on Windows.

* tests/testutils/site.py: Add `IS_WINDOWS` check
* tests/frontend/buildcheckout.py: Generate file with invalid filename
  on the fly
* Remove tests/frontend/project/elements/invalid-chars.

Fixes #842.

Note that this may still cause issues on WSL when running tests on a
shared filesystem, but that seems to be a generic issue on WSL with
`os.rename`.

tests/frontend/buildcheckout.py: Fix bad filename issue for Windows

Closes #842

See merge request !1032

This should save us some cycles in CI, and also allow the
developer to more conveniently lint separately from testing.

Apparently you need to spell Jinja2 with a capital J, otherwise
we find it listed twice in the resulting requirements.txt files.

We no longer run the linters through pytest, so we don't need these
plugins anymore.

Run the linter separately from the tests

See merge request !1033

Tom Coldrick authored Dec 20, 2018 and Tristan Van Berkom committed Jan 03, 2019

In _yaml.py and _frontend/complete.py we were getting pylint warnings
for using collections.Mapping and collections.Iterable, which are
abstract classes now provided from collections.abc. This patch just
uses the classes from the right place.

Use collections.abc for Mapping, Iterable

Closes #831

See merge request !1020

Moving fetch and track to the source command group accidentally dropped
the support for guessing targets for these commands when invoked from a
workspace directory. This brings it back.

_frontend/cli.py: Reinstate support for guessing targets

See merge request !1036

Chandan Singh authored Jan 03, 2019 and Tristan Van Berkom committed Jan 04, 2019

This seems like a better name for the directory, as it more closely
describes the purpose of its contents.

Chandan Singh authored Jan 03, 2019 and Tristan Van Berkom committed Jan 04, 2019

Split the "The MANIFEST.in and setup.py" section in two:
"Managing data files" and "Updating BuildStream's Python dependencies".

Briefly explain the layout of `requirements` directory and add
instructions to use the Makefile added in the last commit.

Add Makefile to update requirements files

See merge request !1035

As we now run tests using `tox`, we don't need to worry about manually
packing and unpacking BuildStream. So, we can remove the preapre stage
entirely.

Update `coverage` and nightly jobs to appropriately cope with this
change. Both these jobs now install all runtime dependencies from
requirements files.

.gitlab-ci.yml: Remove prepare stage

See merge request !1037

Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")

_frontend/app.py: Use buildtree for interactive shell on build failure

See merge request !1039

  * utils.py:_kill_process_tree(): Ignore NoSuchProcess errors

    These are caused because we issue SIGTERM, and if the process
    has not exited after a timeout, we kill it.

  * _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
    redundantly, they are already ignored.

It seems that we were ignoring it after sleeping when terminating
tasks from the scheduler... but we were not ignoring it when performing
the same pattern in the `Plugin.call()` -> `utils._call()` path, so
we were still getting these exceptions at termination time from host
tool processes launched by source plugins.

Fix stack traces discovered with ^C forceful termination.

See merge request !1043

This is not used anywhere outside of the Scheduler, currently
only the Scheduler itself is allowed to queue a job at this level.

If the highlevel business logic for automatic queueing of auxiliary
jobs moves to another location, we can make this public again.

When queuing the special cache management related cleanup and
cache size jobs, we now treat these jobs as special and do the
following:

  * Avoid queueing a cleanup/cache_size job if one is already queued

    We just drop redundantly queued jobs here.

  * Ensure that jobs of this type only run one at a time

    This could have been done with the Resources mechanics,
    however as these special jobs have the same properties and
    are basically owned by the Scheduler, it seemed more straight
    forward to handle the behaviors of these special jobs together.

This fixes issue #753

This changes the deepest callback from when a Job completes to
propagate a JobStatus value instead of a simple boolean, and updates
all of the effected code paths which used to receive a boolean
to now handle the JobStatus values.

This further improves the situation for issue #753, as now we avoid
queueing cache size jobs for pull jobs which are skipped.

This is redundant now that we report it through the JobStatus.

Only queue one cache size job

Closes #753

See merge request !1040

Since we added batch commands, the batch commands print the
text of the commands directly in the message text, but this is wrong.

The detail string is the appropriate place for text of unknown lengths
(the user can actually configure how many max lines of commands they
want to see in their log), the message text itself should be controlled
and brief enough to avoid text wrapping.

sandbox/sandbox.py: Fix regression of command logging

See merge request !1044

Phil Dawson authored Jan 07, 2019 and Jürg Billeter committed Jan 08, 2019

This command has been replacved by the bst source checkout command

element.py: remove documentation reference to source bundle command

See merge request !1041

When code is faulty, in a plugin or in the core, we should
get a stack trace and a BUG message.

sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.

See merge request !1046

Valentin David authored Dec 20, 2018 and Valentin David committed Jan 09, 2019

Fixes #631.

Valentin David authored Dec 20, 2018 and Valentin David committed Jan 09, 2019

Fixes #780.

Remote execution configuration: HTTPS and user configuration

Closes #780 and #631

See merge request !1030

ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.

Fixes #850

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

Closes #850

See merge request !1047

It was 'instance-name' in the documentation.

Rename 'instance_name' option to 'instance-name'

See merge request !1048

This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.

When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.

  * .gitignore: Added new .coverage-reports/ directory

  * .gitlab-ci.yml: Rely on tox to combine and report coverage, only
    tell tox about the COVERAGE_PREFIX so that results can be namespaced
    by CI job name.

    This change also publishes the sources and final combined `.coverage`
    file in an output gitlab artifact for inspection, and lists some missing
    dependencies to the `coverage` job.

  * tox.ini: Add comments and refactor main [testenv] section so that
    other environments dont inherit too much unrelated cruft.

    Generate the coverate reports in the respective {envtmpdir} so that
    all per-process coverage files are prefixed with a full path, ensuring
    that concurrent runs don't mix reports and addressing concerns
    raised in #844.

    Also implemented new `tox -e coverage` environment to combine
    any found coverage and print a report.

  * .coveragerc: Omit .tox/ directory from coverage stats

Fixes issue #844

Also point out that it is possible to run test environments in
parallel using the `detox` tool.

Allow using detox, and fix/refactor collection of coverage reports

Closes #844

See merge request !1051

  * Omit versioneer's _version.py

  * Omit our __main__.py which is used only internally for generating
    documentation

.coveragerc: Omit some things which are irrelevant to cover.

See merge request !1052

Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).

Accept common architecture aliases for the sandbox config for
consistency with arch options.

Accept architecture aliases

See merge request !1034

contributing: fix 'oprtation' and some other typos

See merge request !1055

The algorithm adds elements to a dictionary and then sorts the dictionary
by the discovered depths while recursing - using an OrderedDict is enough
to ensure a stable order.

This fixes `bst show --deps plan ...` reporting different results on
each invocation.

This fixes an aspect of #824

Added a function to report the list of elements which appeared in
a given queue in the order of their first appearance.

For each sample project and expected result order, this test ensures that:

  * bst show --deps plan: Always shows the expected build order.

  * bst source fetch: Always executes in the expected build order.

  * bst build: Always builds in the expected order (disabled).

The build part of the test is currently disabled as the scheduler
seems to be messing with the order even in a `--builders 1` scenario.

Make build plan element list stable

See merge request !1058

Angelos Evripiotis authored Jan 11, 2019 and Angelos Evripiotis committed Jan 11, 2019

Bump the version of pytest-cov, so that we pick up the fix for
the '--no-cov' option breaking in pytest 4.0:

    https://github.com/pytest-dev/pytest-cov/pull/230

This was the error you would get prior to this version:

    _pytest.warning_types.RemovedInPytest4Warning: config.warn has been
    deprecated, use warnings.warn instead

Although it says 'deprecated', it did result in
termination with stack trace.

Angelos Evripiotis authored Jan 11, 2019 and Angelos Evripiotis committed Jan 11, 2019

To help new folks get to grips with pytest, add some tips on other
frequently used features.

pytest-cov==2.6.1, contributing: '--last-failed' and '--no-cov' tips

See merge request !1059

Minor correction, looks like we're not observing this queue
otherwise we'd be seeing crashes.

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

See merge request !1063

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

Closes #858

See merge request !1064

During recent reorganizing of `.gitlab-ci.yml` to work with `tox`, seems
like we missed to update the image used by the overnight aarch tests,
meaning that they currently fail due to `tox` being missing from them.

While these tests will be skipped on MRs usually, here is an example of
what the tests look like if they are actually run with this change:
https://gitlab.com/BuildStream/buildstream/-/jobs/145449561.

Fixes #859.

Chandan/fix overnight aarch

Closes #859

See merge request !1065

Jürg Billeter authored Jan 10, 2019 and Tristan Van Berkom committed Jan 14, 2019

The path of the config file generated by testutils for completion tests
is passed as regular argument, not via COMP_WORDS. Use that config file
in complete_artifact() to ensure the test uses the right artifact
directory.

James Ennis authored Jan 09, 2019 and Tristan Van Berkom committed Jan 14, 2019

Since the artifact subcommand group has been created, we have been able
to tab complete both element names and artifact refs as arguments to
the artifact subcommands (e.g. bst artifact log), this commit adds a
test for this.

completions.py: Add a test for our artifact ref autocompletions

See merge request !1054

which include flatpak_repo plugin needed to build fdsdk

.gitlab-ci.yml: Use latest bst-external

Closes #861

See merge request !1068

This will help us check if BuildStream is working with the latest
version of dependencies, as per our constraints. This job is allowed to
fail but its failure should signal that we need to add stricter
constraints in some of our `.in` requirements files.

.gitlab-ci.yml: Add job to attempt to update dependencies

See merge request !1038

We already have tests for python 3.5 and 3.6 but not 3.7.

Fixes #838.

.gitlab-ci.yml: Add tests for python 3.7

Closes #838

See merge request !1074

Chandan Singh authored Jan 15, 2019 and Javier Jardón committed Jan 15, 2019

Starting from `pytest` version 4.1.0, `Node.get_marker()` has been
removed, and hence our tests break when running with newer versions of
`pytest`. It was deprecated since a while back but it has recently been
removed completely. Use `get_closest_marker()` as a replacement that is
suggested in the changelog, and seems to work fine for our use case.

See https://github.com/pytest-dev/pytest/pull/4564 for more context on
the upstream issue.

One way of verifying this change is that this should fix the recently
added `tests-fedora-update-deps` job, that was failing before due to
this issue.

conftest.py: Don't use deprecated get_marker() function

See merge request !1073

Valentin David authored Dec 10, 2018 and Javier Jardón committed Jan 15, 2019

Requests with error UNAVAILABLE are retried directly. They are
typically issue with setting connection which can happen when the
server is slow at accepting new connections.

Requests with errors ABORTED are converted to temporary exceptions so
that the scheduler can retry the task.

Fixes #815