This should save us some cycles in CI, and also allow the
developer to more conveniently lint separately from testing.

Apparently you need to spell Jinja2 with a capital J, otherwise
we find it listed twice in the resulting requirements.txt files.

We no longer run the linters through pytest, so we don't need these
plugins anymore.

Run the linter separately from the tests

See merge request !1033

Tom Coldrick authored Dec 20, 2018 and Tristan Van Berkom committed Jan 03, 2019

In _yaml.py and _frontend/complete.py we were getting pylint warnings
for using collections.Mapping and collections.Iterable, which are
abstract classes now provided from collections.abc. This patch just
uses the classes from the right place.

Use collections.abc for Mapping, Iterable

Closes #831

See merge request !1020

Moving fetch and track to the source command group accidentally dropped
the support for guessing targets for these commands when invoked from a
workspace directory. This brings it back.

_frontend/cli.py: Reinstate support for guessing targets

See merge request !1036

Chandan Singh authored Jan 03, 2019 and Tristan Van Berkom committed Jan 04, 2019

This seems like a better name for the directory, as it more closely
describes the purpose of its contents.

Chandan Singh authored Jan 03, 2019 and Tristan Van Berkom committed Jan 04, 2019

Split the "The MANIFEST.in and setup.py" section in two:
"Managing data files" and "Updating BuildStream's Python dependencies".

Briefly explain the layout of `requirements` directory and add
instructions to use the Makefile added in the last commit.

Add Makefile to update requirements files

See merge request !1035

As we now run tests using `tox`, we don't need to worry about manually
packing and unpacking BuildStream. So, we can remove the preapre stage
entirely.

Update `coverage` and nightly jobs to appropriately cope with this
change. Both these jobs now install all runtime dependencies from
requirements files.

.gitlab-ci.yml: Remove prepare stage

See merge request !1037

Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")

_frontend/app.py: Use buildtree for interactive shell on build failure

See merge request !1039

  * utils.py:_kill_process_tree(): Ignore NoSuchProcess errors

    These are caused because we issue SIGTERM, and if the process
    has not exited after a timeout, we kill it.

  * _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
    redundantly, they are already ignored.

It seems that we were ignoring it after sleeping when terminating
tasks from the scheduler... but we were not ignoring it when performing
the same pattern in the `Plugin.call()` -> `utils._call()` path, so
we were still getting these exceptions at termination time from host
tool processes launched by source plugins.

Fix stack traces discovered with ^C forceful termination.

See merge request !1043

This is not used anywhere outside of the Scheduler, currently
only the Scheduler itself is allowed to queue a job at this level.

If the highlevel business logic for automatic queueing of auxiliary
jobs moves to another location, we can make this public again.

When queuing the special cache management related cleanup and
cache size jobs, we now treat these jobs as special and do the
following:

  * Avoid queueing a cleanup/cache_size job if one is already queued

    We just drop redundantly queued jobs here.

  * Ensure that jobs of this type only run one at a time

    This could have been done with the Resources mechanics,
    however as these special jobs have the same properties and
    are basically owned by the Scheduler, it seemed more straight
    forward to handle the behaviors of these special jobs together.

This fixes issue #753

This changes the deepest callback from when a Job completes to
propagate a JobStatus value instead of a simple boolean, and updates
all of the effected code paths which used to receive a boolean
to now handle the JobStatus values.

This further improves the situation for issue #753, as now we avoid
queueing cache size jobs for pull jobs which are skipped.

This is redundant now that we report it through the JobStatus.

Only queue one cache size job

Closes #753

See merge request !1040

Since we added batch commands, the batch commands print the
text of the commands directly in the message text, but this is wrong.

The detail string is the appropriate place for text of unknown lengths
(the user can actually configure how many max lines of commands they
want to see in their log), the message text itself should be controlled
and brief enough to avoid text wrapping.

sandbox/sandbox.py: Fix regression of command logging

See merge request !1044

Phil Dawson authored Jan 07, 2019 and Jürg Billeter committed Jan 08, 2019

This command has been replacved by the bst source checkout command

element.py: remove documentation reference to source bundle command

See merge request !1041

When code is faulty, in a plugin or in the core, we should
get a stack trace and a BUG message.

sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.

See merge request !1046

Valentin David authored Dec 20, 2018 and Valentin David committed Jan 09, 2019

Fixes #631.

Valentin David authored Dec 20, 2018 and Valentin David committed Jan 09, 2019

Fixes #780.

Remote execution configuration: HTTPS and user configuration

Closes #780 and #631

See merge request !1030

ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.

Fixes #850

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

Closes #850

See merge request !1047

It was 'instance-name' in the documentation.

Rename 'instance_name' option to 'instance-name'

See merge request !1048

This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.

When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.

  * .gitignore: Added new .coverage-reports/ directory

  * .gitlab-ci.yml: Rely on tox to combine and report coverage, only
    tell tox about the COVERAGE_PREFIX so that results can be namespaced
    by CI job name.

    This change also publishes the sources and final combined `.coverage`
    file in an output gitlab artifact for inspection, and lists some missing
    dependencies to the `coverage` job.

  * tox.ini: Add comments and refactor main [testenv] section so that
    other environments dont inherit too much unrelated cruft.

    Generate the coverate reports in the respective {envtmpdir} so that
    all per-process coverage files are prefixed with a full path, ensuring
    that concurrent runs don't mix reports and addressing concerns
    raised in #844.

    Also implemented new `tox -e coverage` environment to combine
    any found coverage and print a report.

  * .coveragerc: Omit .tox/ directory from coverage stats

Fixes issue #844

Also point out that it is possible to run test environments in
parallel using the `detox` tool.

Allow using detox, and fix/refactor collection of coverage reports

Closes #844

See merge request !1051

  * Omit versioneer's _version.py

  * Omit our __main__.py which is used only internally for generating
    documentation

.coveragerc: Omit some things which are irrelevant to cover.

See merge request !1052

Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).

Accept common architecture aliases for the sandbox config for
consistency with arch options.