.gitlab-ci.yml: Add job to attempt to update dependencies

See merge request !1038

We already have tests for python 3.5 and 3.6 but not 3.7.

Fixes #838.

.gitlab-ci.yml: Add tests for python 3.7

Closes #838

See merge request !1074

Chandan Singh authored Jan 15, 2019 and Javier Jardón committed Jan 15, 2019

Starting from `pytest` version 4.1.0, `Node.get_marker()` has been
removed, and hence our tests break when running with newer versions of
`pytest`. It was deprecated since a while back but it has recently been
removed completely. Use `get_closest_marker()` as a replacement that is
suggested in the changelog, and seems to work fine for our use case.

See https://github.com/pytest-dev/pytest/pull/4564 for more context on
the upstream issue.

One way of verifying this change is that this should fix the recently
added `tests-fedora-update-deps` job, that was failing before due to
this issue.

conftest.py: Don't use deprecated get_marker() function

See merge request !1073

Found during #833.

`git rev-list --boundary tag..HEAD` unfortunately gives boundaries
that are deeper than should when there is a merge commit between `tag`
and `HEAD`. The common ancestory of the two parents of the merge is
a boundary instead of the parent of the branch that is not traversed.

`--ancestry-path` fixes this issue by restricting `git` traversing
those branches.

`git rev-list --boundary HEAD..HEAD` does not return any boundary.
So in this case we need to manually tag the HEAD as a boundary.

buildstream/_gitsourcebase.py: Reduce git history for git describe.

See merge request !1069

Small documentation/comment fixes in context.py

See merge request !1072

Other components will start to reply on cas modules, and not the artifact cache
modules so it should be organized to reflect this.

All relevant imports have been changed.

Part #802

Part of #802

List of methods moved
* Initialization check: made it a class method that is run in a subprocess, for
  when checking in the main buildstream process.
* fetch_blobs
* send_blobs
* verify_digest_on_remote
* push_method

Part of #802

Cas refactor

See merge request !1071

bst-docker-import: fix Chandan's name

See merge request !1076

This was just deadcode, it's not really used here.

To ensure we can run integration tests in parallel, use a tempdir
for the artifact cache of each separate integration test run.

This makes it possible to run multiple full test runs including
integration tests in parallel under detox.

Make integration tests parallelizable

See merge request !1077

Since the CAS refactor, we have not been able to execute
bst-artifact-server. This commit ensures that we can.

This fix closes #867

Ensure that we can execute `bst-artifact-server`

Closes #867

See merge request !1079

This branch makes the following changes:

  * jobs/job.py: No longer stores any interested resource list

    Jobs are ephemeral again, they only ever exist while they
    are running.

  * queues/queue.py: Revert to only handling lists of elements

    Elements pass through the queues, Queue.harvest_jobs()
    replaces Queue.pop_ready_jobs() and now the Queue stops
    creating jobs as soon as there are not enough resources
    for the job.

    Also removed unused `prepare()` abstract method.

  * queues/buildqueue.py: Adapt the part where we launch a job

    This part needs to be reworked anyway, just touch it up for
    now so that it doesnt break with the surrounding changes.

  * jobs/{cachesize,cleanup}job.py: Expose uniform complete callback

    Allows the scheduler to manage resource deallocation for these
    two job completions as a custom thing, at the same phase
    that the Queues take care of their own resource deallocation.

  * resources.py: No longer has knowledge of the job

    Since jobs are ephemeral, they are not a suitable place
    to store the resource identifiers, these must be provided
    by the callers wherever needed.

    Now the main Resources object is owned by the Scheduler
    but shared with Queues, each take care of managing the
    resources of the jobs they create through the same
    resource API.

  * scheduler.py: Reverted to only creating jobs on demand

    This changes the flow of the scheduler such that whenever
    jobs complete, the queues are interrogated for as many
    jobs which can run at the moment but not more; and this
    completely removes the waiting list.

    For the internal cache management jobs, we handle this
    with a little state instead of having a waiting list
    and only launch when the resources permit it.

By abolishing the scheduler waiting list and creating jobs
on demand, we fix the order of element processing and consequently
fix issue #712.

With the scheduler changes, fetch jobs get automatically skipped
so the output is changed, using a separate repo for each element
fixes the test such that every fetch job gets a job launched.

It was saying "There is not enough space to build the given element.",
this makes me think the error is associated to a specific element, but
this does not make sense to show up in a cleanup task.

Instead say "There is not enough space to complete the build.", which
should be more clear that even after cleaning up there is not enough
space.

Scheduler refactor, fix processing order

Closes #712

See merge request !1067

Use utils._tempdir() which in turn uses _signals.terminator() for
this purpose.

This also changes utils._tempdir() to use utils._force_rmtree()
so that it is safe to use for cleaning up the staging directory.

This patch fixes orphaned temporary directories being left behind
in the artifact cache directory at forceful termination time.

element.py: Cleanup temporary staging directories on termination

See merge request !1080

The tests/pipeline directory will be removed, and this test
is the better of the two but also redundant with the one in
tests/frontend/show.

Renamed existing test in show.py to `test_show_except_simple`
and added the better test as `test_show_except` below it.

Grouping bits of internal testing together here

This tests exactly the same thing that is tested in
tests/internals/pluginfactory.py (the new location
of tests/plugins/basics.py).

Now that the remaining test "filter.py" in the plugins directory
tests a specific element, it makes sense to create a place for
testing elements, just like we do for sources.

This used to be an internal test, converted this to use the `cli` fixture.

Created new `tests/format/iteration.py` which tests the order in
which elements are iterated over in various scopes in a loaded
data model.

The tests/format/project.py test already has some tests about how
we error gracefully for bad plugins and bad plugin configurations,
lets put it there rather than tests/pipeline/load.py which we
will remove.

This is where other load time related plugin error handling is
checked, and is the last thing to remove in the `tests/pipeline`
directory.

Remove some redundancy from the test lines.

The remaining test simply loads a project with one element
and asserts a value on it. This is already sufficiently tested
in tests/format/project.py.

This is the directory for all things related to loading.

General refactor in tests directory

See merge request !1062

We should only display commands in detail strings, not in the
message texts.

This also updates tests/integration/sandbox-bwrap.py to expect
the new message string which only contains the command exit status
and not the whole command itself, this does not alter the validity
of the text case which is checking that we can obtain the expected
return value.

sandbox/sandbox.py: Display failed commands in the detail string

See merge request !1081

Currently external plugins are maintaining their own copies of the testing
utilities they need.

We want external plugins to be able to make use of the core testing utils.
This commit exposes the basic utilities which are currently in use in
bst-external plugins. If necessary, more utilities could be exposed in the
future.

Moves the following files from tests/testutils/ to
buildstream/plugintestingutils/:
  * runcli.py
  * integration.py

In order to expose integration.py as part of teh public testing api,
we need to move it into the buildstream module. First we should fix
various linting errors.

In order to expose integration.py as part of teh public testing api,
we need to move it into the buildstream module. First we should fix
various linting errors.

This is needed so we can expose some of our testing utils as a public api for
use by plugin authors.