os.utime() updates both, however, list_artifacts() sorts refs by mtime,
i.e., atime is irrelevant.

get_artifact_fullname() is required to construct the ref. The cache key
alone does not suffice.

_artifactcache: Fix ref in update_mtime()

See merge request !882

Replace path separator with '-' to match Element.normal_name.

tests/testutils/artifactshare.py: Fix has_artifact() to match core code

See merge request !890

Angelos Evripiotis authored Oct 22, 2018 and Phil Dawson committed Oct 23, 2018

Remove two uses of ExitStack that aren't necessary and make the code
a little harder to follow.

plugins/sources: refactor, rm needless ExitStack-s

See merge request !888

The REAPI allows a client to reconnect to an ongoing operation stream by
providing a WaitExecution(). If implemented on server side, BuildStream
will try to recover from connection errors using it.

#630

Handle connection losses during remote build execution

See merge request !806

_downloadablefilesource: handle ValueError-s

Closes #452

See merge request !893

Improve our documentation for configuring an artifact server

See merge request !899

This pointless bare `return` was causing modern pylint to raise an
error.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The python `for`/`else` construct only makes sense if the `for` loop body
has a `break` statement.  Otherwise the use of `else` causes pylint
to raise an error.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

In `_parse_workspace_config` there was a chained version comparison
which is simplified in this commit to quieten a pylint issue.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Where we use a construct `val == foo or val == bar` we can instead use
`val in (foo, bar)` which pylint prefers.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

For Python before 3.6, `path.resolve()` could not take the `strict`
keyword argument.  Linting on such Python versions will raise an unnecessary
issue given the check present.  As such, quieten that lint.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The correct file path was computed but never returned.  This was
never caught because of a test missing consumption of one of its
parameters.  This commit resolves the bug mentioned in #722 leaving
the test to be amended by a future commit.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

This braino would mean that we'd fail to raise the right exception
if we failed to find the right project object when unpickling the
yaml cache.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The imports of _cachekey and utils were unused.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Unfortunately elements don't override `__init__` in general, so
we have to quieten pylint here.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Where we have conditions of the form `var == foo or var == bar` it
can be simplified to `var in (foo, bar)` which pylint prefers.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Unfortunately pylint can't understand that we'll always use subclasses
of Element and as such complains of a no-return function (`assemble()`)
having its return value used.  This quietens that warning.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Pylint recommends the use of dict.get() rather than a multi-line if
statement for handling optional keyword arguments.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

A number of times, abstract methods are called which pylint cannot
possibly understand will only ever happen in subclasses which implement
the abstract methods.  This silences those specific warnings

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Remove `source_kind` as it was unused.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Remove the `context` variable which was unused.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

We use `preexec_fn` in our use of Popen.  Since there are threads
in the codebase, this is not necessarily safe.  However we go to
great lengths to ensure that the main process doesn't spawn threads
(they're used in gRPC a lot) and so it should be safe for our use.
As such, we disable the lint here.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The imports of `stat` and `S_ISDIR` from `stat` should be grouped to
satisfy pylint.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The pylint report dislikes the superfluous parens which used to
be here in this assert.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The pylint tool is unable to understand that the abstract methods
in the artifact cache will only be called when there's concrete
implementations behind them.  Silence the lint errors for these
specific calls.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The `resource` import was unused.  Removed.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The `curses` import is "standard" whereas `click` is not.  As such, despite
alphabetical order normally being the rule, `curses` must come before `click`.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Since default values in arguments to functions and methods are created once
at the compilation of the code, it is dangerous to include list and dict
literals in them.  This changes the use of {} to None.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Unfortunately pylint can't tell this is an abstract method which will
only ever end up invoked from a class which concretises it.  As such
we need to silence this warning.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The pylint tool prefers that conditions of the form "var == foo or var == bar"
be refactored into "var in (foo, bar)".

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

An otherwise empty __init__ passing no arguments to super.__init__
is worthless.  Remove it.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The import of `resource` was unused.
The import of `PlatformError` was unused.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The imports of `Message` and `MessageType` were unused.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The linter dislikes an if/return/else/return where it can be
simplified to a return of the if condition.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The linter cannot detect abstract methods the way we use them, so
this silences the warning which ensues.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The `done()` method does not return anything anywhere else.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The `SourceError` import was never used

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The `assert_ref_in_track()` method's return value was never used so
rationalise it to never return a value.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The use of dictionary literals in argument defaults is disrecommended
due to the way that they are static and thus potentially very confusing.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

In Python 3, inheriting from `object` explicitly is pointless.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Sometimes `dict.get()` is preferable, in this instance it's less clear
so we'll disable that lint

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The variable was unused, remove it.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

We are super-careful to not use threads in places where we might use
Popen and as such this warning is save to quash.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Sometimes `dict.get()` is preferable to an `if` statement, but this time
it's clearer if we keep the structure.  As such, silence the warning.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Correct some multi-line conditional indentation by slight refactor
if necessary.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The lint tool dislikes the use of `if len(SEQ) == 0` type statements
because nominally `len(SEQ)` may be more expensive than `bool(SEQ)`.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Python linters prefer that attributes are defined in __init__ rather
than left to later routines.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The imports of `Message` and `MessageType` were unused.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

In order to quieten the linter, and to make a more correct comparison,
we switch to using the `Code.OK` enumeration from `google.rpc.Code`

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Tristan Maat authored Oct 24, 2018 and Daniel Silverstone committed Oct 25, 2018

We need to do this so that the new linting errors pop up

Versioneer needs to be in the MANIFEST.in and its .pyc needs to be ignored
in the .gitignore.  Also much docs were not being included, nor conftest.py.
Much of the test suite data files needed including, so changed that to a
raw include of everything in the tests/ tree.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

We need to alter how coverage data is recovered in order to work with
newer coverage tooling.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Fix lint issues

See merge request !892

Instead of redundantly documenting "directory" configuration option in
each source plugin, refer to the built-in functionality documentation of
the Source class.

  * source.py: Add anchor for built-in functionality.
  * sources/plugins/*.py: Remove directory config option, and add reference to
    Source's built-in functionality.

Fixes #713.

For each element plugin derived from BuildElement, add reference to the
built-in functionality documentation of BuildElement.

  * buildelement.py: Add anchor for built-in functionality.
  * plugins/elements/*.py: Add refence to BuildElement's built-in
    functionality for each build element.

Related to #713.

plugins: Don't redundantly document common configurations

Closes #713

See merge request !896

We were unconditionally inserting into the yaml cache anything we had
retrieved from the yaml cache.  This had the side-effect of being expensive
on re-runs of commands where yaml hadn't changed.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

_yaml.py: Do not insert into cache if retrieved from cache

See merge request !902

Benjamin Schubert authored Oct 25, 2018 and Benjamin Schubert committed Oct 26, 2018

Currently we would make sure the sandbox had a command before
converting it to a list if it was given as a string. That meant that a string
command would never exist and the check be invalid.

This also adds the same logic in the dummy sandbox for consistency.

Check is command is a str and replace by list before checking existence

Closes #728

See merge request !898

Daniel Silverstone authored Oct 26, 2018 and Phil Dawson committed Oct 26, 2018

In cases where elements end up with large numbers of dependencies,
the n-squared performance in `_extract_depends_from_node` can produce
significant delays during project load.  This patch corrects that.
In addition we were double-retrieving provenance which was unnecessary.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

_loader/loadelement.py: Use enumerate() in dependency extraction

See merge request !901

More information regarding the use of systemd to manage the cache

See merge request !904

The `_sentinel` in `utils.py` was used only for yaml node tracking.
As such, simplify matters by removing it from `utils.py` and move it
to `_yaml.py` which means that we no longer need to try and avoid a
circular import issue by means of runtime importing.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Move _sentinel from utils.py to _yaml.py

See merge request !903

Fix some ruamel warnings during tests

See merge request !889

In order to support things which might need to use the stdout FD
after checkout has completed writing a tarball to it, preserve it
by means of `os.dup()` and `os.dup2()`

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

In order to test things which write tarballs to stdout correctly,
we need to capture the binary output cleanly.  This ensures we're not
potentially mismatching encodings in and out.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

With the new support for saving the stdout FDs cleanly, and binary mode
capture in runcli, enable the tarball-to-stdout test.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Don't close stdout when running 'checkout' to stdout

Closes #486

See merge request !905

The addition of the pull-buildtrees flag will be used to override
the default behaviour of remote artifact cache pulling. Included
is the required changes to completions tests.

The addition of the pullbuildtrees option to the users global
buildstream.conf will be used to override the default behaviour
of remote artifact cache pulling.

The addition of cached buildtrees being included in element
artifacts has led to mostly redundant download overheads when
pulling from a remote artifact server. As such the default behaviour
of pull shouldn't fetch the buildtree object if available.

_stream.py: propagate if pullbuildtrees has been set via cli or
user config to the PullQueue.

pullqueue.py: add extensible attributes to determine required/
excluded directories for element pull jobs.

element.py: extend relevant pull logic for specified subdir
consideration and ensure push logic does not lead to partial
artifact pushes.

_artifactcache/: artifactcache.py & cascache.py inclusion of
helper functions for subdir artifact checking & fetching, fetch
logic extended to only pull required artifact directories.
extract & checkout updated to handle full/partial operation.

tests/: addition of integration test pullbuildtrees.py,
testutils/artifactshare.py has_artifact changed to return
artifact digest on true condition.