Benjamin Schubert authored Oct 25, 2018 and Benjamin Schubert committed Oct 26, 2018

Currently we would make sure the sandbox had a command before
converting it to a list if it was given as a string. That meant that a string
command would never exist and the check be invalid.

This also adds the same logic in the dummy sandbox for consistency.

Check is command is a str and replace by list before checking existence

Closes #728

See merge request !898

Daniel Silverstone authored Oct 26, 2018 and Phil Dawson committed Oct 26, 2018

In cases where elements end up with large numbers of dependencies,
the n-squared performance in `_extract_depends_from_node` can produce
significant delays during project load.  This patch corrects that.
In addition we were double-retrieving provenance which was unnecessary.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

_loader/loadelement.py: Use enumerate() in dependency extraction

See merge request !901

More information regarding the use of systemd to manage the cache

See merge request !904

The `_sentinel` in `utils.py` was used only for yaml node tracking.
As such, simplify matters by removing it from `utils.py` and move it
to `_yaml.py` which means that we no longer need to try and avoid a
circular import issue by means of runtime importing.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Move _sentinel from utils.py to _yaml.py

See merge request !903

Fix some ruamel warnings during tests

See merge request !889

In order to support things which might need to use the stdout FD
after checkout has completed writing a tarball to it, preserve it
by means of `os.dup()` and `os.dup2()`

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

In order to test things which write tarballs to stdout correctly,
we need to capture the binary output cleanly.  This ensures we're not
potentially mismatching encodings in and out.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

With the new support for saving the stdout FDs cleanly, and binary mode
capture in runcli, enable the tarball-to-stdout test.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

Don't close stdout when running 'checkout' to stdout

Closes #486

See merge request !905

Tristan/architecture docs

See merge request !907

We shouldn't need it to persist now that we cache failed build dirs.

This change breaks the test
`tests/integration/shell.py::test_sysroot_workspace_visible`.
I can no longer see a use-case for this test.
AIUI, it tested that the failed build sysroot stored in the builddir has
the workspace's files in, despite the workspace being unmounted.
I believe this behaviour is made redundant by cached buildtrees.

This fixes part of #539

This includes changes in app.py:
* Interactive failure shell no longer uses the failed build sysroot,
  defaulting to the cached build tree.

Changes in element.py are:
* Errors caused by building don't store the failed build sysroot,
  instead storing that a sandbox can be created to debug the error.
* When staging sources, will stage the element's cached build tree if it
  exists.

Changes in _exceptions.py:
* BstError.sandbox is now a flag of whether a sandbox can be opened up
  to debug the error.

Changes in widget.py:
* Don't try to print any information about the sandbox.

Changes in _message.py:
* Fix documentation so Message.sandbox is not a directory any more.

This is part of #539

This is related to #539

i.e. instead of taking a directory that must contain "root" and
"scratch", and treating "root" as the root, use the directory directly.

In element.py:
* __sandbox takes the `bare_sandbox` arg, to pass into the sandbox's constructor

In sandbox.py:
* If bare_sandbox, `_root` is the passed-in directory, and `__scratch`
  is None.
* Trying to use `__scratch` when bare_sandbox is True is a bug.

In _mount.py:
* Don't get the value of `__scratch` if it's not needed.

This is part of #539

This is related to #539

Jonathan/debug remote failed builds

See merge request !869

_frontend/app.py: Set correct element-path in interactive bst-init

See merge request !912

Benjamin Schubert authored Oct 19, 2018 and Benjamin Schubert committed Nov 01, 2018

This includes a new command mimicking pytest-runner so that we
can drop this dependency

This was the only setup_requires dependency that we had and
will make like easier for people behind proxies

Remove dependency on pytest-runner

See merge request !886

Angelos Evripiotis authored Oct 18, 2018 and Tristan Van Berkom committed Nov 01, 2018

Remove the 'MAKEFLAGS' and 'V' defaults.

Now that there is a specialised 'make' element, these make-specific
defaults don't make sense here. This element is meant to be a blank
slate for folks to build on.

Note that this is a breaking change, that will require some users to
make changes to their .bst files if they are expecting these environment
variable to be set.

_versions.py: BST_FORMAT_VERSION bumped to 18.

This fixes issue #718

BREAK: manual.yaml: don't set any default env vars

Closes #718

See merge request !916

Since the core Python `ChainMap.get()` implements with:

    self[key] if key in self else default

The double-chain-lookup is expensive.  This simple change solves that
for our ChainMap derived structure.  As such it improves matters
for #466 somewhat.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

_yaml.py: Implement `get()` for `ChainMap`

See merge request !919

plugins/sources/pip.py: Accomodate characters '-','.','_' for packages

See merge request !914

Fixes: 825963b5 ("element: Make "--sysroot" take a bare directory")

element.py: Pass bare_directory also to SandboxRemote()

See merge request !928

Issue happens with sourceware.org server.

Fixes #742

Use less fetchers in overnight build to work-around limitations from servers.

Closes #742

See merge request !929

richardmaw-codethink authored Nov 05, 2018 and richardmaw-codethink committed Nov 05, 2018

"_frontend/app.py: Set correct element-path in interactive bst-init"
added a dependency on the tmp_path test fixture, which was added by 3.8.

dev-requirements: Update pytest dependency to 3.8

See merge request !930

Get the artifact cache from the context instead.

The artifact cache backend does not affect build outputs and we anyway
no longer have pluggable artifact cache backends. This hardcodes
CASCache instead of removing the entry completely to avoid cache key
changes.

This changes CASCache from a subclass to a delegate object of
ArtifactCache. As the lower layer, CASCache no longer deals with
elements or projects.

Fixes #659.

Split up artifact cache and CAS cache

Closes #659

See merge request !922

Fix issue with _pretty_size with large numbers of bytes

See merge request !799

Replaces the custom one which was erroneously added during development.
Fixes #746.

_sandboxremote.py: Use the standard SandboxError

Closes #746

See merge request !931

Update our instructions/examples regarding how we declare remote caches

See merge request !932

There are arguments for and against using onerror - see issue #735
for details.

Use _force_rmtree instead of custom code.

Closes #735

See merge request !927