richardmaw-codethink authored Sep 05, 2018 and richardmaw-codethink committed Sep 13, 2018

This can get out of sync with other two cache states,
and we can do without it.

Simplify element state by removing `__cached`

See merge request !784

We can't include a socket in a CAS tree, but it's mostly meaningless to do so
since there can't possibly be a process serving it.

Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type."

See merge request !783

richardmaw-codethink authored Sep 04, 2018 and richardmaw-codethink committed Sep 14, 2018

Overriding the config with a custom config file on the command-line
prevents it merging in the test-specific configuration
and can permit it to attempt to initialise the user's cache.

richardmaw-codethink authored Sep 05, 2018 and richardmaw-codethink committed Sep 14, 2018

They were moving the whole tmpdir to move the project repository.
This moves the cache directories etc. as well, meaning cli.run can't find them.
This was worked around by setting configure=False,
but this has the side-effect of making use of the user's caches,
which it should not be doing for reproducibility reasons.

By changing the tempdir layout to have the project in a subdirectory
we can move the project around for the relative workspace tests
without losing track of the configured state directories,
so we can leave configure=True and avoid touching the user's caches.

Fix tests that attempt to access the home directory

See merge request !780

This was previously append_required_artifacts(), which presumed that
we knew at startup time what the cache keys of all elements in the
loaded pipeline would be.

This fixes unexpected deletions of required artifacts when
dynamic tracking is enabled with `bst build --track-all target.bst`

This commit renames test_never_delete_dependencies() to
test_never_delete_required(), renders the test more readable by renaming
some elements and reordering some statements and makes the comments more
straight forward and accurate.

These tests were not checking that we fail for the expected reasons.

Added `res.assert_task_error(ErrorDomain.ARTIFACT, 'cache-too-full')`
where we expect to fail because the cache is too full.

This allows one to modify a file in an existing git repo,
as opposed to adding a new one.

 * create_element_size()

   Now uses a git Repo object instead of a local source, and
   returns the repo.

 * update_element_size()

   Added this function which can now resize the expected output
   of an element generated with create_element_size(), useful
   to allow testing sized elements with the tracking feature.

Same test as test_never_delete_required(), except that this test ensures
that we never delete required artifacts when their cache keys are
discovered dynamically during the build.

Don't delete required artifacts when tracking is enabled

See merge request !793

We want to check if some file is already cached here, not the parent
directory.

Tristan Maat created the original file, so he is added as the author.

* Rename it to _commit_directory() because… it is what it does; and
  also for symmetry with _fetch_directory().
* Rename digest to dir_digest to make it clear this is a digest for a
  directory. A following commit will also reuse the same variable name
* Document method.

* Rename tree to dir_digest to make it clear this is a Digest object,
  and not a Tree object.
* Add documentation

Remove unneeded cruft.

Bunch of cleanups

See merge request !798

Somehow I missed this when originally forking the file from the click
library, now noticing that we should have followed what was written
in: https://github.com/pallets/click/blob/master/LICENSE

richardmaw-codethink authored Sep 10, 2018 and Tristan Van Berkom committed Sep 16, 2018

Since we now set PWD in the environment of builds
existing builds may behave differently so must cache differently now.

richardmaw-codethink authored Sep 11, 2018 and Tristan Van Berkom committed Sep 16, 2018

The current directory isn't always in the python module search path,
so we have to ensure it is for the script to work.

Strictly speaking, the user may already have a modified PYTHONPATH
at which point PYTHONPATH=".${PYTHONPATH+:$PYTHONPATH}" is necessary,
but it's probably premature to overcomplicate the documentation like that
before we discover it's a problem.

Address post-merge review of Ensure PWD is set in process environment

See merge request !788

Update contributing guide

See merge request !801

This ensures that option expressions are resolved in the project
level overrides before attempting to composite them on the instantiated
elements. Seems this is a regression from introducing the include
directive.

This fixes issue #658

This is a regression test for issue #658

Fix override options

Closes #658

See merge request !802

Since ArtifactCache.setup_remotes() can be expensive and should only
happen once, this commit will assert() if it is called a second time
on an artifact cache instance.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The initialization of remotes is done by ArtifactCache.setup_remotes()
and as such it was wrong for these tests to be calling
CASCache.initialize_remotes() a second time.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

The SandboxRemote used to construct its own CASCache which was
considered dangerous.  This patch replaces that with acquisition of
the cache via the Platform singleton, hopefully eliminating issues
from having more than one artifact cache object in a single process.

Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>

sandbox/_sandboxremote.py: Acquire cache via Platform

See merge request !797

When configuring a push remote and specifying either the client-cert
or the client-key, then both must be specified. This ensures we
get an informative error instead of a stack trace and BUG.

Fixes issue #625

Test that we get the expected error when configuring a client-cert
without client-key, or the inverse.

Fix artifact config crash

Closes #625

See merge request !804

richardmaw-codethink authored Sep 04, 2018 and Tristan Van Berkom committed Sep 18, 2018

This fixes all devices being mapped to the non-existant device 0,
which prevents being able to use even safe devices like /dev/null
through the hardlinks FUSE layer.

richardmaw-codethink authored Sep 04, 2018 and Tristan Van Berkom committed Sep 18, 2018

This is needed to permit access to the device nodes added to /dev
on Linux when FUSE is used as root.

The chroot sandbox only works with all privileges,
so there's no explicit check for being root
or having the appropriate capabilities.

A check for whether it's running as root isn't needed on Linux with bubblewrap
because /dev or its devices are mounted on top of the FUSE layer,
so device nodes are accessed directly rather than through the FUSE layer.

fix chroot sandbox devices

See merge request !781

bst build returns "missing project.conf" when a project.conf is invalid
This results in an existing project with malformed yaml being dismissed
and attempts to make a new project

Added new exception

Incorrect error when malformed project.conf

Closes #642

See merge request !792

The SKIPPED message type is now used to indicate the end of a task which
was successful without having to perform the given task.

This overhauls the use of `Queue.done()` and therefore queues do not
need to provide a processed/skipped return value from `done()`. Instead
this is replaced with the action of raising a `SkipJob` exception from
within `Queue.process()`.

Adds a test to ensure that BuildStream alerts the user of a skipped push
when the remote already has the artifact cached.

This removes the timed_activity for an element _push action. This is
unnecessary as the job is already being timed elsewhere.

Pulled/Pushed messages will no longer be produced from within
element.py, instead they will be produced during CasCache push() and
pull() appropriately.

Message consistency has also been improved.

Both pulling and pushing INFO messages are now status messages.

Calls to the messaging API through `self.context.message()` have now
been switched to `element.info`.

Add SkipJob for indicating a skipped activity

See merge request !765

Tristan Van Berkom authored Sep 19, 2018 and Tristan Van Berkom committed Sep 19, 2018

This actually improves reliability of the status bar because we
now disable it completely in the case that not all of the terminal
escape sequences are supported on the given terminal.

This replaces the few functions we were using, to move the cursor
up one line, move it to the beginning of the line, and to clear a
line, with low level functions provided by the curses module in
the standard library.

This change makes it easier for downstream distro package maintainers
to package BuildStream, particularly on Fedora.

Asides from changing _frontend/status.py, this commit includes the
following changes:

  * _frontend/app.py: Use python isatty() function to determine if
                      we are connected to a tty, instead of relying
		      on blessings.

  * setup.py: Remove the dependency on blessings.

_frontend/status.py: Completely remove the blessings dependency from BuildStream

See merge request !808

Fix rst link formatting for guideline links

See merge request !811

This fixes an issue where the re-instantiated Source used with
Source mirroring enabled is not completely initialized.

Failing to load the ref from the project.refs file for instance,
will result in a crash at `fetch` time.

This fixes issue #666

source.py: Fix re-instantiation

Closes #666

See merge request !817

The check for directory should have always been here. The early warning
test doesn't work for `bst checkout` because caches haven't been configured
at the point of the test, so I've had to remove it.

The return value is always upload_vdir.ref.

Also add docstring to method and remove return value since it was useless

Remote exec: Remove early warning and check directory is not None

See merge request !800

Chandan Singh authored Sep 18, 2018 and Chandan Singh committed Sep 21, 2018

Currently, `source-bundle` command is entirely broken as it tries to stage the
sources in a directory that doesn't exist. Fix it by ensuring that we create
the necessary directories before calling any methods that try to use those
directories.

This fix comes with a regression test to ensure that the basic use-case
of `source-bundle` continues to work in future.

Fixes #651.