Issue #446

  - Slight modification to the wording of existing error message.
  - Closes #446

This is to avoid inconsistencies when dealing with paths inside an
elements directory

Tiago Gomes authored Jun 28, 2018 and Tristan Van Berkom committed Jul 02, 2018

Before running a command in the sandbox, check its existence and fail
early if it does not.

This fixes issue #289.

The OSTree summary file is no longer used.

initialize_remotes() no longer fetches remote refs.

They weren't actually catching it if you checked-out the entire depended
element, instead of just the specified split domains

It was redirecting for checkout operations

This makes it unnecessary to update the free space in the mock object in
the middle of tests.

This allows code generation with ./setup.py build_grpc

This imports protos from https://github.com/googleapis/googleapis/ and
the modules generated with ./setup.py build_grpc.

This imports protos from https://github.com/bazelbuild/remote-apis/ and
the modules generated with ./setup.py build_grpc.

https://github.com/bazelbuild/remote-apis/pull/18/

Content Addressable Storage specified in the Remote Execution API.

Fixes #387.

This makes cache cleanup LRU.

The CAS server now supports LRU.

No longer used.

No longer used.

Fixes #134, #138, #148, #217, #268, #276, #443, #460.

The only existing artifact cache implementation always returns True.

Fixes #473.

buildstream/_stream.py: Added deps param to _prepare_sandbox function call
buildstream/element.py: Added deps param and logic to _prepare_sandbox function

_reachable_refs_dir() failed to add directory digests to the set of
reachable objects.

When the cache quota is hit, we will remove any elements not required
for the current build, until our cache is only filled halfway.

Added the new Context.recorded_messages() context manager, this
causes messages to be logged in a dedicated log file before being
propagated to the log handler, and also provides an open log handle
for the capturing and logging of stdout/stderr from child processes.

This comes with the additional accessors:

   Context.get_log_handle()
   Context.get_log_filename()

  o element.py / plugin.py: Removed supporting logging code, and derive
                            the log handle from Context.

  o _scheduler/scheduler.py, _scheduler/queues/queue.py: Adapt to new Job initializer API for the logfile

  o _scheduler/jobs/job.py: Run job activities within the new context manager
                            which turns on logging for a job. Also removed a lot
			    of custom logging abstract methods which are unneeded.

  o _scheduler/jobs: Job implementations need not implement so much custom logging.

Daniel Playle authored Jul 02, 2018 and Jürg Billeter committed Jul 18, 2018

When testing locally on some systems, it appears that pylint fails on
the order of imports for `buildstream/_frontend/app.py`. This commit
changes the order of these imports.

This commit addresses issue !453.

Instead of re-using the user-facing `buildstream/buildstream-fedora`
image for tests (that comes with BuildStream pre-installed), use
`buildstream/testsuite-fedora` image that is supposed to be used
exclusively for these tests.

Similarly, use the dedicated `buildstream/testsuite-debian` image
instead of `buildstream/buildstream-debian`.

Fixes #458.

For related discussion, see
BuildStream/buildstream-docker-images#8.

Before this option was listed in help but was ignored when
creating the source-bundle.

Issue #468

Jordan Petridis authored Jul 14, 2018 and Javier Jardón committed Jul 20, 2018

`ruamel.yaml` seems to require `Python.h` header file to build.

`python3-devel` is what provides it for Fedora.

Also remove the recomendation to install psutil as we need to build
other python modules anyway (like ruamel)

Completes 96d07153

William Salmon authored Jul 05, 2018 and Javier Jardón committed Jul 20, 2018

These dependencies are needed to use the plugins that are installed
as part of this guild.

As the dependencies added are not dependencies of the core package
they have been added to a separate section.

The lzip package is needed to build gnome.
https://wiki.gnome.org/Newcomers/BuildSystemComponent

One of the tests added is configured to be skipped for now, as dumping
binary data is causing a bad descriptor exception when using the pytest
capture module.

Closes #263.

This causes the new artifact tests to pass (unless you happened
to already have a recent enough version of pytest-cov, in which case
you didn't notice the breakage).

I hit the following backtrace running `bst --debug push` and `bst
--debug build`:

    pid:16736 id:000[--:--:--][][] START   Push
    pid:16736 id:000[--:--:--][][] START   Loading pipeline
    pid:16736 id:000[00:00:00][][] SUCCESS Loading pipeline
    pid:16736 id:000[--:--:--][][] START   Resolving pipeline
    pid:16736 id:000[--:--:--][][] BUG     'MesonElement' object has no attribute '_Element__cache_key'

        Traceback (most recent call last):
          File "/home/sam/.local/bin/bst", line 8, in <module>
            sys.exit(cli())
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
            return self.main(*args, **kwargs)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 162, in override_main
            standalone_mode=standalone_mode, **extra)
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
            rv = self.invoke(ctx)
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
            return _process_result(sub_ctx.command.invoke(sub_ctx))
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
            return ctx.invoke(self.callback, **ctx.params)
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
            return callback(*args, **kwargs)
          File "/home/sam/.local/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func
            return f(get_current_context().obj, *args, **kwargs)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 471, in push
            app.stream.push(elements, selection=deps, remote=remote)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 341, in push
            fetch_subprojects=True)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 814, in _load
            fetch_subprojects=fetch_subprojects)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in load
            for meta in meta_elements
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in <listcomp>
            for meta in meta_elements
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 894, in _new_from_meta
            element = project.create_element(artifacts, meta)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_project.py", line 184, in create_element
            element = self._element_factory.create(self._context, self, artifacts, meta)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_elementfactory.py", line 57, in create
            return element_type(context, project, artifacts, meta, default_config)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 196, in __init__
            super().__init__(meta.name, context, project, meta.provenance, "element")
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 171, in __init__
            self.debug("Created: {}".format(self))
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 407, in debug
            self.__message(MessageType.DEBUG, brief, detail=detail)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 652, in __message
            self.__context.message(message)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_context.py", line 338, in message
            self._message_handler(message, context=self)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/app.py", line 665, in _message_handler
            text = self.logger.render(message)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 585, in render
            return self._render(message)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 620, in _render
            text += widget.render(message)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 228, in render
            _, key, missing = plugin._get_display_key()
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 1138, in _get_display_key
            cache_key = self._get_cache_key()
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 991, in _get_cache_key
            return self.__cache_key
        AttributeError: 'MesonElement' object has no attribute '_Element__cache_key'

The issue here is that when `--debug` is enabled, Plugin.__init__()
tries to log a message like this:

        pid:16929 id:001[--:--:--][????????][ main:tracker.bst                   ] DEBUG   Created: meson element at tracker.bst [line 1 column 0]

The log formatter is trying to get the cache key of the element in
order to show it in the log, but the attribute hasn't yet been set
because we've not got to the Element() constructor.

Phillip Smyth authored May 29, 2018 and Tristan Van Berkom committed Jul 20, 2018

Fixes issue #455

Tristan Van Berkom authored Jul 20, 2018 and Tristan Van Berkom committed Jul 20, 2018

This will break builds which use `git describe` to determine
their version number until a more elegant solution is implemented,
and will unblock work on caching of build trees in the artifacts.

Fixes issue #455

Instead, we are enabling the merge commit feature in gitlab and
hopefully reducing some of the friction in review this way.

Tristan Van Berkom authored Jul 22, 2018 and Tristan Van Berkom committed Jul 23, 2018

This improves staging performance by avoiding making any
copies of the objects in the cloned repository in the staging
area before removing the .git directory.

Fixes #488

plugins/sources/git.py: Use --shared instead of --hardlinks

Closes #488

See merge request !557

Qinusty authored Jul 19, 2018 and Tristan Van Berkom committed Jul 23, 2018

This change will allow for more detailed info messages throughout
the CASCache.

Qinusty authored Jul 19, 2018 and Tristan Van Berkom committed Jul 23, 2018

Work towards fixing issue #275

Qinusty authored Jul 19, 2018 and Tristan Van Berkom committed Jul 23, 2018

Work towards fixing issue #275

Indicate where artifacts are going to and coming from in the log

Closes #275

See merge request !553

buildstream/element.py: Adding build tree to cache
buildstream/_versions.py: Bumping BST_CORE_ARTIFACT_VERSION

Adding caching build trees (limited)

See merge request !560

doc/source/install_linux_distro.rst: buildstream doesn't hard depend on ostree or pygobject anymore

Closes #492

See merge request !558

This addresses issue #501.

Remove misleading info message

See merge request !567

Add test to ensure gracefull handling of exception thrown while loading
a pipeline due the python's max recursion depth being exceeded.

This is part of the work for issue #203

Because the RecursionError exception was introduced in Python 3.5, until we
drop support for for Python 3.4, we must use RuntimeError.

Phil/203 BuildStream crashes when dependency tree too deep

See merge request !512

tests/frontend/workspace.py: Added tests

_stream.py: Added functionality for workspace open -f

See merge request !549