Commit 3fa79d8d, part of an initiative for caching of the failed builds,
introduced a call to Element._update_state() after a job completes and before
entering the error handling harness intended for handling plugin raised errors.

Element._update_state() can result in triggering plugin code to
run, so this is incorrect, and causes raised errors to crash BuildStream
if they happen here.

After analyzing the code, it appears that this additional call to
Element._update_state() is unneeded, and was only added because the
state needs to be updated for a failure as well as a success.

Instead, we now have the BuildQueue call Element._assemble_done()
unconditionally, regardless of whether the build was successful
or not, which has the same effect and also reads better.

In addition, added a FIXME comment that we are still conditionally
updating the artifact cache size from BuildQueue.done() only
if the build is successful, which is incorrect because failed builds
also increase the local artifact cache size - to fix this we need
to communicate the added artifact size through Element._assemble()
regardless of whether the build succeeded or failed.

_scheduler/queues/queue.py: Don't call update state outside of error handling harness

See merge request !988

Instead of tag information being fetched which can change with time,
they are tracked and saved in the projects.refs/.bst. Then we re-tag
automatically the closest tag so that `git describe` works and is
reproducible.

This new feature is opt-in with the new `track-tags` configuration,
and must be used to fix modules which are broken by our new policy
of omitting the `.git/` repository when staging git sources.

This fixes issue #487

Track of git tags and save them to reproduce minimum shallow repository

Closes #487

See merge request !906

In element, platform asks for host os and architecture to default to when
SandboxConfig is initialised.
This changes element cache keys so those have been updated in the tests.

Also update tests to be consistent with this

These are contained within the platform field.

Same format as OptionArch tests, but with OS's.

  o _platform/linux.py: Add linux32 flag to send to sandbox bwrap when the
    build arch is x86-32 and the machines arch is x86-64 or similarly with
    aarch32 and aarch64.
  o sandbox/_sandboxbwrap.py: Use flag passed to start bwrap command with
    linux32 if set.

Execution environment reqs

Closes #775

See merge request !969

Get cascache.py to use this

Stops unneccesary pushing of builds that have already been built, just checks
the action cache to begin with.

Fixes #628

The endpoint may not necessarily be the same as execution-service endpoint.

Part of #628

Part of #628

Remote-execution client flow optimisation

Closes #628

See merge request !982

Instead of duplicating the whole test body, run it twice while
checking for a warning or an error depending on the parameter.

This also ensures it is exposed via the main buildstream __init__.py
file, fixes the imports of CoreWarnings, and adjusts the documentation
links to point to the right place.

Now the commands have documentation linkability so we can link to
them when referring to them anywhere in the docs. This reference
was previously referring to a bogus non-existent `build-stream track`,
maybe this dates way back to the days before we named the frontend `bst`.

Fixes the configurable warning part of this to properly link
to the project.conf documentation describing what configurable
warnings actually are.

Also fix the plugin raise the actual configurable warning which
it advertizes that it raises; which is 'inconsistent-submodule',
not 'inconsistent-submodules'.

Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
make things a bit more readable and comprehensive.

Misc cleanups

See merge request !993

This is guaranteed to be called only once for a given session once the
sources are known to be Consistency.CACHED, if source tracking is enabled
in the session for this source, then this will only be called if the
sources become cached after tracking completes.

 o Unlisted submodule warning

   Now the git plugin will issue a configurable warning if a submodule
   exists and is used (checking out the submodule is not disabled),
   but is not specified in the source configuration.

 o Invalid submodule warning

   Now the git source plugin will issue a warning if the configuration
   specified a submodule which does not exist in the underlying git repository.

As a side effect, this patch also changes the flow control of the git
plugin such that submodules which are explicitly set to not be
checked out, are also not fetched but instead ignored completely.

Now that we have Source.validate_cache(), this is a better place to emmit
the ref-not-in-track warning, since it will be emmitted at the earliest
opportunity and not only at Source.stage() or Source.init_workspace().

This also allows us to remove the `self.tracked` local state, and cleanup
some convoluted calling paths, removing some unnecessary parameters from
the usual codepaths and making the plugin overall more readable.

A function for removing paths from a git repo, can also
be used to remove submodules.

 o Test that it is not triggered in show before fetch, because we
   don't know about the unlisted submodules yet

 o Test that it is triggered by a fetch command

 o Test that it is triggered by `show` after having completed a
   fetch command, since now we have the repository and know about
   the unlisted submodule

 o Test all of this under warning or error conditions (parameterized
   for fatal-warnings)

   o Test that it is not triggered in show before fetch, because we
     don't know the submodules yet so we cannot know if they are
     valid or not.

   o Test that it is triggered by a fetch command

   o Test that it is triggered by `show` after having completed a
     fetch command, since now we have the repository and know which
     specified submodules are invalid

   o Test all of this under warning or error conditions (parameterized
     for fatal-warnings)

Implement submodule warnings

See merge request !996

Fix bst source-checkout not working with open workspaces

Closes #807

See merge request !997

This allows specifying whether or not the code believes a None
value is acceptable if explicitly expressed to be None in the YAML.

A previous patch 3ba544b8 changed
the behavior to accept None for all values across the board, along
with allowing explicitly setting the `default_value` to `None` which
caused the code to be much more readable (this was the main motivation
of the patch, but it had the side effect of allowing None for everything).

In the majority of cases we load YAML however, it either has a value or
it is not specified in the YAML, and None is hardly ever acceptable
to be explicitly specified (it may be in the case you want to override
something with nothing in the YAML, but it is the edge case).

This had the side effect of causing unexpected crashes where the
YAML specifies an empty dictionary for instance.

Instead of forcing the caller to handle a possible None value, give
the choice to the caller if they really want to accept a None value,
and raise the LoadError otherwise.

This fixes issue #803, which is a crash due to receival of an
unexpected None value through _yaml.node_get()

Allow plugins to conveniently leverage the new `allow_none` parameter,
and have Plugin.node_get_member() raise a LoadError in the usual
case where None is not an acceptable value to be explicitly set in
the user provided YAML.

This adds a regression test for issue #803

Only optionally allow None values in user provided YAML

Closes #803

See merge request !999

They are too specific to be included by default

Recommendation is if you are building in Linux is to use the
ones begin used in the freedesktop-sdk project, for example

See #645

Remove default strip-commands

Closes #645

See merge request !987

When using aliases there are multiple remotes used in the cache
repository. When fetching, tags are not updated if the were previously
fetched from a different remote. Commits that not in a branch and only
tagged do not get fetched if the tag is not fetched.

Fixes #812

Force updating tags when fetching git repository

Closes #812

See merge request !1000

i.e. with a given directory and filename, check parent directories until
either a directory with the filename is found, or you reach the root of
the filesystem.

This is a part of #222

Changes to _context.py:
* Context has been extended to contain a WorkspaceProjectCache, as there
  are times when we want to use it before a Workspaces can be initialised
  (looking up a WorkspaceProject to find the directory that the project is
  in)

Changes to _stream.py:
* Removed staging the elements from workspace_open() and workspace_reset()

Changes in _workspaces.py:
* A new WorkspaceProject contains all the information needed to refer back
  to a project from its workspace (currently this is the project path and
  the element used to create this workspace)
* This is stored within a new WorkspaceProjectCache object, which keeps
  WorkspaceProjects around so they don't need to be loaded from disk
  repeatedly.
* Workspaces has been extended to contain the WorkspaceProjectCache, and
  will use it when opening and closing workspaces.
* Workspaces.create_workspace has been extended to handle the staging of
  the element into the workspace, in addition to creating the equivalent
  WorkspaceProject file.

This is a part of #222

This is a part of #222

This is a part of #222

This is a part of #222

This is a part of #222

cli: Interactively warn if the user is trying to close the workspace they're using to load the project

This involves changes in:
* _stream.py:
  * Add the helper Stream.workspace_is_required()
* userconfig.yaml:
  * Add a default value for prompt.really-workspace-close-project-inaccessible
* _context.py:
  * Load the prompt 'really-workspace-close-project-inaccessible' from
    user config.
* cli.py:
  * If buildstream is invoked interactively, prompt the user to confirm
    that they want to close the workspace they're using to load this
    project.

This is a part of #222

This is a part of #222

This is a part of #222

Support invoking buildstream from a workspace outside a project

See merge request !924

Known issues:
* `bst shell` works, but `bst shell COMMANDS...` doesn't, because click
  has no way of separating optional args from variable-length args.
* `bst checkout` and `bst source-checkout`'s usage strings mark LOCATION
  as an optional argument. Because click gets confused if there's an
  optional argument before a mandatory argument, I had to mark LOCATION
  as optional internally.
* `bst workspace open` makes no sense with element being optional, so
  I skipped it.
* `bst workspace close` will probably need to be revisited when multiple
  projects can own one workspace.
* `bst workspace reset` will happily delete the directory you're
  currently in, requiring you to `cd $PWD` to see the contents of your
  directory.
  I could exclude the top-level directory of the workspace being
  deleted, but it is entirely valid to run workspace commands from deeper
  in the workspace.

This is a part of #222

This is a part of #222

This is a part of #222

This is a part of #222

Guess the element when running commands in a workspace

See merge request !954