richardmaw-codethink authored Sep 05, 2018 and richardmaw-codethink committed Sep 13, 2018

This can get out of sync with other two cache states,
and we can do without it.

Simplify element state by removing `__cached`

See merge request !784

We can't include a socket in a CAS tree, but it's mostly meaningless to do so
since there can't possibly be a process serving it.

Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type."

See merge request !783

richardmaw-codethink authored Sep 04, 2018 and richardmaw-codethink committed Sep 14, 2018

Overriding the config with a custom config file on the command-line
prevents it merging in the test-specific configuration
and can permit it to attempt to initialise the user's cache.

richardmaw-codethink authored Sep 05, 2018 and richardmaw-codethink committed Sep 14, 2018

They were moving the whole tmpdir to move the project repository.
This moves the cache directories etc. as well, meaning cli.run can't find them.
This was worked around by setting configure=False,
but this has the side-effect of making use of the user's caches,
which it should not be doing for reproducibility reasons.

By changing the tempdir layout to have the project in a subdirectory
we can move the project around for the relative workspace tests
without losing track of the configured state directories,
so we can leave configure=True and avoid touching the user's caches.

Fix tests that attempt to access the home directory

See merge request !780

This was previously append_required_artifacts(), which presumed that
we knew at startup time what the cache keys of all elements in the
loaded pipeline would be.

This fixes unexpected deletions of required artifacts when
dynamic tracking is enabled with `bst build --track-all target.bst`

This commit renames test_never_delete_dependencies() to
test_never_delete_required(), renders the test more readable by renaming
some elements and reordering some statements and makes the comments more
straight forward and accurate.

These tests were not checking that we fail for the expected reasons.

Added `res.assert_task_error(ErrorDomain.ARTIFACT, 'cache-too-full')`
where we expect to fail because the cache is too full.

This allows one to modify a file in an existing git repo,
as opposed to adding a new one.

 * create_element_size()

   Now uses a git Repo object instead of a local source, and
   returns the repo.

 * update_element_size()

   Added this function which can now resize the expected output
   of an element generated with create_element_size(), useful
   to allow testing sized elements with the tracking feature.

Same test as test_never_delete_required(), except that this test ensures
that we never delete required artifacts when their cache keys are
discovered dynamically during the build.

Don't delete required artifacts when tracking is enabled

See merge request !793

We want to check if some file is already cached here, not the parent
directory.

Tristan Maat created the original file, so he is added as the author.

* Rename it to _commit_directory() because… it is what it does; and
  also for symmetry with _fetch_directory().
* Rename digest to dir_digest to make it clear this is a digest for a
  directory. A following commit will also reuse the same variable name
* Document method.

* Rename tree to dir_digest to make it clear this is a Digest object,
  and not a Tree object.
* Add documentation

Remove unneeded cruft.

Bunch of cleanups

See merge request !798

Track amount of bytes required to commit or pull an artifact and return
this data on pull() and commit().

The commit() return value is more accurate than computing the size of
the artifact staging directory, as it takes deduplication into account.

This will be used in a following commit to dynamically update the cache
size.

Change pull jobs to dynamically update the cache size, as build jobs
were already doing it.

The pathways between pull and build jobs are similiar and the code can
be updated appropriately.

As the cache size is always dynamically updated, we no longer need to
run the CacheSize job.

Fixes #573.

Replace set_cache_size() with subtract_cache_size(), for symmetry with
add_artifact_size().

Is no longer used now that we update the cache size dynamically all the
time.

Also the ArtifactCache.compute_cache_size() is no longer used externally
and can be removed.

Ensure that add_artifact_size() is called from the main process.