Concurent instances of bst should share cache without issues

Background

Currently we do not properly deal with concurrency in artifact cache.

For instance we could get the same issue #520 (closed), but while scanning removed artifacts.

Artifacts in cache should not be removed if still planned to be used by a task in a pipeline of any instance.

Also we should be able to add/remove artifacts/extracts concurrently without failure.

Acceptance Criteria

  • Scanning artifact/extract directories can handle deletion of artifacts/extracts.
  • Artifacts and extracts can be locked.
  • Adding and removing of artifacts is safe to do concurrently (I believe it is already)
  • Adding and removing of extracts is safe to do concurrently (I think it is not safe, we raise errors when we cannot commit extract directory).
  • Calculating/caching the size of artifact cache should work across instances

Edited by Valentin David