WIP: Source transform plugin

Description

In april, a discussion about BuildStream's integration with other package-managers (pip,dep,npm,cargo etc.) started.

It's been discussed that a new base plugin, similar to Source but tracks/fetches things based on previous sources, could pave the way for possible package-manager integrations and using those, users could push pip install, yarn, dep ensure steps in their build-commands to fetch stage.

This and this discusses the behaviour of the new plugin.

Changes proposed in this merge request:

• New base plugin introduced, alternative to Source, called SourceTransform
• SourceTransform is extended from Source for DRY purposes.
• Source._fetch & Source._track (and their overridden versions in SourceTransform) are called with kwarg previous_sources.
  • Source ignores it
  • SourceTransform uses it to implement the behaviour below:

# SourceTransform.track()
This requires that previous Sources are not only tracked, but that
they are also *fetched* for the tracked version, so we know that
all previous sources are available to stage.

Running SourceTransform.track() involves first staging all the
previous sources to a temporary directory, and then running
SourceTransform.track()

The result of SourceTransform.track() is an updated ref, like any
other Source.

Taking rust as the example of choice, the result of it's track()
implementation is a simple python dictionary representation of
a Cargo.lock file.

# SourceTransform.fetch()

The result of the transform's fetch() implementation is that the
transform will download the precisely required versions of all
dependencies according to it's own ref, and cache them as normal
in the source cache.

Unlike SourceTransform.track(), SourceTransform.fetch() does not
require the context of the previous sources.

How to use it?

Final commit also introduces a concrete usage example or SourceTransform, called dep_transform for now, that fetches go dependencies using the tool dep. It will be removed from the MR when un-WIPped. An example bst file:

kind: manual

depends:
- filename: platform.bst

- filename: go.bst
  type: build

sources:
- kind: git
  url: REPO_URL
  track: master

- kind: dep_transform

# .. build and install commands, go build etc.

During track: This example requires a Gopkg.toml file and tries to generate Gopkg.lock file out of that. If the repo already has a lockfile, it uses that.

During fetch: Using the previously acquired Gopkg.lock file, it downloads dependencies and populates a vendor directory.

Note: this plugin is an opinionated prototype at the moment. It might fit for your use case yet.

This merge request, when approved, will close issue/bug: #381 (closed)

---