Make it easy to test BuildStream against external plugins

Background

We intend to greatly reduce the number of plugins in BuildStream core. However, we rely on these plugins in our test suite, using them to exercise many parts of our public API. In order to retain the benefit to test coverage of having a wide selection of plugins in BuildStream core, we need an easay way of leveraging the test suites of plugins from external repositories

Obtaining the external plugins

We maintain a list of plugin repositories that we wish to test against. We keep track of a known good version of each plugin which we use.

My suggestion would be that plugin dependencies be installed at the beginning of the CI job. While this will slow down the plugin testing job compared to pre installing the dependencies in a docker image, installing them as part of the pipeline will mean that we only need to maintain one set of information about the plugins we wish to test against.

Running the tests

As a start, I think we can mandate that plugin repos which are to be included in BuildStream's CI allow their tests to be exercised via 'setup.py test'

In the future we could perhaps use a BuildStream project to keep track of these external plugins and time allow them to make use of custom test commands. However, the former solution has the advantage of making it trivial to view nicely formatted pytest outputs in the GitLab.

The tests should be runnable as part of our main test suite.

The original proposal can be seen here

---