Skip to content

Make it easy to test BuildStream against external plugins

Phil Dawsonrequested to merge
phil/external-plugin-testing into master

Description

This merge request will allow us to easily incorporate the test suites of external plugins into our own test suite.

Registering a plugin to test against

To add a plugin repository's tests into the BuildStream test suite add a new ExternalPluginRepo object to the EXTERNAL_PLUGINS list in external_plugins.py. This should specify a git repository containing the external plugin and a known ref/commit to test against. As an example, to test against the bst-external plugins you would have

 ExternalPluginRepo(
        name='bst-external',
        url='https://gitlab.com/BuildStream/bst-external.git',
        ref='0.6.1'
    )

An additional parameter test_match_patterns may also be specified. This is a list of shell style glob patterns. Only tests from the repo which match these patterns will be run. This allows us to make use of the tests for a subset of plugins from an external repository. By default, all tests from a repositories tests directory will be run

Running the tests

The tests of any registered plugins will be run automagically as part of a tox invocation. Any command line options given to tox to forward (i.e. after --) will be passed to the pytest invocation which runs the test suite of any external plugins. Alternatively, it is possible to manually run the script which invokes external plugins' test suites at tests/external_plugins.py.

The main test environment will run both the core BuildStream test suite and any tests from specified external plugins. For convenience, I've also added the environments external and internal which will only run the external plugin tests or BuildStream's core tests respectively.

Limitations

There are a few limitations to this MR. These are generally annoying from a developer perspective, but I'm not conviced they should block this (others may disagree).

  • We still need to install any none-python dependencies in our testing environment (either locally or in the test suite docker images). This is no different from our current setup, but may mean there are more things to keep track of here as we move plugins away from core.
  • Each time the test suite is run, fresh clones are made of each external plugin under tests. It would be nice to cache the repos between tests runs. Mainly so that these tests may be run whilst offline, though there is also the obvious speed benefit.
  • It isn't currently (In this MR) possible Running individual tests from external plugins via our tox setup. This is due to the way in which any additional arguments are passed to all invocations of pytest, of which there is one for each repo. Similarly, pytest options such as --lf won't play nicely here.

To demonstrate that it works, this MR is running the test suite of. bst-plugins-template.

This merge request, when approved, will close: #922 (closed)

Edited by Phil Dawson

Merge request reports