Add support for running tests in BuildStream

Background

From the discussion on the mailing list:

As software integrators, we would like to run tests as part of BuildStream builds and be able to validate (at a later stage in build) that the tests for a given element have passed. Ideally, such tests should have the following properties:

1.) Tests should not block reverse dependency builds.

2.) Strong association between an element and its tests.

3.) Additional dependencies for testing an element should not be required while building it.

4.) Elements should be able to assert that the tests for their dependencies have passed.

5.) Test elements are not necessarily pass-through elements - they can have meaningful output such as coverage reports.

Task description

The proposed solution is to introduce the concepts of 'conditions' and 'abstract dependencies'. For more detail, see the full proposal from the mailing list

  • Add a new conditions field to the bst element file with subfields provides and requires, e.g.
conditions:
  provides:
    tested:
    - app-test.bst
    packaged:
    - app-pkg.bst
  requires:
    - tested
  • Add abstract dependencies (#777)

  • Add functionality to bst show to check the conditions fields of an element (for possible sample outputs, see the examples under part 3 of the mailing list proposal)

  • Expose a Python API for staging an element's build tree (see part 4 of the mailing list proposal for the justification for this)

  • Provide a method to guarantee that the build tree we want to stage (for the purposes of running tests) exists in the dependency artifact

Acceptance Criteria

Users can run tests as part of the build of an element such that the test properties outlined above are satisfied.

Edited by Chiara Tolentino