Expose templated source tests

Description

Make it possible for plugin authors to leverage buildstream's suite of generic source tests.

1. In order to run a custom source type through this test suite a plugin author must: Adding the following code into a conftest.py in a project's test suite will allow pytest to discover the suite of source tests.

   import pytest
   from buildstream.plugintestutils import sourcetests_collection_hook
   
   
   # This hook enables pytest to collect the templated source tests from
   # buildstream.plugintestutils
   def pytest_sessionstart(session):
       sourcetests_collection_hook(session)

   The tests will then be discovered as part of whichever test package contains this code in a conftest.py.

2. Subclass buildstream.plugintestutils.Repo, implementing the functionality appropriate to their source plugin

3. Use the register_repo_kind method provided by buildstream.plugintestutils to register the new Repo type with the test suite. This must be done somewhere which willl be executed prior to pytest performing test collection, probably in a conftest.py or __init__.py file at an appropriate location(s) in the test suite.

Changes proposed in this merge request:

• Expose Repo class as part of the buildstream.plugintestutils API
• Move templated source tests from tests.sources.generic into buildstream.plugintestutil._sourcetests.
• Provide an easy way to register source types to run though the templated test suites

This merge request, when approved, will close issue/bug: #944 (closed)

Note:

This MR is most easily reviewed commit by commit. The majority of the files changed are ammending imports due to the movement of repo.py from tests.testutils to buildstream.plugintestutils

added 2 commits

• b8ac354c - plugintestutils._utils: Copy subset of tests.testutils.site
• 297cab48 - Make templated source tests available in buildstream.plugintestutils

Compare with previous version

added 4 commits

• 30f69d48 - testutils: move repo.py into buildstream.plugintestutils
• e3520ce8 - plugintestutils: Add copy of testutils.junction.py
• 3a59d299 - plugintestutils._utils: Copy subset of tests.testutils.site
• f1da70b2 - Make templated source tests available in buildstream.plugintestutils

Compare with previous version

added 4 commits

• 662d7113 - testutils: move repo.py into buildstream.plugintestutils
• 01caf6ad - plugintestutils: Add copy of testutils.junction.py
• ac005c1b - plugintestutils._utils: Copy subset of tests.testutils.site
• 9c668df6 - Make templated source tests available in buildstream.plugintestutils

Compare with previous version

unmarked as a Work In Progress

changed the description

added 12 commits

• 9c668df6...bcf02294 - 8 commits from branch master
• da5a629d - testutils: move repo.py into buildstream.plugintestutils
• 94d31066 - plugintestutils: Add copy of testutils.junction.py
• 79b59c97 - plugintestutils._utils: Copy subset of tests.testutils.site
• ae80b19b - Make templated source tests available in buildstream.plugintestutils

Compare with previous version

added 1 commit

• 6ec72e3b - Make templated source tests available in buildstream.plugintestutils

Compare with previous version

added 25 commits

• 6ec72e3b...f6de43de - 21 commits from branch master
• 11097fe1 - testutils: move repo.py into buildstream.plugintestutils
• 2d9efd62 - plugintestutils: Add copy of testutils.junction.py
• 2ad0adda - plugintestutils._utils: Copy subset of tests.testutils.site
• d8873c36 - Make templated source tests available in buildstream.plugintestutils

Compare with previous version

added 8 commits

• d8873c36...d20165a6 - 4 commits from branch master
• e39174de - testutils: move repo.py into buildstream.plugintestutils
• 47d606b0 - plugintestutils: Add copy of testutils.junction.py
• 76218b1c - plugintestutils._utils: Copy subset of tests.testutils.site
• fe6d397d - Make templated source tests available in buildstream.plugintestutils

Compare with previous version

Two things I expected to work in the local dev case.

1. I can run all of the sources tests like this:

 $ tox -e py36 -- buildstream/plugintestutils/_sourcetests/
======================================= no tests ran in 1.44 seconds =======================================
ERROR: InvocationError for command '/home/jennis/Software/buildstream/.tox/py36/bin/pytest --basetemp /home/jennis/Software/buildstream/.tox/py36/tmp --cov=buildstream --cov-config .coveragerc buildstream/plugintestutils/_sourcetests' (exited with code 5)
_________________________________________________ summary __________________________________________________
ERROR:   py36: commands failed

This doesn't run the tests, whereas, tox -e py36 -- tests/internals/, for example, would run all the internals tests

2. When I specify an individual test file, most of the tests are now skipped (they were not previously):

 $ tox -e py36 -- buildstream/plugintestutils/_sourcetests/track.py

[..snip...]

py36 runtests: commands[0] | pytest --basetemp /home/jennis/Software/buildstream/.tox/py36/tmp --cov=buildstream --cov-config .coveragerc buildstream/plugintestutils/_sourcetests/track.py
=========================================== test session starts ============================================
platform linux -- Python 3.6.7, pytest-4.3.0, py-1.8.0, pluggy-0.9.0 -- /home/jennis/Software/buildstream/.tox/py36/bin/python
cachedir: .tox/py36/.pytest_cache
rootdir: /home/jennis/Software/buildstream, inifile: setup.cfg
plugins: xdist-1.26.1, timeout-1.3.3, forked-1.0.2, env-0.6.2, datafiles-2.0, cov-2.6.1
collected 13 items                                                                                         

buildstream/plugintestutils/_sourcetests/track.py::test_track[kind0-inline] SKIPPED                  [  7%]
buildstream/plugintestutils/_sourcetests/track.py::test_track[kind0-project.refs] SKIPPED            [ 15%]
buildstream/plugintestutils/_sourcetests/track.py::test_track_recurse[kind0-1] SKIPPED               [ 23%]
buildstream/plugintestutils/_sourcetests/track.py::test_track_recurse[kind0-10] SKIPPED              [ 30%]
buildstream/plugintestutils/_sourcetests/track.py::test_track_recurse_except[kind0] SKIPPED          [ 38%]
buildstream/plugintestutils/_sourcetests/track.py::test_cross_junction[kind0-inline] SKIPPED         [ 46%]
buildstream/plugintestutils/_sourcetests/track.py::test_cross_junction[kind0-project.refs] SKIPPED   [ 53%]
buildstream/plugintestutils/_sourcetests/track.py::test_track_include[kind0-inline] SKIPPED          [ 61%]
buildstream/plugintestutils/_sourcetests/track.py::test_track_include[kind0-project.refs] SKIPPED    [ 69%]
buildstream/plugintestutils/_sourcetests/track.py::test_track_include_junction[kind0-inline] SKIPPED [ 76%]
buildstream/plugintestutils/_sourcetests/track.py::test_track_include_junction[kind0-project.refs] SKIPPED [ 84%]
buildstream/plugintestutils/_sourcetests/track.py::test_track_junction_included[kind0-inline] SKIPPED [ 92%]
buildstream/plugintestutils/_sourcetests/track.py::test_track_junction_included[kind0-project.refs] SKIPPED [100%]
 
[...snip...]
---------------------------------------------------------------------------------------------------
TOTAL                                                              13906  12026    13.52%


======================================== slowest 20 test durations =========================================

(0.00 durations hidden.  Use -vv to show these durations.)
======================================== 13 skipped in 1.67 seconds ========================================

Now perhaps this just requires another pytest argument...? If so, this should probably be invoked by default.

Regarding: !1261 (e39174de, comment 155589243) I just did a whole run of the test suite and none of the tests in plugins/_sourcetests/* were tested. Is this expected?

@jennis, !1261 (comment 155589243)

This is a result of how I've made use of the templated tests in BuildStream's test suite. Given what you did is the obvious thing to try if you just want to run the templated tests, I'll first explain what you've observed, then ask what would make you happier.

The explanation

I'll explain for 1) , but the reason is the same for 2). The reason that doesn't do what you expect is because of the where register_repo_kind is called in the test suit. As it stands these are in the init.py of tests/sources, which is not executed with tox -e py36 -- buildstream/plugintestutils/_sourcetests/ This means that no repo types are registered when the tests execute, giving you the output you observed.

As it stands, what you attempted can be achieved by running for example

tox -- tests/sources -k _sourcetests

the _sourcetests tests will be collected here because the sourcetests_collection_hook is called in the conftest.py in tests/sources/, the -k argument is then matching tests from the _sourcetests module.

Possible changes

Moving where in BuildStream's test suite where register_repo_kind and sourcetests_collection_hook will change when the templated tests are collected.

Moving the registering of plugin kinds into a root (as in the project root) conftest.py would make your tox command work as you expect but I really don't like this idea as it means having tests stuff outside of our /tests directory.

Perhaps adding a custom pytest mark such that it would be possible to just do tox -- --sourcetests to run those tests only (I'm not sure what the best name for the mark is, but you get the idea) would make life easier/more intuative?

added 25 commits

• fe6d397d...9605f798 - 21 commits from branch master
• de42f02f - testutils: move repo.py into buildstream.plugintestutils
• fdd2aec3 - plugintestutils: Add copy of testutils.junction.py
• 733ca94f - plugintestutils._utils: Copy subset of tests.testutils.site
• 557965ce - Make templated source tests available in buildstream.plugintestutils

Compare with previous version

added 1 commit

• d521931e - Make templated source tests available in buildstream.plugintestutils

Compare with previous version