fixup! WIP: contributing: more clarity on testing

and use the integration cli helper.

- End-to-end testing is the most valuable.

- tests which observe how these behaviors impact and define the outward user

  experience.

You must test your changes in an end-to-end fashion. Consider the first end to

be the appropriate user interface, and the other the change you have made. The

aim for testing is to make assertions about how you impact and define the

be the appropriate user interface, and the other end to be the change you have

made.

The aim for our tests is to make assertions about how you impact and define the

outward user experience. You should be able to exercise all code paths via the

user interface, just as one can test the strength of rivets by sailing dozens

of ocean liners.

of ocean liners. Keep in mind that your ocean liners could be sailing properly

*because* of a malfunctioning rivet. End-to-end testing will warn you that

fixing the rivet will sink the ships.

The primary user interface is the cli, so that should be the first candidate

'end' for testing. Most of the value of BuildStream comes from what you can

achieve with the cli.

We also have a *"Public API Surface"*, which you should consider the next

target. This is mainly for advanced users to implement their plugins against.

- Useful for Internal API surfaces, e.g. YAML and CasCache.

- Test what the function does, rather than what it should do, need to think in thelarger context.

- Don't fool yourself that because a function has a test that passes, that it

  is doing the right thing.

- If something is not reachable in end-to-end testing, perhaps it is dead

  code and if so should be removed, instead of tested.

- Unit tests can provide direct indication of what is broken.

- You should be able to exercise all code paths via the user interface, just

  as one can test the strength of rivets by sailing dozens of ocean liners,

- providing coverage isn't all of testing.

- it could turn out that your ocean liners are sailing properly *because* of

  a malfunctioning rivet, this is also a very undesirable situation.

- End-to-end testing and unit-testing are not mutually exclusive, you must

  implement end-to-end tests.

You should first aim to write tests that exercise your changes from the cli.

This is so that the testing is end-to-end, and the changes are guaranteed to

work for the end-user. The cli is considered stable, and so tests written in

terms of it are unlikely to require updating as the internals of the software

change over time.

It may be impractical to sufficiently examine some changes this way. For

example, the number of cases to test and the running time of each test may be

too high. It may also be difficult to contrive circumstances to cover every

line of the change. If this is the case, next you can consider also writing

unit tests that work more directly on the changes.

We also have a :ref:`*"Public API Surface"*<contributing_public_api_surface>`,

which you should consider a secondary target. This is mainly for advanced users

to implement their plugins against.

Note that both of these targets for testing are guaranteed to continue working

in the same way across versions. This means that tests written in terms of them

will be robust to large changes to the code. This important property means that

BuildStream developers can make large refactorings without needing to rewrite

fragile tests.

Another user to consider is the BuildStream developer, therefore internal API

surfaces are also targets for testing. For example, the YAML loading code and

the CasCache. Remember that these surfaces are still just a means to the end of

providing value through the cli and the Public API Surface.

It may be impractical to sufficiently examine some changes in an end-to-end

fashion. For example, the number of cases to test and the running time of each

test may be too high. Such typically low-level things, e.g. parsers, may also

be tested with unit tests - alongside the mandatory end-to-end tests.

It is important to write unit tests in such a way that they do not break due to

changes unrelated to what they are meant to test. For example, if the test

require the test to be rewritten. Pure functions that only rely on the Python

Standard Library are excellent candidates for unit testing.

Unit tests only make it easier to implement things correctly, end-to-end tests

make it easier to implement the right thing.

Measuring performance

---------------------