[#365] Added Tasty integration
Description
Problem: we have no good way of structuring our nettests. As a result,
we have to resort to comment
to indicate which tests succeeded/failed,
and the output is all jumbled together and hard to understand/parse.
Furthermore, when one test fails, the whole suite aborts without
executing the remaining tests.
Solution: integrate Morley.Nettest with Tasty. This way, we get all the benefits that come with using Tasty:
- structured tests and output
- skipping/running only a few tests using
--pattern
- run all tests and report those that failed
- ability to have hunit/hspec/hedgehog + nettests all in the same executable
- ability to pass config options via environment variables as well as CLI options
(Note: by convention, an env variable name is the same as the CLI option name, except it's all caps, underscores instead of hyphens, and prefixed with TASTY_
. For example, --nettest-skip-network
corresponds to TASTY_NETTEST_SKIP_NETWORK
).
Example usage (with caps interface):
test :: TestTree
test = nettestScenarioCaps "storage is 1" $ do
addr <- addressResolved <$> originate OriginateData {..}
checkStorage addr (toVal @Natural 1)
test :: TestTree
test = nettestScenarioCaps "storage is 1" scenario
scenario :: MonadNettest caps base m => m ()
scenario = do
addr <- addressResolved <$> originate OriginateData {..}
checkStorage addr (toVal @Natural 1)
Example usage (without caps interface):
test :: TestTree
test = nettestScenario "storage is 1" $ \impl
addr <- addressResolved <$> niOriginate impl OriginateData {..}
niCheckStorage impl addr (toVal @Natural 1)
test :: TestTree
test = nettestScenario "storage is 1" $ scenario
scenario :: NettestScenario m
scenario impl = do
addr <- addressResolved <$> niOriginate impl OriginateData {..}
niCheckStorage impl addr (toVal @Natural 1)
Tasty's --help
now looks like this:
Mmm... tasty test suite
Usage: cleveland-test [-p|--pattern PATTERN] [-t|--timeout DURATION]
[-l|--list-tests] [-j|--num-threads NUMBER] [--xml ARG]
[-q|--quiet] [--hide-successes]
[--color never|always|auto] [--ansi-tricks ARG]
[--nettest-alias-prefix STRING]
[-A|--nettest-node-url STRING]
[-P|--nettest-node-port NATURAL NUMBER]
[-I|--nettest-client-path STRING]
[-d|--nettest-data-dir STRING] [-S|--nettest-use-https]
[-V|--nettest-verbose]
[--nettest-import-secret-key SECRET_KEY]
[--nettest-skip-network] [--hedgehog-replay ARG]
[--hedgehog-show-replay ARG] [--hedgehog-tests ARG]
[--hedgehog-discards ARG] [--hedgehog-shrinks ARG]
[--hedgehog-retries ARG]
Available options:
-h,--help Show this help text
-p,--pattern PATTERN Select only tests which satisfy a pattern or awk
expression
-t,--timeout DURATION Timeout for individual tests (suffixes: ms,s,m,h;
default: s)
-l,--list-tests Do not run the tests; just print their names
-j,--num-threads NUMBER Number of threads to use for tests execution
--xml ARG A file path to store the test results in
Ant-compatible XML
-q,--quiet Do not produce any output; indicate success only by
the exit code
--hide-successes Do not print tests that passed successfully
--color never|always|auto
When to use colored output (default: 'auto')
--ansi-tricks ARG Enable various ANSI terminal tricks. Can be set to
'true' (default) or 'false'.
--nettest-alias-prefix STRING
[Morley.Nettest] A prefix to prepend to every alias
created with 'newAddress' or 'newFreshAddress'.
-A,--nettest-node-url STRING
[Morley.Nettest] Remote node URL
-P,--nettest-node-port NATURAL NUMBER
[Morley.Nettest] Remote node port
-I,--nettest-client-path STRING
[Morley.Nettest] Path to tezos-client binary
-d,--nettest-data-dir STRING
[Morley.Nettest] Path to tezos-client data directory
-S,--nettest-use-https [Morley.Nettest] Use HTTPS to communicate with the
remote node
-V,--nettest-verbose [Morley.Nettest] Increase verbosity (pass several
times to increase further)
--nettest-import-secret-key SECRET_KEY
[Morley.Nettest] Saves input secret key with a
`nettest` alias. This key will be used when
performing nettest actions
--nettest-skip-network [Morley.Nettest] Run only emulator tests and skip
network tests.
--hedgehog-replay ARG Replay token to use for replaying a previous test run
--hedgehog-show-replay ARG
Show a replay token for replaying tests
--hedgehog-tests ARG Number of successful test cases required before
Hedgehog will pass a test
--hedgehog-discards ARG Number of discarded cases allowed before Hedgehog
will fail a test
--hedgehog-shrinks ARG Number of shrinks allowed before Hedgehog will fail a
test
--hedgehog-retries ARG Number of times to re-run a test during shrinking
Related issue(s)
Resolves #365 (closed)
✅ Checklist for your Merge Request
Related changes (conditional)
-
Tests (see short guidelines)
-
If I added new functionality, I added tests covering it. -
If I fixed a bug, I added a regression test to prevent the bug from silently reappearing again.
-
-
Documentation
Stylistic guide (mandatory)
-
My commits comply with the following policy. -
My code complies with the style guide.