Skip to content

[#365] Added Tasty integration

Diogo Castro requested to merge diogo/#365-tasty into master

Description

Problem: we have no good way of structuring our nettests. As a result, we have to resort to comment to indicate which tests succeeded/failed, and the output is all jumbled together and hard to understand/parse. Furthermore, when one test fails, the whole suite aborts without executing the remaining tests.

Solution: integrate Morley.Nettest with Tasty. This way, we get all the benefits that come with using Tasty:

  • structured tests and output
  • skipping/running only a few tests using --pattern
  • run all tests and report those that failed
  • ability to have hunit/hspec/hedgehog + nettests all in the same executable
  • ability to pass config options via environment variables as well as CLI options

(Note: by convention, an env variable name is the same as the CLI option name, except it's all caps, underscores instead of hyphens, and prefixed with TASTY_. For example, --nettest-skip-network corresponds to TASTY_NETTEST_SKIP_NETWORK).

Example usage (with caps interface):

test :: TestTree
test = nettestScenarioCaps "storage is 1" $ do
  addr <- addressResolved <$> originate OriginateData {..}
  checkStorage addr (toVal @Natural 1)
test :: TestTree
test = nettestScenarioCaps "storage is 1" scenario

scenario :: MonadNettest caps base m => m ()
scenario = do
  addr <- addressResolved <$> originate OriginateData {..}
  checkStorage addr (toVal @Natural 1)

Example usage (without caps interface):

test :: TestTree
test = nettestScenario "storage is 1" $ \impl
  addr <- addressResolved <$> niOriginate impl OriginateData {..}
  niCheckStorage impl addr (toVal @Natural 1)
test :: TestTree
test = nettestScenario "storage is 1" $ scenario

scenario :: NettestScenario m
scenario impl = do
  addr <- addressResolved <$> niOriginate impl OriginateData {..}
  niCheckStorage impl addr (toVal @Natural 1)

Tasty's --help now looks like this:

Mmm... tasty test suite

Usage: cleveland-test [-p|--pattern PATTERN] [-t|--timeout DURATION] 
                      [-l|--list-tests] [-j|--num-threads NUMBER] [--xml ARG] 
                      [-q|--quiet] [--hide-successes] 
                      [--color never|always|auto] [--ansi-tricks ARG] 
                      [--nettest-alias-prefix STRING] 
                      [-A|--nettest-node-url STRING] 
                      [-P|--nettest-node-port NATURAL NUMBER] 
                      [-I|--nettest-client-path STRING] 
                      [-d|--nettest-data-dir STRING] [-S|--nettest-use-https] 
                      [-V|--nettest-verbose] 
                      [--nettest-import-secret-key SECRET_KEY] 
                      [--nettest-skip-network] [--hedgehog-replay ARG] 
                      [--hedgehog-show-replay ARG] [--hedgehog-tests ARG] 
                      [--hedgehog-discards ARG] [--hedgehog-shrinks ARG] 
                      [--hedgehog-retries ARG]

Available options:
  -h,--help                Show this help text
  -p,--pattern PATTERN     Select only tests which satisfy a pattern or awk
                           expression
  -t,--timeout DURATION    Timeout for individual tests (suffixes: ms,s,m,h;
                           default: s)
  -l,--list-tests          Do not run the tests; just print their names
  -j,--num-threads NUMBER  Number of threads to use for tests execution
  --xml ARG                A file path to store the test results in
                           Ant-compatible XML
  -q,--quiet               Do not produce any output; indicate success only by
                           the exit code
  --hide-successes         Do not print tests that passed successfully
  --color never|always|auto
                           When to use colored output (default: 'auto')
  --ansi-tricks ARG        Enable various ANSI terminal tricks. Can be set to
                           'true' (default) or 'false'.
  --nettest-alias-prefix STRING
                           [Morley.Nettest] A prefix to prepend to every alias
                           created with 'newAddress' or 'newFreshAddress'.
  -A,--nettest-node-url STRING
                           [Morley.Nettest] Remote node URL
  -P,--nettest-node-port NATURAL NUMBER
                           [Morley.Nettest] Remote node port
  -I,--nettest-client-path STRING
                           [Morley.Nettest] Path to tezos-client binary
  -d,--nettest-data-dir STRING
                           [Morley.Nettest] Path to tezos-client data directory
  -S,--nettest-use-https   [Morley.Nettest] Use HTTPS to communicate with the
                           remote node
  -V,--nettest-verbose     [Morley.Nettest] Increase verbosity (pass several
                           times to increase further)
  --nettest-import-secret-key SECRET_KEY
                           [Morley.Nettest] Saves input secret key with a
                           `nettest` alias. This key will be used when
                           performing nettest actions
  --nettest-skip-network   [Morley.Nettest] Run only emulator tests and skip
                           network tests.
  --hedgehog-replay ARG    Replay token to use for replaying a previous test run
  --hedgehog-show-replay ARG
                           Show a replay token for replaying tests
  --hedgehog-tests ARG     Number of successful test cases required before
                           Hedgehog will pass a test
  --hedgehog-discards ARG  Number of discarded cases allowed before Hedgehog
                           will fail a test
  --hedgehog-shrinks ARG   Number of shrinks allowed before Hedgehog will fail a
                           test
  --hedgehog-retries ARG   Number of times to re-run a test during shrinking

Related issue(s)

Resolves #365 (closed)

✅ Checklist for your Merge Request

Related changes (conditional)

  • Tests (see short guidelines)

    • If I added new functionality, I added tests covering it.
    • If I fixed a bug, I added a regression test to prevent the bug from silently reappearing again.

  • Documentation

    • I checked whether I should update the docs and did so if necessary:
    • I updated changelog files of all affected packages released to Hackage if my changes are externally visible.

Stylistic guide (mandatory)

Edited by Diogo Castro

Merge request reports