Run only network tests without specifying a pattern from the cmd line
Clarification and motivation
Currently, if you want to run only cleveland's network tests, and skip cleveland's emulator and "regular" tasty tests, you have to use a pretty complex tasty pattern, like the one we're using in our CI:
- nix run -f ../../ci.nix tezos-client
-c ../../result/bin/morley-test
--nettest-run-network
--pattern "\$1 == \"On network\" || \$NF == \"On network\" || \$0 ~ /.On network./"
--xml=../../test-report.xml
This is unfortunate, and it becomes a burden when you have to adapt this pattern to work with different CIs.
For example, buildkite's escaping of $
works differently than gitlab's ci.
Ideally, we'd like to be able to specify a simple flag in the command line, like --nettest-run-only-network
, and then we'd programmatically create the corresponding pattern to filter the test tree.
There used to be a bug in tasty
that prevented us from doing this programmatically, but it's since been fixed and released in tasty 1.4.
Beware that we'll have to decide whether this flag takes precedence over the related flags (--nettest-run-network
, --nettest-no-run-network
, and the CI
env variable).
We should update the documentation here, and the tests here.
Acceptance criteria
- We can pass a flag like
--nettest-run-only-network
which skips all non-network tests - We've updated our CI script to use this flag
- We've updated the related docs and tests