Ensure `NettestEnv` is not evaluated before the tests are run
Clarification and motivation
For context, please read this comment:
https://github.com/feuerbach/tasty/issues/228#issuecomment-718081424
Say we have a test like the below, where:
- we use network capabilities
- and we want to omit it from the tasty logs when
--nettest-no-run-network
is on,
withNettestEnv $ \envMb ->
testGroup "On network" $
case envMb of
Nothing -> []
Just env ->
[ testGroup "compare interpreter with reference implementation"
(mkTestTree env)
]
In the CI, we might want to invoke the test suite with --nettest-no-run-network
, and without putting tezos-client
on the PATH
:
nix-build ci.nix -A packages.cleveland.tests.morley-test
cd code/cleveland
../../result/bin/morley-test --nettest-no-run-network --xml=../../test-morley-report.xml
In light of the issue describe in comment linked above, we now know that the first time the TestTree is traversed, all options will have their default values, so NoRunNetworkOpt
will be Unset
, and therefore envMb
will return Just env
.
Matching on envMb
forces the IO thunk (because of the way nettestEnvFromOpts
was written). And because tezos-client
is not on the PATH
, the IO will crash:
https://gitlab.com/morley-framework/morley/-/jobs/817164821#L39
So it's important that IO NettestEnv
is not evaluated while the TestTree
is being built/traversed, but only when the tests start running.
We could solve in one of two ways:
- changing
nettestEnvFromOpts
's type fromIO (Maybe NettestEnv)
toMaybe (IO NettestEnv)
, that way we could scrutinize theMaybe
layer without forcing evaluation of the IO thunk. But, if we ever accessed aNettestEnv
field we'd fall into the same trap again. - don't use
unsafePerformIO
, changewithNettestEnv
from(Maybe NettestEnv -> TestTree) -> TestTree
(Maybe (IO NettestEnv) -> TestTree) -> TestTree
IO NettestEnv
would internally use aMVar
to cache its result, such that evaluating it multiple times would only create aNettestEnv
once (see theonce
orio-memoize
packages).
I think we should avoid using unsafePerformIO
altogether. This means not caching the NettestEnv
, and creating a new one from scratch for every test, but I don't think this matters much, the suite's running time is completely dominated by the tests themselves anyway.
Turns out caching NettestEnv
is important (see this comment), so let's go with Option 2 instead.
Acceptance criteria
Running the above test with --nettest-no-run-network
and without tezos-client
on the PATH does not crash.