Skip to content

Draft: [#578] Handle callstacks properly for hedgehog errors

Description

Help needed: I've spent way too much time trying to write tests, but ultimately failed miserably. Long story short, tasty-hedgehog doesn't export anything from its internals, so running a Hedgehog test is only possible through testProperty, which means using launchTestTree, which means captured output depends on the value of TERM environment variable. Furthermore, since traces are produced by Hedgehog, I didn't manage to find a way to limit context size, which means expected output gets into the context and matches with itself, so tests give out false negatives. So, in summary, I need some help with this. EDIT: I've added a few tests in e1fba6c0. It is pretty awkward, but doable.

Quirk: The original issue is about pure exceptions. Unfortunately, pure exceptions do not usually carry much if any meaningful stacktraces. The best we can do is to show the trace at the place where the error is caught, i.e. wherever *TestProp* was called. It's still better than pointing to cleveland internals, but not "great". There's not much we can do about this.


Problem: When errors (assertion or otherwise) are thrown from a scenario inside Hedgehog property, the call stack can point somewhere into the cleveland internals, not into the user code.

The problem is twofold. First, IO errors are handled internally by Hedgehog in 'evalIO', so we need to catch and rethrow those to avoid that. Second, Hedgehog handles call stacks internally also, and assumes source location is on the top of call stack, which in cleveland almost always wrong.

Solution: Solve both issues

  • Catch and rethrow exceptions form the scenario inside property
  • Handle exceptions in *TestProp* manually: catch, check for 'WithCallStack', use lowest stack element for source position, generate a Hedgehog failure.

Related issue(s)

Resolves #578 (closed)

Checklist for your Merge Request

Related changes (conditional)

  • Tests (see short guidelines)

    • If I added new functionality, I added tests covering it.
    • If I fixed a bug, I added a regression test to prevent the bug from silently reappearing again.
  • Documentation

    • I checked whether I should update the docs and did so if necessary:
    • I updated changelog files of all affected packages released to Hackage if my changes are externally visible.

Stylistic guide (mandatory)

Edited by Nikolay Yakimov

Merge request reports

Loading