Don't use test binary as runner/runner helper in tests
We're incorrectly using the test binary as the Runner binary for the shell integration tests.
Similar to how the custom-executor compiles a binary before tests are executes, the same could be done for Runner.