Consolidate helpers to run tests in subprocess

The following discussion from !1499 (merged) should be addressed:

• @BenjaminSchubert started a discussion: (+1 comment)

  We seem to have multiple places that run tests in subprocess. Could we consolidate the setup for it in a single place maybe?

  A (nice?) improvement could be to add a custom pytest test collector, that recognizes names like subprocess_test, which would automatically run the test in a subprocess. We can also defer this to later.