Revise stdout/stderr handling of Executors

Before we can separate the submission and monitoring of jobs, we need to ensure that stdout, stderr (and stdin) are handled asynchronously of the connection. This means that all executors should optionally write the stdout and stderr to files (and read the stdin from a file).

  • SshExecutor: stdout/stderr/stdin should be provided with the paths on the remote host.
  • LocalExecutor: the three files need to be provided with the path on the local host (worker)
  • ClusterExectors: the three files need to provided with the path on the cluster.

As inputs and outputs they should be part of the ShellCommand.

At the some time it is important that there is at least the option to read the output as stream/pipe synchronously. For instance, it must be possible to read stdout and stderr of bsub command executed by the LsfExecutor(LocalExecutor) or the LsfExecutor(SshExecutor) for the submission.

Therefore, the stdout, stderr, and stdin parameters need to have three possible values

  • A path. Then output is simply written to/read from a file.
  • A "null" value (maybe /dev/null). Then output/input is ignored.
  • A pipe or stream. Then the output/input can be written to/read from the pipe synchronously (needed for bsub etc.)

All three values need to be implemented for the LocalExecutor and the SshExecutor. For the ClusterExecutors a pipe/stream type can be implemented with an NotImplementedException, because we currently don't synchronous I/O for them.

Edited by Philip Reiner Kensche