kubernetes executor should use attach instead of exec to run build scripts
When starting job POD, kubernetes executor configures it to start with detected shell waiting for commands on stdin. It is however not sending commands to that shell, instead it is using what is essentially a kubectl exec
.
It becomes a problem, when long running command is executed. Connection goes at least via APIServer and Kubelet, which both must be up for the whole duration of the command, this is not very cloud-native, because cluster operators are free to restart these components for maintenance/updates, which break long running build. executor cannot repeat exec
as it is not idempotent and therefore not safe.
If kubernetes executor used attach
to a running pod to send commands, then even if connection to pod is disturbed, it doesn't stop command from being executed, runner can always reattach to same pod and continue receiving job output without reporting a failure