Skip to content

Kube attach strategy hangs when log file is deleted.

Summary

When running with FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY set to false and you delete the log file from disk the job ends up running until the job times out.

Steps to reproduce

.gitlab-ci.yml
variables:
  DURATION: 3600
  FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY: "false"

job:
  script:
  - 'for i in $(seq 1 $DURATION); do echo $(date); sleep 1; done'
  - echo "done"
  1. Run a the above .gitlab-ci.yml file.
  2. When the job log start to output the date-time run the follow command to delete the log file kubectl exec -it -c helper $POD_RUNNING_JOB -- sh -c 'rm /builds/root/playground.tmp/logs/output.log'. /builds/root/playground.tmp` should be the path where the build is running.

Actual behavior

We get an error that a file doesn't exist anymore and a timeout error and job runs forever.

Expected behavior

Gracefully fail the job

Relevant logs and/or screenshots

Screen_Shot_2020-06-08_at_14.32.35

Environment description

config.toml contents
[[runners]]
  name = "kubernetes"
  url = "http://192.168.190.160:3000"
  token = "o5hBR1gQYzi6gwMvb_d1"
  executor = "kubernetes"
  [runners.kubernetes]
    image = "alpine:3.11"
    privileged = true
    namespace = "default"

Used GitLab Runner version

Master branch

Edited by Romuald Atchadé