Incomplete log collected when job cancelled, no indication in log of timestamp when job was terminated
Summary
When a job is cancelled log captured by GitLab is missing the most recent output generated by the job before it terminated. IT is also missing any indication of when the job was actually terminated.
This is important for a couple of reasons:
- the captured job log should accurately capture everything that was output up until the instant the job ceased to run, so that users know exactly what tasks were and were not executed by the job, especially operations which update data or systems outside of the job itself
- having a timestamped entry written to the job log just before the container terminates provides assurance to users that jobs are in fact ceasing to run at the time indicated by the GitLab front end, and that all output log data has been captured
This issue was raised by a Premium customer in ZD internal link.
Steps to reproduce
- create a job which executes the following script
counter.sh
:
i=1
while [ $i -lt 900 ]
do
echo "$i: `date`" | tee -a /tmp/count.out
sleep 1
i=`expr $i + 1`
done
- trigger the job, and then log into the job container and run
tail -f /tmp/count.out
- cancel the job in the GitLab UI
- observe that the
tail
output includes a few extra lines that the job log itself
.gitlab-ci.yml
stages:
- build
image: ubuntu:latest
build-job:
stage: build
script:
- sh counter.sh
Actual behavior
Final lines of job log captured by GitLab:
67: Sun Apr 2 07:01:10 UTC 2023
68: Sun Apr 2 07:01:11 UTC 2023
69: Sun Apr 2 07:01:12 UTC 2023
Final lines of tail -f
output from job container itself:
70: Sun Apr 2 07:01:13 UTC 2023
71: Sun Apr 2 07:01:14 UTC 2023
72: Sun Apr 2 07:01:15 UTC 2023
Expected behavior
Relevant logs and/or screenshots
job log
Add the job log
Environment description
config.toml contents
Add your configuration here