Pipeline job trace not fully updated when runner cancelled
Summary
Bug reported by customer on zd 217349
When a pipeline job gets cancelled the runner is not able to send the last traces because the build token as expired so the job trace is not updated. This causes a discrepancy between job trace logs, and runners log.
- The flow here observed in the customer case is definitely expected and works as we (GitLab) wanted it to work. However, there is one problem with this flow: the last few lines - from between when last update before cancel was done and when job is fully terminated - will be never added to the job log. And they may contain some important information.
Steps to reproduce
- Create a project
- Create a gitlab-ci.yml file
- Have the pipeline trigger job.
- Cancel the job
- Compare runner syslog with pipeline job trace
Example Project
What is the current bug behavior?
Pipeline job trace output ends up short from what actually the runner logs. the last traces don't get log at the pipelien job log.
What is the expected correct behavior?
Pipeline job trace output and the runner log should match.
Relevant logs and/or screenshots
Slack discussion here with #g_runner group
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)