2024-10-18: GitLab Runner v17.5 breaking bugs
Customer Impact
Self-managed Runners are failing after they are upgraded to 17.5
Current Status
-
2024-10-18T21:30:00Z Runner version 17.5.1 has been released and includes a fix for the "failed to get user home dir: $HOME is not defined" error. No action besides upgrading Runner is required. Visit the following docs for how to upgrade the
gitlab-runner
version: - 2024-10-18T10:00:00Z Self-managed Runners are failing after they are upgraded to 17.5. The current workaround is to downgrade affected Runners back to version 17.4.
More information will be added as we investigate the issue. For customers believed to be affected by this incident, please subscribe to this issue or monitor our status page for further updates.

References and helpful links

Recent Events (available internally only):
- Feature Flag Log - Chatops to toggle Feature Flags Documentation
- Infrastructure Configurations
- GCP Events (e.g. host failure)
Deployment Guidance
- Deployments Log | Gitlab.com Latest Updates
- Reach out to Release Managers for S1/S2 incidents to discuss Rollbacks, Hot Patching or speeding up deployments. | Rollback Runbook | Hot Patch Runbook
Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:
- Corrective action ❙ Infradev
- Incident Review ❙ Infra investigation followup
- Confidential Support contact ❙ QA investigation
Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in our handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.
Security Note: If anything abnormal is found during the course of your investigation, please do not hesitate to contact security.
No timeline items have been added yet.
- Show closed items
Relates to
- gitlab-org/gitlab-runner #38259
- gitlab-org/gitlab-runner #38254
- gitlab-org/gitlab-runner #38252
- gitlab-org/gitlab-runner #38255
- gitlab-org/gitlab-runner #3830317.10
- gitlab-org/gitlab-runner #3830217.8
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Ghost User added an incident timeline event
added an incident timeline event
- A deleted user added IncidentActive Source::IMAIncidentDeclare incident severity3 labels
added IncidentActive Source::IMAIncidentDeclare incident severity3 labels
- Ghost User assigned to @jcstephenson
assigned to @jcstephenson
- Ghost User changed the description
Compare with previous version changed the description
- Ghost User changed the severity to Medium - S3
changed the severity to Medium - S3
- Ghost User added a resource link
added a resource link
- Alvin Gounder mentioned in merge request gitlab-com/support/support-pages!52 (merged)
mentioned in merge request gitlab-com/support/support-pages!52 (merged)
- 🤖 GitLab Bot 🤖 added ServiceNeeded label
added ServiceNeeded label
- Alvin Gounder mentioned in issue gitlab-org/gitlab-runner#38255 (closed)
mentioned in issue gitlab-org/gitlab-runner#38255 (closed)
- Alvin Gounder mentioned in issue gitlab-org/gitlab-runner#38256 (closed)
mentioned in issue gitlab-org/gitlab-runner#38256 (closed)
- Michele Bursi added severity2 label and removed severity3 label
- Ghost User changed the severity to High - S2
changed the severity to High - S2
- Owner
Upgrading this so a severity2 as the impact is broadening.
- Runners teams is investigating the issue.
- Currently GitLab.com Runners are not currently impacted and fully operational.
- The impact is currently narrowed down to self-hosted runners.
- The team is looking at short-term mitigation for customers
- Status is page will be updated to reflect the incident and impact
cc: @kwanyangu as IMOC
- Kennedy Wanyangu changed the description
Compare with previous version changed the description
- Maintainer
Hi @jcstephenson,
This issue now has the CorrectiveActionsNeeded label, this label will be removed automatically when there is at least one related issue that is labeled with corrective action or infradev. Having an issue related with these labels helps to ensure a similar incident doesn't happen again.
If you are certain that this incident doesn't require any corrective actions, add the CorrectiveActionsNotNeeded label to this issue with a note explaining why.
You are welcome to help improve this comment.
- 🤖 GitLab Bot 🤖 added CorrectiveActionsNeeded label
added CorrectiveActionsNeeded label
- Maintainer
Hi @jcstephenson,
Thanks for taking part in this incident! It looks like this incident needs an async Incident Review issue, please use the Incident Review link in the incident's description to create one.
We're posting this message because this issue meets the following criteria:
- It is severity1 / severity2, or has a review-requested label
- There is no related issue with an incident-review label
If you are certain that this incident doesn't require an incident review, add the IncidentReviewNotNeeded label to this issue with a note explaining why.
Thanks for your help!
You are welcome to help improve this comment.
- 🤖 GitLab Bot 🤖 added IncidentReviewNeeded label
added IncidentReviewNeeded label
- Owner
Next Steps
-
@tmaczukin to revert for the culprit commit to the
17-5-stable
branch. - @ajwalker help @tmaczukin with merging the revert and getting it ready for release (1.5h to merge)
-
@tmaczukin @ajwalker Wait for the green pipeline on the
17-5-stable
branch. -
@tmaczukin to trigger a new release to
17.5.1
with the revert (2h).
-
@tmaczukin to revert for the culprit commit to the
Collapse replies - Developer
The MR with the revert: Fix home directory detection (gitlab-org/gitlab-runner!5087 - merged)
- Developer
Along with the
$HOME
problem, we had reports of another issue too:That will be fixed with gitlab-org/gitlab-runner!5088 (merged).
We should also release this along with 17.5.1.
- Kennedy Wanyangu added Hosted RunnersLinux label
added Hosted RunnersLinux label
- Kennedy Wanyangu removed Hosted RunnersLinux label
removed Hosted RunnersLinux label
- Kennedy Wanyangu added ServiceCI Runners label and removed ServiceNeeded label
added ServiceCI Runners label and removed ServiceNeeded label
- Vlad Mihai mentioned in issue gitlab-org/gitlab-runner#38252 (closed)
mentioned in issue gitlab-org/gitlab-runner#38252 (closed)
- Ghost User mentioned in issue on-call-handovers#5417 (closed)
mentioned in issue on-call-handovers#5417 (closed)
- Developer
Status update 2024-10-18 14:03 UTC
- Both MRs with the fixes have been merged:
- The changes from both MRs have been cherry-picked into the
17-5-stable
branch - We have to wait for another ≈ 1.5h for the pipeline on the
17-5-stable
branch to succeed - Cut the
17.5.1
release (DRI: @tmaczukin)
Edited by Denys Mishunov - Michael Hofmann mentioned in incident cki-project/infrastructure#402 (closed)
mentioned in incident cki-project/infrastructure#402 (closed)
- Samuel Bachmann mentioned in issue gitlab-org/gitlab-runner#38259 (closed)
mentioned in issue gitlab-org/gitlab-runner#38259 (closed)
- Darren Eastman marked this incident as related to gitlab-org/gitlab-runner#38259 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38259 (closed)
- Darren Eastman marked this incident as related to gitlab-org/gitlab-runner#38254 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38254 (closed)
- Darren Eastman marked this incident as related to gitlab-org/gitlab-runner#38256 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38256 (closed)
- Darren Eastman marked this incident as related to gitlab-org/gitlab-runner#38261
marked this incident as related to gitlab-org/gitlab-runner#38261
- Darren Eastman removed the relation with gitlab-org/gitlab-runner#38261
removed the relation with gitlab-org/gitlab-runner#38261
- Darren Eastman marked this incident as related to gitlab-org/gitlab-runner#38260 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38260 (closed)
- Darren Eastman marked this incident as related to gitlab-org/gitlab-runner#38252 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38252 (closed)
- Paul Slaughter changed the description
Compare with previous version changed the description
- Paul Slaughter added IncidentResolved label and removed IncidentActive label
added IncidentResolved label and removed IncidentActive label
- Developer
update: Runner version 17.5.1 has been released and includes a fix for the "failed to get user home dir: $HOME is not defined" error and others that were reported (see thread). No action besides upgrading Runner is required. Visit the following docs for how to upgrade the
gitlab-runner
version:- https://docs.gitlab.com/runner/install/linux-repository.html
- https://docs.gitlab.com/runner/install/osx.html
- https://docs.gitlab.com/runner/install/windows.html
- https://docs.gitlab.com/runner/install/docker.html
Marking this incident as resolved and closing it per the handbook.
- Paul Slaughter closed
closed
- Paul Slaughter changed the incident status to Resolved by closing the incident
changed the incident status to Resolved by closing the incident
- Developer
question: Should we kick off an IncidentReview for this? I don't have a lot of context here, so please respond to this thread, and I'll make it happen
Collapse replies - Developer
@pslaughter , a customer request an RCA for this incident. It's be great if we have one, thank you
- Developer
Thanks for the update @rotanak! I created an incident review and included a Root-Cause-Analysis section here Incident Review: GitLab Runner v17.5 breaking bugs (#18744 - closed).
- Alvin Gounder marked this incident as related to gitlab-org/gitlab-runner#38255 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38255 (closed)
- Sokunrotanak Srey (Rotanak) added Incident-CommsStatus-Page label
added Incident-CommsStatus-Page label
- GitLab Release Tools Bot mentioned in issue gitlab-org/release/tasks#13773 (closed)
mentioned in issue gitlab-org/release/tasks#13773 (closed)
- ops-gitlab-net mentioned in issue reliability-reports#266 (closed)
mentioned in issue reliability-reports#266 (closed)
- Paul Slaughter marked this incident as related to #18744 (closed)
marked this incident as related to #18744 (closed)
- Paul Slaughter mentioned in issue #18744 (closed)
mentioned in issue #18744 (closed)
- Developer
update: An incident review issue was created for this incident with @tmaczukin assigned as the DRI. If you have any review feedback please add it to Incident Review: GitLab Runner v17.5 breaking bugs (#18744 - closed).
@tmaczukin please feel free to reassign review DRI if there's someone more appropriate for the role
2 1 - 🤖 GitLab Bot 🤖 removed IncidentReviewNeeded label
removed IncidentReviewNeeded label
- Maintainer
Hi @jcstephenson @kkyrala @jarv
This issue has CorrectiveActionsNeeded set. Please either fix this by adding one of the corresponding labels, or add a
::NotNeeded
scoped label with an explanation if you are sure that it is not needed. Adding the::NotNeeded
scoped label will prevent these notifications, otherwise this notice will repeat in 7 days.Once the criteria is met, CorrectiveActionsNeeded label will be removed automatically. See the handbook page on
::Needed
and::NotNeeded
labels for more information about how the labels are set and removed.
You are welcome to help improve this comment.
- Nicole Williams mentioned in issue gitlab-org/gitlab-runner#37359 (closed)
mentioned in issue gitlab-org/gitlab-runner#37359 (closed)
- Darren Eastman mentioned in issue gitlab-com/Product#13748 (closed)
mentioned in issue gitlab-com/Product#13748 (closed)
- Maintainer
Hi @jcstephenson @kkyrala @jarv
This issue has CorrectiveActionsNeeded set. Please either fix this by adding one of the corresponding labels, or add a
::NotNeeded
scoped label with an explanation if you are sure that it is not needed. Adding the::NotNeeded
scoped label will prevent these notifications, otherwise this notice will repeat in 7 days.Once the criteria is met, CorrectiveActionsNeeded label will be removed automatically. See the handbook page on
::Needed
and::NotNeeded
labels for more information about how the labels are set and removed.
You are welcome to help improve this comment.
- Maintainer
Hi @jcstephenson @kkyrala @jarv
This issue has CorrectiveActionsNeeded set. Please either fix this by adding one of the corresponding labels, or add a
::NotNeeded
scoped label with an explanation if you are sure that it is not needed. Adding the::NotNeeded
scoped label will prevent these notifications, otherwise this notice will repeat in 7 days.Once the criteria is met, CorrectiveActionsNeeded label will be removed automatically. See the handbook page on
::Needed
and::NotNeeded
labels for more information about how the labels are set and removed.
You are welcome to help improve this comment.
- Maintainer
Hi @jcstephenson @kkyrala @jarv
This issue has CorrectiveActionsNeeded set. Please either fix this by adding one of the corresponding labels, or add a
::NotNeeded
scoped label with an explanation if you are sure that it is not needed. Adding the::NotNeeded
scoped label will prevent these notifications, otherwise this notice will repeat in 7 days.Once the criteria is met, CorrectiveActionsNeeded label will be removed automatically. See the handbook page on
::Needed
and::NotNeeded
labels for more information about how the labels are set and removed.
You are welcome to help improve this comment.
Collapse replies - Owner
There has already been a retrospective for this incident, where corrective actions were created.
I've cross-linked the corrective actions to be related to this incident to satisfy the condition triggering this reminder message.
- Jack Stephenson marked this incident as related to gitlab-org/ci-cd/tests/runner-incept#35 (closed)
marked this incident as related to gitlab-org/ci-cd/tests/runner-incept#35 (closed)
- Jack Stephenson marked this incident as related to gitlab-org/gitlab-runner#38303
marked this incident as related to gitlab-org/gitlab-runner#38303
- Jack Stephenson marked this incident as related to gitlab-org/ci-cd/tests/runner-incept#36
marked this incident as related to gitlab-org/ci-cd/tests/runner-incept#36
- Jack Stephenson marked this incident as related to gitlab-org/gitlab-runner#38305 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38305 (closed)
- Jack Stephenson marked this incident as related to gitlab-org/gitlab-runner#38306 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38306 (closed)
- Jack Stephenson marked this incident as related to gitlab-org/gitlab-runner#38304 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38304 (closed)
- Jack Stephenson marked this incident as related to gitlab-org/gitlab-runner#38302 (closed)
marked this incident as related to gitlab-org/gitlab-runner#38302 (closed)
- 🤖 GitLab Bot 🤖 removed CorrectiveActionsNeeded label
removed CorrectiveActionsNeeded label
- John Skarbek mentioned in incident #18984 (closed)
mentioned in incident #18984 (closed)