<!-- The purpose of this Incident Review is to serve as a classroom to help us better understand the root causes of an incident. Treating it as a classroom allows us to create the space to let us focus on devising the mechanisms needed to prevent a similar incident from recurring in the future. A root cause can **never be a person** and this Incident Review should be written to refer to the system and the context rather than the specific actors. As placeholders for names, consider the usage of nouns like "technician", "engineer on-call", "developer", etc.. --> ## Key Information <!--- Customers Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "All users with repositories on file-01 node" Requests Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "Any requests from Australia during the CloudFlare outage" Incident Severity: Copy from the incident. Use the severity at the end of the incident, not the beginning. Use the label rather than text - "~Severity::*" Start Time: Use the time when service started to be effected. Not the time the incident was opened (if they are different) End Time: Use the time when the incident was mitigated. Don't include the time in a monitoring state unless the monitoring indicated continuing problems. Total Duration: Use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations. Link to Incident Issue: Link to Incident Issue ---> | Metric | Value | | ------ | ------ | | Customers Affected | None (GitLab only) See: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19438#note_2384177539 | | Requests Affected | `PUT /api/jobs/:id` (api endpoint runner uses to update jobs) | | Incident Severity | ~"severity::2" | | Start Time | [Mar 6, 2025, 3:47 AM UTC](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19438) | | End Time | [Mar 7, 2025, 6:56 PM UTC](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19438#note_2385908983) | | Total Duration | 39.15 hours (1 day, 15hr, 9 min) | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19438 | ## Summary On March 6th, GitLab internal users began reporting that jobs remained stuck in "running" status for minutes or hours after logs indicated successful completion. This issue stemmed from requests being incorrectly classified as unauthenticated, resulting in lower IP-based rate limits. The problem exclusively affected GitLab internal users, who encountered these unauthenticated request rate limits. Additionally, the runner-side retry mechanism for updates was broken, preventing runners from updating job status during retry attempts. The impact was limited to internal GitLab customers since the issue only affected private runners. ## Details On March 6th internal GitLab users started to report that jobs were stuck in running for minutes/hours even after the logs printed that the jobs had succeeded. 1. The runner's job update requests to GitLab's `PUT /api/v4/jobs/:job_id` were rate limited because they were incorrectly classified as unauthenticated. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/183764 2. This occurred because the RackAttack rate limiter on the GitLab Rails side failed to properly handle job tokens that were simultaneously passed in both the header and request parameters. - `RackAttack` only handled properly the condition where the job token was passed in the request parameters. - If the job token was passed in the header under `Private-Token`, then Rack Attack attempted to look up the user associated with the PAT, and finding none, it assumed this was an unauthenticated request. - Unauthenticated requests caused a lower rate limit per ip address which GitLab hit on March 6th 3. The [runner retry mechanism was also broken](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/38651) so that when the runner retried the update request it sent an empty body instead of rewinding and retrying with the body from the last request. FAQ: - Will this affect CI minutes? While this would have increased CI minute usage it has limited impact since only internal users were affected and gitlab namespaces have a quota of 0 which gives us unlimited minutes. - Was there an associated code change that suddenly caused gitlab rails to not handle the job token in the header? Likely, not a recent change. This Auth could have been failing silently for a long time because it won't actually block the request it just changes the rate limit for the ip address that is considered 'unathenticated'. We should address this by using a single source of truth for the job token auth code in both RackAttack and in the api. Additionally, we should consider if requests continue on to the api layer if they are un-authenticated or if we should alert if they fail the authentication at one layer and pass it at another. Stan identified some older code changes that contributed: > In summary, I think there was a confluence of a number of events: > > 1. GitLab Runner v16.10 added the `PRIVATE-TOKEN` header to support Cells routing. > 2. This had the side effect of causing `PUT /api/v4/jobs/:id` requests to land in the `throttle_unauthenticated_api` bucket. > 3. Based on https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/25809#note_2227059901, the limit for `throttle_unauthenticated_api` has not changed since at least since 2024-11. > 4. CI pipelines in `gitlab-org` projects have gradually added more builds over time. > 5. Users started many simultaneous pipelines around the time in question. - Why was this only affecting private/internal runners? We hit the lower rate limit that was being applied to the ip address if the request was considered unauthenticated. ## Outcomes/Corrective Actions <!-- This section should be used to link any corrective action issues or epics which have been created as a result of this incident. If there are vague learnings which do not have an obvious action, use the "What can be improved" section below --> 1. Log the response to from the GitLab in the runner logs - [https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19438#note_2383859631](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/38656) 1. Fix the broken runner retry mechanism (in dev) - https://gitlab.com/gitlab-org/gitlab-runner/-/issues/38651 1. Fix in-consistent logging of the `method` at the RackAttack layer - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/183888 1. Single source of truth for Authentication logic in APIGuard and RackAttack (auth group scoped issue) - https://gitlab.com/gitlab-org/gitlab/-/issues/524408 1. Apply separate RackAttack rate limits for internal API calls - https://gitlab.com/gitlab-org/gitlab/-/issues/524410 1. Fix Rack Attack incorrectly rate limiting runner API (merged) - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/183764 1. Alerting for stuck jobs (jobs failing to update gitlab rails) - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19468 1. Rack Attack and APIguard establish correct order for job token lookups (verify scoped issue) - https://gitlab.com/gitlab-org/gitlab/-/issues/524645 1. Stop passing in job-token to `PRIVATE-TOKEN` header - https://gitlab.com/gitlab-org/gitlab-runner/-/issues/38677 ## Learning Opportunities ### What went well? <!-- Use this section to highlight what went well during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. _example:_ 1. We quickly discovered a recently changed feature flag through the event log which enabled fast mitigation of the impact, as well as pulling in the engineer involved to further diagnose. 2. We escalated through dev escalations, which brought in Person X. They knew that Person Y had expertise with the component in question, which enabled faster diagnosis. --> 1. Great detective work by numerous developers/SRE's to find the root cause despite ambiguous logs 2. No customer impact - only internal impact to the GitLab team ### What was difficult? <!-- Use this section to highlight opportunities for improvement discovered during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. If the improvement seems like a simplest change, consider adding it as a corrective action above instead. This section can be empty if all of the improvements can have corrective actions. _example:_ 1. The runbooks/playbooks for this service are out of date and did not contain the information necessary to troubleshoot the incident. 2. The incident happened at a time when nobody with expertise on the service was available. --> 1. Logging was incomplete and inconsistent, making it difficult to find why the updates from runner to GitLab were failing - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19438#note_2385407012 - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19438#note_2383859631 2. We have multiple stacks for authentication and different code paths to authenticate the job token. We should be using a SSoT. - https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19438#note_2385000030 3. We had no alerting that told us the jobs were stuck, instead we reactively relied on internal customer reports. 4. Bugs in more than one component - Runner bug around retries exacerbating the gitlab rails bug 5. We thought we found the root cause when we found some 500 errors for the endpoint but they were not the cause. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19438#note_2383676673 ## Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** - [x] Announce the incident review in the incident channel on Slack. ``` :mega: @here An incident review issue was created for this incident with <USER> assigned as the DRI. If you have any review feedback please add it to <ISSUE_LINK>. ``` ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Use the description section to write a few paragraphs explaining what happened - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Add any appropriate labels based on the incident issue and discussions - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] Close the review before the due date