# [#INC-11437: git pushes are failing via ssh](https://app.incident.io/gitlab/incidents/11437) Generated by Cheryl Li on 23 Jun 2026 20:04. All timestamps are local to Etc/UTC # Key Information <!--- Customers Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "All users with repositories on file-01 node" Requests Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "Any requests from Australia during the CloudFlare outage" Incident Severity: Copy from the incident. Use the severity at the end of the incident, not the beginning. Use the label rather than text - "~Severity::*" Start Time: Use the time when service started to be effected. Not the time the incident was opened (if they are different) End Time: Use the time when the incident was mitigated. Don't include the time in a monitoring state unless the monitoring indicated continuing problems. Total Duration: Use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations. Link to Incident Issue: Link to Incident Issue ---> | Metric | Value | | ------ | ------ | | Customers Affected | All GitLab.com users | | Requests Affected | Approximately 10M git requests | | Incident Severity | Severity 1 | | Impact Start Time | Tue, 23 Jun 2026 16:55:00 UTC| | Impact End Time | Tue, 23 Jun 2026 18:23:00 UTC | | Total Duration | 1 hour, 2 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22364 | # Summary **Problem**: Git operations over SSH failed for many users due to authentication errors, preventing repository access. **Impact**: Git operations over SSH failed globally during the outage. **Causes**: GitLab Shell is the SSH gateway for GitLab — it handles incoming SSH connections (via its own SSH daemon, gitlab-sshd), authenticates users against the GitLab Rails internal API, and then proxies git operations to Gitaly over gRPC. This incident was triggered by a configuration change that made an existing TLS secret available to gitlab-shell. Along with adding the TLS secret, the change also enabled the topology service for gitlab-shell. Enabling the topology service activated new routing logic for /allowed requests, which gitlab-shell uses to authorize SSH operations. This routing logic had not been validated in the GitLab.com environment, where the topology service returned a cell address that was not valid as an internal API endpoint. **Response strategy**: We reverted the configuration that enabled the Topology Service for GitLab Shell and rolled out the fix across all zones. SSH access to repositories has been restored and we are monitoring for further issues. ## Timeline 1. **2026-01-23** [0da306a4f](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/commit/0da306a4f) — Cells and TLS enabled on Staging 1. **2026-03-16** [ae972bb7](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/commit/ae972bb7) — Cells and TLS enabled in Production 2. **2026-04-29** [!1435](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/1435) this added the funtionality to GitLab shell to route /allowed requests via Topology Service 3. **2026-06-11** [Add Topology Service mTLS support to GitLab Shell](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/5111) merged. This enables the topology service for gitlab shell when both `global.appConfig.cell.enabled` and `global.appConfig.cell.topologyServiceClient.tls.enabled` are `true`. These two settings were already enabled on both staging and production so deploying this config change would enable TS lookups on shell when deployed. 4. **2026-06-23** [Production Chart bump](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/5513) 1. This change was fully deployed to production before staging, see [timeline](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22163#note_3485644235) 5. **2026-06-23 16:40** Deployment starts on Production 6. **2026-06-23 17:23** Deployment finishes on Production 7. **2026-06-23 17:20** Incident declared 8. **2026-06-23 18:23** Full recovery # What went well? Use this section to highlight what went well during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. 1. Broad SRE response when joining the incident/zoom 2. Parallel remediation work to revert the problematic chart bump and to [update the config map](https://runbooks.gitlab.com/kube/k8s-operations/#emergency-configmap-edit) to restore service as fast as possible. # What was difficult? Use this section to highlight opportunities for improvement discovered during the incident. Capturing this helps understand informal processes and expertise, and enables undocumented knowledge to be shared. If the improvement seems like a simplest change, consider adding it as a corrective action above instead. Think about how to improve response next time, and consider any patterns pointing to broader issues, like “key person risk.” 1. Helm chart bumps are problematic to link together and gate due to structure of k8s-workloads. 2. Assessing if smoke tests had run in what environment for a helm chart bump. We can have better linkage of the MRs, their results from CI and results from smoke tests. 3. We need to look more closely at when and how we bump helm chart versions for GitLab.com so people knowledgeable of the changes can better know when one of their changes will roll out. 4. What alerting could have helped us catch onto this sooner? # Investigation Details <details> <summary> Timeline</summary> # Incident Timeline **2026-06-23** **16:55:00** **Impact started at** Custom timestamp "Impact started at" occurred **17:20:24** **Incident reported by Andrew Newdigate** Andrew Newdigate reported the incident Severity: Severity 2 Status: Investigating **17:20:29** **Escalated to GitLab.com Production** Your workflow manually escalated the incident to the escalation path [GitLab.com Production](https://app.incident.io/gitlab/on-call/escalation-paths/01K5YWH36WZZX2YT9HNRR0WMB0) **17:20:29** **Escalated to Incident Manager Oncall (IMOC)** Your workflow manually escalated the incident to the escalation path [Incident Manager Oncall (IMOC)](https://app.incident.io/gitlab/on-call/escalation-paths/01K8VN9TTAJH6N3WKNCAMNXQ40) **17:20:38** **Escalation acknowledged by Raynard Omongbale** Raynard Omongbale acknowledged the escalation **17:20:45** **Escalation acknowledged by José Iván Vargas López** José Iván Vargas López acknowledged the escalation **17:35:41** **Message from Igor Wiedler** Igor Wiedler's message was pinned by Dave Smith suspecting potentially https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/5513 **17:38:38** **Severity upgraded from Severity 2 → Severity 1** Raynard Omongbale shared an update Severity: ~~Severity 2~~ → Severity 1 **17:38:39** **Escalated to Infrastructure Leadership** Your workflow manually escalated the incident to the escalation path [Infrastructure Leadership](https://app.incident.io/gitlab/on-call/escalation-paths/01K7QAHP7P28668C5MHTHKJM34) **17:38:59** **Message from Cameron McFarland** Cameron McFarland pinned their own message We changed from chart 10.0.1 to 10.1.0 around 16:03 UTC for GPRD. **17:39:07** **Escalation acknowledged by Steve Abrams** Steve Abrams acknowledged the escalation **17:51:00** **Identified at** Custom timestamp "Identified at" occurred **17:56:33** **Message from Dave Smith** Dave Smith pinned their own message Question for delivery to answer in review - why were there so many days between chart bump MRs? **17:59:10** **Message from Dave Smith** Dave Smith pinned their own message We have found an issue with the latest bump done to the GitLab chart which affect how gitlab-shell interacts with the topology service. That chart bump is being reverted, but we are also pursuing a quicker edit of the config map to see if we can remedy sooner **18:04:45** **Message from Dave Smith** Dave Smith pinned their own message another question for post incident review - why was topology service enabled? **18:11:36** **Image posted by David Leach** David Leach posted an image to the channel Can see the requests to topology service beginning to come down source (https://dashboards.gitlab.net/goto/afq0yndo8ej28e?orgId=1)  **18:20:02** **Image posted by David Leach** David Leach posted an image to the channel Also highlighting that toplogy service metrics show an error rate: https://dashboards.gitlab.net/goto/afq0z5g4upn9ca?orgId=1 but no detail about what kind of error  **18:23:00** **Fixed at** Custom timestamp "Fixed at" occurred **18:23:08** **Message from Igor Wiedler** Igor Wiedler's message was pinned by Dave Smith we are recovered **18:28:13** **Status changed from Investigating → Monitoring** Raynard Omongbale shared an update Status: ~~Investigating~~ → Monitoring **18:28:13** **Monitoring at** Custom timestamp "Monitoring at" occurred **18:35:30** **Message from Jenny Kim** Jenny Kim's message was pinned by Cameron McFarland There was a set of smoke test that ran recently, about an hour ago for gstg https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/24898210 The `push_over_ssh_spec` _passed_ then. **19:01:06** **Message from Aaron Richter** Aaron Richter's message was pinned by Cameron McFarland I think three distinct things happened, and only one of them is a bug: 1. Topology Service returned NotFound: claim not found. This is correct, expected behavior. gitlab.com (http://gitlab.com) is a single-cell deployment; the TS has no claim registered for these routes. Returning NotFound is the right answer to "which cell owns this route?" when the answer is "this isn't a routed claim." Not a bug. 2. gitlab-shell caught that and decided to fall back to the default host. Also correct by design — the resolver's whole contract (from resolver.go) is "if TS is disabled, errors, or returns non-PROXY, return empty string and the caller uses the default host." The intent is sound. Not a bug in intent. 3. The fallback misrouted the request. Instead of re-issuing the original POST to the internal API host (gitlab-webservice-internal-api.gitlab.svc:8181), the fallback sent it to the public http://gitlab.com host, which 301-redirected http→https, and the HTTP client followed the 301 by converting POST → GET. Rails has no GET route for /internal/allowed → 404 → SSH auth fails. This is the bug. **19:11:46** **Update shared** Cheryl Li (via @incident) shared an update We’ve identified the issue as a gitlab-shell configuration regression that routed Git authorization checks incorrectly (resulting in a surge of `/api/v4/internal/allowed` 404s and SSH auth failures). Mitigation is in place (revert + config changes applied) and SSH Git operations are recovering; we’re in **Monitoring** to confirm error rates and request volume remain back to baseline. Next: • Continue monitoring SSH error ratio and internal API `/internal/allowed` 4xx rates for the next 30–60 minutes. • Complete the remaining revert work in **pre** and **staging** to prevent repeat impact. • Capture follow-ups to tighten gating so staging smoke test failures block promotion of Helm/config changes. **19:47:19** **Message from Jenny Kim** Jenny Kim's message was pinned by Cheryl Li I pieced together what happened with the deploy and smoke tests situation and put together a timeline. tl;dr smoke tests worked as intended, just extremely late, as the initial failed pipeline went unnoticed https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22163#note_3485644235 **19:57:04** **Incident resolved and entered the post-incident flow** Cheryl Li shared an update Status: ~~Monitoring~~ → Documenting We’ve continued to see SSH Git operations succeed after the mitigation/revert. Post-incident, we'll continue to monitor: • `gitlab_sshd` error ratio • Internal API auth endpoints (especially `/api/v4/internal/allowed`) 4xx rate and request volume # Investigation Notes </details> Any details you may want to add about the investigation can go here. # Lessons Learned ### General lessons _What did we learn from this incident that we want to carry forward?_ 1. [Good explanation](https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/22163#note_3485644235) from Jenny for how the workflow for helmfile chart bumps currently works with commentary on what we can improve. There is definitely room to improve linking and blocking MRs that bump helm charts. 1. Helm chart features that depend on existing config values can be silently activated by a chart bump — even when the intent of the change is unrelated. New chart features should use explicit opt-in flags rather than inferring enablement from pre-existing settings ("double-latch defaults"). Reviewers and authors need to be aware of what global config values are already set in each environment. 2. Changes that alter traffic routing patterns (e.g. enabling a new service as an intermediary for auth requests) should go through a Change Request to raise visibility, require staging validation before production, and ensure the right people are aware of the rollout. 3. Smoke tests worked as intended but ran too late and the initial failed pipeline went unnoticed — the linkage between chart bump MRs, their CI results, and smoke test outcomes needs to be more visible and blocking. 4. The incident was identified via user reports, not internal alerting. Any change that routes traffic through a new code path should be accompanied by alerting coverage for that path before it reaches production. ### What can we do to reduce detection time by half? 1. **Add SSH error-ratio alerting:** The existing gitlab-shell metrics did not capture this failure. [gitlab-shell!1501](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/1501) and [gitlab-shell!1499](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/1499) introduce a connection-level SSH reliability SLI and count internal API failures toward it — this should cause the git SLO alert to fire for this class of failure in future. 2. **Fix Topology Service error ratio metric:** The TS error ratio was counting expected `NotFound` responses as errors, masking the signal. Fixed in [runbooks!10888](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/10888). 3. **Include SSH traffic in SLA calculations:** This incident affected a covered experience but did not surface in SLA data because SLA is derived from Cloudflare HTTP logs only. [gitlab-com/gl-infra/observability/team#4617](https://gitlab.com/gitlab-com/gl-infra/observability/team/-/work_items/4617) tracks closing this gap. 4. **Gate production Helm rollouts on staging smoke test success:** The smoke test for `push_over_ssh_spec` passed on staging but ran too late to block the production deployment. [production-engineering#29257](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29257) tracks making this gate blocking. ### What can we do to reduce resolution time by half? 1. **Runbook coverage for emergency ConfigMap edits:** SREs hand-edited the gitlab-shell ConfigMap to restore service while the chart revert was in progress. This was effective but not all SREs would have known how to do it. The new `glsh kube configmap` tool ([runbooks!10890](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/10890)) makes this step documented and accessible. 2. **Faster chart deployment cycle:** The gap between chart bump MRs (days to weeks) meant the problematic change was harder to identify and the revert took longer to propagate. [production-engineering#29258](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29258) tracks reducing this to hours. 3. **Better linkage between chart bumps and their constituent changes:** During the incident it was difficult to quickly identify what had changed. Improved MR linking and review guardrails ([common-template-copier!761](https://gitlab.com/gitlab-com/gl-infra/common-template-copier/-/merge_requests/761), [k8s-workloads/gitlab-com!5526](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/5526)) will help reviewers and responders understand the scope of a bump at a glance. 4. **Ensure knowledgeable engineers are aware when their changes roll out:** The team that introduced the TS routing logic was not aware the chart bump would activate it in production. Improving the notification and review chain for chart bumps would allow the right people to be on standby or to flag concerns before rollout. # Follow-ups **Follow-up** **Owner** [Add MR Review Instructions to k8-workloads and helm charts to discourage the use of double-latch defaults](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29259) Andrew Newdigate [Reduce the time to deployment of charts from days or weeks to hours](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29258) Andrew Newdigate [CRITICAL P1/S1 Gate the production rollout of Helm deployments on the success of Staging Smoke Tests](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29257) Dave Smith # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [ ] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] For all S1 and S2 incidents, begin the [Feature Change Lock (FCL) process](https://handbook.gitlab.com/handbook/engineering/#feature-change-locks) and [open an issue in the FCL project](https://gitlab.com/gitlab-com/feature-change-locks/-/issues/new?description_template=feature-change-lock). - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [ ] S1 incidents [require a public RCA within 7 days](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/#timeline-that-we-expect-for-reviews-to-be-completed) of the incident. If this review cannot be made public, [create a separate public RCA](https://handbook.gitlab.com/handbook/engineering/root-cause-analysis/#how-to-perform-an-rca). - [x] Close the review before the due date - [x] Go back to the incident channel or page and close out the remaining post-incident tasks