# Summary Assess and track the impact of the hosted runner multi-region fallback on customers who use group IP address restrictions, and record how to confirm it and what to tell them. ## What is happening Hosted Linux runner VMs used to run only in Google Cloud `us-east1`. The fleet now falls back to `us-east5` when `us-east1` is short of capacity, with `us-east4` and `us-central1` also configured. Customers with **Restrict access by IP address** commonly allowlist the published `us-east1` ranges, because that is the region our docs name. Their container registry pulls now fail whenever a job lands in a fallback region. Retries often land back in `us-east1` and succeed, so it looks intermittent. Why the registry and not the clone: - `git clone` and artifacts reach Rails through the internal `ci-gateway` as a `10.x` address, which the instance-wide `globally_allowed_ips` of `10.0.0.0/8` covers. This is the documented [hosted runner exemption](https://docs.gitlab.com/user/group/access_and_permissions/#gitlabcom-access-restrictions). - The registry token request goes to `gitlab.com/jwt/auth` over the public internet and arrives with the VM's public Google Cloud address, which `10.0.0.0/8` cannot match. So hosted-runner registry pulls have always been subject to group IP restrictions. Nothing regressed. Only the addresses changed. Git LFS fails the same way, tracked in [gitlab-org/gitlab#634546](https://gitlab.com/gitlab-org/gitlab/-/issues/634546). ### Related Issues/MRs/Epics - Announcement: [multi-region fallback, Slack, 2026-10-20](https://gitlab.slack.com/archives/CJWA4E9UG/p1792490580659029) - Issue/epic: [production-engineering#30082](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30082) - Related infra: [#30087](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30087) (global allowlist), [#30129](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/30129) (VM egress), [#29616](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29616), [incident-15507](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23311)/[Slack](https://gitlab.enterprise.slack.com/archives/C0CAY5TF7JR) - Same failure shape: [gitlab-org/gitlab#634546](https://gitlab.com/gitlab-org/gitlab/-/issues/634546) (Git LFS) - Docs: [IP range](https://docs.gitlab.com/user/gitlab_com/#ip-range), [group access and permissions](https://docs.gitlab.com/user/group/access_and_permissions/#restrict-group-access-by-ip-address) - Runbook: [CI runner networking](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/ci-runner-networking.md) - KB: [Hosted runner jobs fail when pushing to registry](https://support.gitlab.com/hc/en-us/articles/26862112835100-Hosted-runner-jobs-fail-when-pushing-to-registry-access-forbidden) ### Status / What actions have been taken so far Root cause confirmed on one ticket, by correlating a denied pull against the IP restriction enforcer log and resolving the client address to `us-east5`. Other tickets share the symptom and are pending the check below. The announcement did not call out the effect on IP-restricted groups, which is why early tickets were triaged without that context. No infrastructure change is being requested. The fallback is deliberate capacity work. ### Timeline / Important Dates - ~2026-10-15: `us-east5` subnets and fleet config land for the `us-east1` amd64 shards. - 2026-10-20: [announced in Slack](https://gitlab.slack.com/archives/CJWA4E9UG/p1792490580659029). - From 2026-10-21: tickets begin arriving. ## What impact will this have on users? Customers who use hosted runners and restrict group access by IP, where the allowlist enumerates Google Cloud regions. Jobs fail on image pulls in proportion to how often the fleet spills out of `us-east1`, which rises during weekday capacity shortages. Customers cannot fix this in their pipeline. The outcome depends on which region their VM was placed in, which they neither choose nor can see. ### What this may look like for Support Anticipated Support Impact: moderate. Small affected population, since it needs IP restrictions plus hosted runners, but highly visible to each customer because it blocks pipelines and looks random. Expect Severity 2 and 3 tickets describing CI as unreliable. What errors or messages users may report: ``` ERROR: Job failed: failed to pull image "registry.gitlab.com/<group>/<project>/<image>:<tag>" with specified policies [always]: Error response from daemon: pull access denied for registry.gitlab.com/<group>/<project>/<image>, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (manager.go:469:0s) ``` Customers report that `git clone` works, that retries sometimes succeed, and that nothing changed on their side. All three are accurate. **The trap.** The same error, and the same `Denied container registry permissions` log line with empty `authorized_actions`, is also produced by the CI/CD job token allowlist or an insufficient role on a cross-project pull. The only discriminator is whether a paired enforcer line exists. Confirm before replying. While reproducing this in a test group, the job token cause came up instead, from an address that was correctly allowlisted. **How to confirm.** In [Kibana](https://log.gprd.gitlab.net), `pubsub-rails-inf-gprd*`, scoped to the failure time: ``` json.subcomponent.keyword: "auth_json" AND json.message: "Denied container registry permissions" AND json.requested_project_path: "<group>/<project>" ``` Take `json.correlation_id` and search it on its own: - An `Attempting to access IP restricted group` line with `allowed: false` confirms the IP cause. Resolve `json.ip_address` against [cloud.json](https://www.gstatic.com/ipranges/cloud.json). A Google Cloud address outside `us-east1` is the confirmation. - No such line, or `allowed: true`, means the job token or role cause. Check **Settings > CI/CD > Job token permissions** on the target project, and the user's role there. Note that `json.status` is **200** on a refused pull, not 403, because `jwt/auth` issues a token with an empty `access` claim and the registry then refuses it. Two field gotchas: - `kubernetes.*` describes the webservice pod that served the request, always in `us-east1`. It is not the runner's region. The client is `json.meta.remote_ip`. - `json.group_full_path` is the group being accessed, possibly a subgroup. The restriction comes from the root ancestor, so query with a wildcard. What workarounds/solutions are available?: - **Add the fallback ranges.** IPv4 and IPv6 prefixes for `us-east5` from [cloud.json](https://www.gstatic.com/ipranges/cloud.json), filtered on `scope`. Fixes today, leaves them exposed when the fleet next spills into `us-east4` or `us-central1`. All three is around 228 prefixes. - **Use their own runners.** A group or project runner with a fixed egress address, allowlisting only that address. The only option that stops their allowlist tracking our infrastructure, and what the KB recommends. State the trade-off plainly: allowlisting cloud regions admits any tenant in those regions. Our only GitLab-exclusive ranges are `34.74.90.64/28` and `34.74.226.0/24`, which are the Web/API fleet, not runners. The settings page takes one entry at a time ([bulk entry proposed](https://gitlab.com/gitlab-org/gitlab/-/issues/468998)). The [Groups API](https://docs.gitlab.com/api/groups/) takes `ip_restriction_ranges` as one comma-separated string, but **replaces** rather than appends, so have them read the current value first. Point them at the live `cloud.json`, not a snapshot, since `us-east1` has gained prefixes recently. ### Do users need to be contacted? - [x] No - [ ] Yes, please link to the [marketing ops issue](https://handbook.gitlab.com/handbook/support/workflows/sending_notices/#mass-emails-through-marketing-department): Needs a decision. The affected population is identifiable, and proactive notice would likely cost less than the tickets, but it needs Runners Platform input on whether the fallback regions are stable enough to publish. ## DRIs/Contacts for questions and approvals for communications/action items - Slack Channel: `#support_gitlab-com`, `#g_runners_platform` - Product or Development DRI: Runners Platform - Security DRI (if applicable): n/a - Support DRI: - Support Manager DRI (if needed): ## Proposed follow-up - [ ] Open a `gitlab-org/gitlab` issue for the registry equivalent of [#634546](https://gitlab.com/gitlab-org/gitlab/-/issues/634546). None exists today, so there is nothing public to share with customers. - [ ] Docs MR: scope the hosted-runner exemption wording, and add the fallback regions to the [IP range page](https://docs.gitlab.com/user/gitlab_com/#ip-range), which still lists `us-east1` only. The exemption note already scopes out Git LFS, so the registry is a second undocumented exception of a documented class. - [ ] Ask Runners Platform to consider enabling `log_external_ip`, off by default, which prints the VM's external address into the job log. That would let affected customers diagnose this themselves. - [ ] Update the KB article with the regional-fallback detail and the confirmation steps above. ## Communication to Support team 1. Announced to team in 1. [x] `#support_gitlab-com` 1. [x] Support Week in Review [(SWIR)](https://gitlab.com/gitlab-com/support/readiness/support-week-in-review/-/issues/?sort=created_date&state=all&first_page_size=10), in `Things to know about` If you have a matching ticket, comment with what the enforcer check showed so we can size the spread.