## Bug Description Every request to the Artifact Registry's Runway endpoints is refused at the Cloudflare edge with a `403` WAF block page ("Sorry, you have been blocked") instead of reaching the service. Both the staging host (`staging.artifact-registry-gke.svc.gitlab.net`) and the production host (`artifact-registry-gke.svc.gitlab.net`) are affected, on every path probed, so the deployed service is unreachable from the reporting vantage point and no request is observable in AR's access log. ## Steps to Reproduce 1. From a machine with plain internet access and no credentials, request the staging root: ```shell curl -i "https://staging.artifact-registry-gke.svc.gitlab.net" ``` 2. Repeat against the production host: ```shell curl -i "https://artifact-registry-gke.svc.gitlab.net" ``` 3. Repeat against any path — the block is not path-scoped: ```shell for p in "/" "/-/health" "/-/readiness" "/v2/" "/api/v1/organizations"; do curl -s -o /dev/null -w "$p -> %{http_code}\n" \ "https://staging.artifact-registry-gke.svc.gitlab.net$p" done ``` No authentication is involved in any of the above; these are anonymous GETs. Reproducibility: always Reproduced independently: yes, on staging and production from a second machine (see Actual Behavior), at 2026-09-03 05:21 UTC. A second engineer (@rchanila) is also blocked, from Sydney and not via the `MEL` edge — reported, not verified by me. ## Expected Behavior The endpoints answer from the Artifact Registry, not from the edge. - [`docs/dev/gitlab-com-infrastructure.md`](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/dev/gitlab-com-infrastructure.md) states that "the staging endpoint is public" and that "a steady stream of `404` responses in the access logs is normal background noise" from internet scanners. A `404` in AR's own access log is the documented behavior for an unrouted path; a `403` HTML page from Cloudflare, with no AR access-log row at all, is not. - [ADR-024](https://gitlab.com/gitlab-org/ops/artifact-registry/-/blob/main/docs/adr/024_infrastructure_delivery.md) places Cloudflare in front of AR as the DDoS/WAF/rate-limiting edge and routes GitLab Rails to the AR REST API *through* that edge. A WAF rule that refuses unauthenticated GETs to `/` refuses them for every client behind it. - Issue [#1132](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1132), filed 2026-09-02 22:47 UTC, reproduces requests that **did** traverse the staging edge and reach AR (`302` responses, `cf-cache-status: MISS`, and AR access-log rows). That is roughly six hours before the first observation here, so the endpoint was serving through Cloudflare very recently. ## Actual Behavior Every request returns `403` with a Cloudflare-generated block page. The response carries `server: cloudflare` and **no `x-request-id`**, which is how it is known the request never reached AR. Reporter's original observation: ``` $ curl -i "https://staging.artifact-registry-gke.svc.gitlab.net" HTTP/2 403 date: Thu, 03 Sep 2026 05:00:47 GMT content-type: text/html; charset=UTF-8 cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 expires: Thu, 01 Jan 1970 00:00:01 GMT referrer-policy: same-origin x-frame-options: SAMEORIGIN server: cloudflare cf-ray: a352315a5f68d853-MEL ``` Independent reproduction, 2026-09-03 05:21:55 UTC, `cf-ray: a35250543ce0f0cd-MEL`. The body is the Cloudflare interstitial: ``` Sorry, you have been blocked You are unable to access ... Why have I been blocked? This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data. ``` Matrix of what was probed, all at 2026-09-03 05:22 UTC: | Target | User agent | Status | |---|---|---| | staging `/` | curl default | 403 | | staging `/` | Chrome 128 UA string | 403 | | production `/` | curl default | 403 | | production `/` | Chrome 128 UA string | 403 | | staging `/-/health` | curl default | 403 | | staging `/-/readiness` | curl default | 403 | | staging `/v2/` | curl default | 403 | | staging `/api/v1/organizations` | curl default | 403 | So the block is not path-scoped, not user-agent-scoped, and not staging-only. Two people are affected from two different locations. The probes tabulated above all egress from one network and hit the Melbourne (`MEL`) edge; separately, @rchanila reports the same block from Sydney, reaching a different Cloudflare edge. That second report is the reporter's, not independently verified here, and it rules out a rule scoped to a single source IP or a single edge PoP. Correlation ID: none exists. Cloudflare answered before AR did, so there is no AR-side request ID to trace; the `cf-ray` values above are the only handles. ## Suggested Fix **This section is a hypothesis, not a finding.** The response is a Cloudflare WAF *block* action, not a managed challenge (no `cf-mitigated` header, no JS challenge, no interstitial retry) and not Cloudflare Access (an Access-protected host redirects to a login page rather than returning a block page). That points at a security rule on the Runway-provisioned Artifact Registry zone that ADR-024 names, rather than at anything in AR's Go code — consistent with the fact that no AR build changed between #1132's successful edge traversal and this block. What I would check first, in order: 1. The recent change history on the Cloudflare zone for `*.artifact-registry-gke.svc.gitlab.net` — a WAF managed-ruleset version bump, a new custom rule, or an IP/ASN reputation rule deployed in the last ~6 hours. 2. Whether the rule is scoped by geography or ASN. A single-IP or single-PoP rule is already ruled out by the two affected people above, but both are in Australia, so a country- or ASN-scoped rule fits the evidence just as well as a zone-wide one. **This is the open question that decides the severity** — see below. A probe from outside Australia settles it in one request. 3. Cloudflare's Security Events / firewall log for the `cf-ray` values above, which names the rule that fired. The two sibling edge defects, [#1132](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1132) (HEAD rewritten to GET, dot-dot paths normalized) and [#1131](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1131) (edge caches 404s), are different defects on the same edge and are not duplicates of this one. They are worth reading together as evidence that the edge configuration in front of AR is not currently well understood by the team. ## Suggested DRI **Workstream:** infrastructure / the Cloudflare edge, not an AR format or service stream. The fix lands in the Runway-provisioned Cloudflare zone configuration that ADR-024 describes, not in AR's Go code — which is why this is `artifact-registry::infrastructure` and not `artifact-registry::server`, even though the symptom is that the server does not answer. The workstream assignments table in `docs/roadmap/closed-beta.md` carries **no row for infrastructure or the edge**; the closest owners it names are the "Infra readiness" readiness gate (owner: Infra) and the "Runway (deployment platform)" external workstream (owner: Infra). Sibling edge issue #1132 is likewise unassigned. So triage should route this to Infra/Runway rather than to an AR stream DRI. This is a suggestion for triage, not an assignment. The issue is filed unassigned. ## Labels to Apply - `type::bug` - `severity::2` — a major feature is broken with no workaround: the deployed service answers nothing. Not `severity::1`, because a complete-outage claim needs the block to be global, and every affected person so far is in Australia — a country- or ASN-scoped rule fits the same evidence. **If Infra confirms the rule is not geo- or ASN-scoped, this is `severity::1`**: production is then wholly unreachable, including the Rails-to-AR path in ADR-024. Filed at S2 per the rubric's instruction to file the lower level when between S1 and S2 and say why, because S1 halts the beta and triage escalates faster than the beta un-halts. Not `severity::3`, because no workaround is known and nothing about the endpoint currently works. - `Category:Artifact Registry` - `artifact-registry::infrastructure` - `bug::availability` - `regression` — #1132's reproduction traversed this edge to AR about six hours earlier - `AR-Blocks::GA` — maps mechanically from `severity::2` - No `frontend` / `backend`: the fix lands in edge configuration, which is neither. Sibling edge issue #1132 carries neither for the same reason. - No `format::*` / `repo-type::*`: the block is format-independent and was reproduced on paths belonging to no format. ## Additional Context | | | |---|---| | Environment | Runway staging **and** production (`gitlab-runway-staging`, `gitlab-runway-production`) | | Org slug | n/a — no request reached the service | | Repository | n/a | | Format | n/a — reproduced on `/`, `/-/health`, `/-/readiness`, `/v2/`, `/api/v1/organizations` | | Auth method | none; all probes were anonymous GETs | | Client and version | `curl` (system default UA), and a Chrome 128 UA string | | First observed (UTC) | 2026-09-03 05:00:47 | | Correlation ID | none — Cloudflare answered before AR; `cf-ray: a352315a5f68d853-MEL` and `a35250543ce0f0cd-MEL` | | Workaround | none known; not tried | **What was ruled out:** path scoping, user-agent scoping, and staging-only scoping (see the matrix above). **What was not checked, and matters most:** whether the block is global or scoped by geography/ASN. Two people are blocked from two Australian cities through two different Cloudflare edges, which rules out a single-IP and a single-PoP rule but not a country- or ASN-scoped one. A probe from outside Australia, or Cloudflare's Security Events view for the `cf-ray` values, settles it — and settles whether this is S1 or S2. **Known change history:** none that the reporter is aware of; no AR release is implicated, since #1132's successful edge traversal and this block bracket a window with no intervening AR deploy known to the reporter. **Related, not duplicates:** [#1132](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1132) and [#1131](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1131), both open defects in the same Cloudflare edge in front of AR.