Production Engineering: Networking (#1676) · Epics · GitLab Infrastructure Team

Production Engineering: Networking

## Additional Networking happenings None of note this week. ## Overview  This epic is a the high level epic for all project work across the Networking and Traffic Management categories. ## Hours Worked | Epic | Category | DRI | Total Hours | |------|----------|-----|-------------| | [Networking - incoming requests work - Q2 FY27](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2031) | ~"infra-category::KTLO" | @sabrams | 0.0 | | [Networking - Keep the Lights on work - Q2 FY27](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2030) | ~"infra-category::KTLO" | @sabrams | 0.0 | | [Simplify Rate Limiting Configuration](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1534) | ~"infra-category::Tech Debt" | @swiskow | 0.0 | | [Network Cost Efficiency & Savings](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2066) | ~"infra-category::KTLO" | @sabrams | 0.0 | ## Project Work ### :hourglass: Work In Progress These epics are currently in progress | **Topic** | **Start Date** | **Target End Date** | **Summary** | |-----------|----------------|---------------------|-------------| | [Networking - incoming requests work - Q2 FY27](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2031) @sabrams (+0 participants) ~"group::Networking & Incident Management" | 2026-05-01 | 2026-07-31 | **2026-05-27**: :clock1: **total hours spent this week by all contributors**: 1 To be triaged: [Key customer rate limit request](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29087) _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2031#note_3387739037_  | | [Networking - Keep the Lights on work - Q2 FY27](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2030) @sabrams (+0 participants) ~"group::Networking & Incident Management" | 2026-05-01 | 2026-07-31 | **2026-05-27**: 🕐 total hours spent this week by all contributors: 20 :tada: **achievements**: - [We have successfully updated the machine type for HAProxy Pages VMs in Staging](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22190) , the next step would be to do the same for Production, we are currently in a PCL that ends on June 2nd so the change will be scheduled following that. This will result in [six figure annual cost savings](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28084). :issue-blocked: **blockers**: * (copied from last week) A variety of PCI issues have come in (https://gitlab.com/gitlab-com/gl-security/security-assurance/security-compliance-commercial-and-dedicated/observation-management/-/work_items/1554#detailed-remediation-plan-or-linked-epicissue---dri-remediation-manager). We've started to look through them but have no capacity to proceed at this time (due dates were generally May 14). Please provide some guidance if we should deprioritize other Code Yellow work in favor of the PCI issues. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2030#note_3387739077_  | | [Simplify Rate Limiting Configuration](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1534) @swiskow (+0 participants) ~"group::Networking & Incident Management" | 2026-05-04 | 2026-10-31 | **2026-07-01**: The rate-limiting migration's final track — moving the request-throttling layer onto the new framework — landed this week ([gitlab!239466](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239466) merged 2026-07-02), and the first attempt to switch it on ran in shadow (observe-only) mode on staging. A latent bug made the new middleware error on nearly every request, but its fail-open design meant no request was ever blocked or slowed and the code never ran in production; the flags were switched back off, the fix merged the same day ([gitlab!243638](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/243638)), and the rollout will retry once it deploys. No customer impact, no production behaviour change, and no asks for leadership. :clock1: **total hours spent this week by all contributors**: 34 **Outcomes this week** - **Stage 2b middleware merged — the last migration track is now in the codebase.** [gitlab!239466](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239466) (mwoolf, "Add Labkit::RateLimit shadow + enforce middleware for Rack::Attack") merged 2026-07-02 with approvals from hmerscher + reprazent, closing out the multi-week reprazent design review. It ships the request-throttle migration as a shadow-then-enforce middleware gated by six per-cohort `wip` flags. **All flags default off, so it is dormant on merge** — deployed to `.com` with no behaviour change until a cohort is explicitly enabled. Tracks [Stage 2b #28852](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28852). - **Feature-flag removal is live on `.com`.** [gitlab!239802](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239802) (merged 2026-06-24) has ridden the deploy train through production, so every migrated ApplicationRateLimiter cohort is now permanently on the new framework with the temporary rollout flags gone. [#28876](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28876) stays open only until the final close-outs below land. **In flight** - **[Stage 2b #28852](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28852) — shadow rollout retry.** The first shadow enablement was attempted on staging on 2026-07-03 and immediately surfaced a nil-rule crash in the middleware (details under Risks). It was contained to staging, the fix [gitlab!243638](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/243638) (mwoolf, approved reprazent) merged the same day, and the plan is to re-enable shadow per cohort once it deploys, confirm the divergence metric emits, and hold under the go/no-go threshold for 24h before any enforce. Tracked on [gitlab#602804](https://gitlab.com/gitlab-org/gitlab/-/issues/602804). - **Config-from-registry [#29054](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29054)** progressed: mwoolf's [gitlab!240022](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240022) ("Resolve rate limit threshold/interval from labkit registry") came **out of Draft** and is ready for review; hmerscher's alternative [gitlab!242446](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242446) remains Draft. - **Stage 2b design follow-ups** spun out of the merged middleware and are in triage: [#28853](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28853) (config evolution — callables/precedence/static), [#29052](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29052) (refine rule action semantics: limit/log/skip), [#29319](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29319) (deprecate `GITLAB_THROTTLE_DRY_RUN` in favour of the `log` action), [#29320](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29320) (model the RackAttack user allowlist as an explicit skip rule). - **[labkit-ruby!315](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/merge_requests/315)** (mwoolf, Lua-script extraction) — no movement again this week; still open, still waiting on a reviewer. **Risks & decisions** - **Contained incident (staging only, no user impact):** the first Stage 2b shadow enablement raised a `NoMethodError` on nearly every request — the middleware read `result.rule.name` on fail-open/unmatched results, which carry a nil rule. The middleware's fail-open guard meant `Rack::Attack` stayed the sole enforcer throughout, so **no request was blocked or delayed and the flags were never enabled in production**. The visible symptoms were an absent divergence metric and log/Sentry volume, which is what prompted disabling the flags. Root cause and fix in [gitlab!243638](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/243638); post-mortem on [gitlab#602804](https://gitlab.com/gitlab-org/gitlab/-/issues/602804). - **Lesson / action items from the post-mortem:** the unit specs used a fake result that always carried a rule, so the real SDK's nil-rule path was never exercised — the primary gap. Follow-ups: test the middleware against the real SDK result contract; add a canary that alerts when the middleware records block calls but the divergence metric is empty (a blind gate); surface the guarded-exception rate as a metric; and verify an enforced block returns a real 429 before any enforce flag is turned on. - **Risk (carry-over):** the [regex-timeout fail-open #28882](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28882) still has no movement since 2026-05-08 and continues to gate a safe Stage 2b *enforce*. **Asks** - None this week. The Stage 2b retry and config-from-registry work are team-internal and not blocked on leadership. **Upcoming reviews / approval needs** - [gitlab!240022](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240022) — config-from-registry, now out of Draft (mwoolf); needs reviewers. - [labkit-ruby!315](https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/merge_requests/315) — Lua-script extraction (mwoolf); stalled ~3 weeks, needs a reviewer. **Next week** - Deploy the nil-rule fix ([gitlab!243638](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/243638)) and retry the Stage 2b shadow rollout per cohort on staging; confirm the divergence metric emits and sits under the go/no-go threshold for 24h before any enforce. - Land the post-mortem action items (real-SDK-contract tests and the blind-gate canary) so the retry is observable. - Close out the migration: confirm [#28876](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28876) and close the superseded [gitlab!236795](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/236795) and the dead cohort-5 adapter [gitlab!236515](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/236515) — both still open, carried from last week. - Progress config-from-registry ([gitlab!240022](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240022) / [gitlab!242446](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242446) / [#29054](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29054)). - Resolve the regex-timeout risk ([#28882](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28882)) that gates enforcement. **Phase 2 stage status — start vs end of week** (start = state at the prior roll-up, 2026-06-26) | Stage | Issue | Start of week | End of week | |-------|-------|---------------|-------------| | ApplicationRateLimiter → Labkit (full migration) | [#28808](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28808) | 🔄 In verification — enforcing on `.com`; FF-removal merged and deploying | 🔄 In verification — FF-removal live on `.com`; all cohorts permanently always-on | | Feature-flag cleanup | [#28876](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28876) | 🔄 [gitlab!239802](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239802) merged 2026-06-24 (deploying; `.com` pending) | ✅ Code live on `.com`; issue open only pending close of superseded [gitlab!236795](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/236795) | | Stage 2b — RackAttack migration | [#28852](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28852) | 🔄 In review — reframed to explicit ordered rules; [gitlab!239466](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239466) 1 approval | 🔄 Middleware [gitlab!239466](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239466) merged 2026-07-02 (2 approvals), deployed dormant; shadow rollout attempted on staging → nil-rule crash (no prod/user impact), fix [gitlab!243638](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/243638) merged same day; retry pending deploy | | Config from registry / static Limiters | [#29054](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29054) | 🔄 In flight — hmerscher assigned; 2nd MR [gitlab!242446](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242446) (draft) | 🔄 In flight — [gitlab!240022](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240022) out of Draft (mwoolf); [gitlab!242446](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242446) still Draft (hmerscher) | _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1534#note_3510373104_  **Nested Epics: 2** • https://gitlab.com/groups/gitlab-org/-/epics/22423+ • https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2021+ | | [Network Cost Efficiency & Savings](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2066) @sabrams (+0 participants) ~"group::Networking & Incident Management" | | | | ### :soon: Ready Linked epics that are ready to start | **Topic** | |-----------| | [Migrate to Envoy Gateway for GitLab.com](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1641) ~"group::Networking & Incident Management" | | [[Phase 1] Multi-platform Gateway: Envoy Gateway Deployed in Non-Production Environments](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1644) ~"group::Networking & Incident Management" | | [Consolidate DNS management at GitLab](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1473) ~"group::Networking & Incident Management" | ### :arrow_forward: Next These are the epics we will be focusing on next | **Topic** | **Target Start Date** | **Summary** | |-----------|-----------------------|-------------| | [Update Application Limits to match published rate limits](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1771) (+0 participants) ~"group::Networking & Incident Management" | | | | [IPv6 Support Implementation for GitLab.com Services](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1656) (+0 participants) ~"group::Networking & Incident Management" | | | | [Understand and monitor Networking Costs for Cells Data Transfer](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1668) (+1 participants) ~"group::Networking & Incident Management" | | | ### :stop_button: Triage Epics that are currently being triaged | **Topic** | **Summary** | |-----------|-------------| | [Secure Cloudflare Token Management](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1761) ~"group::Networking & Incident Management" | **2026-04-08**: There is no update this week. This epic has been paused to prioritise [Code Yellow - Rate Limiting and DDoS FY26Q2](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/2008). _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1761#note_3229420869_  | | [Lighten Change Request Process](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2053) ~"group::Networking & Incident Management" | | | [Further Standardize Cloudflare Usage across GitLab](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1669) ~"group::Networking & Incident Management" | | | [Enable teams to self-serve DNS configuration](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1634) ~"group::Networking & Incident Management" | | | [Autoscaling HAProxy](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1596) ~"group::Networking & Incident Management" | | | [Paved paths for enabling rate and application limits](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1586) ~"group::Networking & Incident Management" | | | [Rate limiting - Phase 2: Simplify application level configuration](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1503) ~"group::Networking & Incident Management" | **2025-01-10**: Epic created  | | [CloudFlare exporter improvements](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1456) ~"group::Networking & Incident Management" | | | [Fine-grained webservice deployments for isolation and attribution](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1417) ~"group::Networking & Incident Management" | | ### :octagonal_sign: Blocked Epics that are currently blocked | **Topic** | **Start Date** | **Target End Date** | **Summary** | |-----------|----------------|---------------------|-------------| | [Remove remaining NGINX config from GitLab.com](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2047) @haseeb_ahmed @thisisshreya (+0 participants) ~"group::Networking & Incident Management" | 2026-02-02 | 2026-05-31 | **2026-06-24**: :issue-blocked: `**blockers**:` * This item is blocked as of now, pending follow-ups to protect against any incident recurrence ([comment](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/2047#note_3451290785) for more details) _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2047#note_3486276666_  | ### :white_check_mark: Completed Work Items that have been completed <details> | **Topic** | **Started** | **Ended** | **Summary** | |-----------| ------------| ----------| ------------ | | [Keep The Lights On - Production Engineering Ops](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1283) ~"group::Networking & Incident Management" | 2024-04-03 | 2025-09-02 | **2025-08-27**: No updates this week. This epic will be deprecated and combined with the [Foundations KTLO epic](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1531) following [recent team changes](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1672). | | [Networking & Incident Management - "Keep the Lights on" work - FY26](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1531) ~"group::Networking & Incident Management" | 2025-01-01 | 2026-01-30 | **2026-01-27**: :clock1: **total hours spent this week by all contributors**: 16 - @sarahwalker spent some time as EOC coordinator ensuring transitioning the EOC shift to a [Friday start](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/28018) went smoothly - @ahanselka set up a new incident field to [track incidents related to DAP ](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28222) - @ahanselka is working on a [runbook](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/27940) for Cloudflare outages - @devin some IMOC coordination, including onboarding new members and working with a new Tier 2 rotation _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1531#note_3036061794_  | | [Networking & Incident Management incoming requests work - FY26](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1530) ~"group::Networking & Incident Management" | 2025-01-01 | 2026-01-30 | **2026-01-27**: :clock1: **total hours spent this week by all contributors**: 8 **Support/Customer Success** - @sabrams - https://gitlab.com/gitlab-com/request-for-help/-/work_items/3842 - Long running RFH working with a customer moving to Dedicated who is hitting rate limits. We've been monitoring their traffic for the last 2 months as they ramp up their operations and may need to now temporarily add them to the bypass list ([as previously agreed upon](https://gitlab.com/gitlab-com/request-for-help/-/issues/3842#note_2924992020)). We've requested more information to make a better forecast of traffic growth before adding them so we better understand the risk. - @donnaalexandra - https://gitlab.com/gitlab-com/request-for-help/-/work_items/4076 - Help with debugging a rate limit problem where it is not working in certain types of self-managed installations. This rate limit is related to a Groups API function, so we've left initial feedback but redirected to ~"group::organizations". **Customer Success** - @haseeb_ahmed - https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28170 - Adding new subdomains for Customer Success. **Verify** - @sarahwalker - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21134 - helping review a CR to enable gRPC. Unfortunately this is [more complicated than a simple toggle](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21134#note_3039603629), so we've redirected to a request for help issue for higher level of guidance to self-serve in the config-mgmt repo. ----- _On the radar / Triage_ - @sabrams - https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/27721 - Solutions architecture reaching out for further events help. We've deferred to Security since these limits relate directly to security exploitation angles. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1530#note_3036061846_  | | [Re-Enable Authenticated API and Web rate Limits to match published limits](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1541) ~"group::Networking & Incident Management" | 2025-01-07 | 2025-11-28 | **2025-11-25**: :sparkles: **THIS EPIC IS FINALLY READY TO BE CLOSED** :sparkles: :clock: **Total hours spent this week by all contributors:** 6-8 hours on retrospective creation, demo preparation, blocking excessive traffic, and wrapping up. :books: The retrospective of this project has been completed and [summarised here](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20866#note_2915046911). :playground_slide: [Demo slides are available here](https://docs.google.com/presentation/d/1MV8YWuNPcN9kQ4W7AKrQft0vTQwTYmb-0zAOJdHiufk/edit?slide=id.g1d24c3e4ddd_5_52#slide=id.g1d24c3e4ddd_5_52) with the intention to record a short video walkthrough when Donna recovers from whatever Flu/ Cold is impacting her voice right now. :sneezing_face: :tada: **Achievement Summary** :tada: **Rate limit Implementations** - [Authenticated API Rate Limits enabled in GitLab Application](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20828) - [Authenticated Web Rate Limits enabled in GitLab Application](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20641) - [API Rate Limits enabled in Cloudflare to provide edge layer protection](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20828#note_2902233452) - [Runner API rate limits split out from core Authenticated API rate limits](https://gitlab.com/gitlab-org/gitlab/-/issues/462537) **Additional Wins** - [Identified and plan in place to reduce excessive API requests from editor extensions ](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/issues/1671)(**327M+** requests/week) - [RackAttack rate limit headers now emitted from the Application](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/25770) (release available for all GitLab platforms) - [First Example of the Breaking Change Process](https://gitlab.com/gitlab-com/Product/-/issues/new?description_template=Breaking-Change-Exception) - [Alerts enabled to notify if rate limits are disabled again](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/26420) - [More bypasses identified to clean up](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20872#note_2909618057) Thank you everyone for the collaboration and patience. :bow: _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1541#note_2910994589_  | | [Create SME Tier 2 Pilot Program](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1600) ~"group::Networking & Incident Management" | 2025-04-09 | 2025-10-31 | **2025-10-28**: 🚀 **Closing status update!** 🚀 📋 **Problem:** GitLab's single-tier on-call structure hadn't scaled with growth from 54 to 93 services in two years. SREs increasingly needed subject matter expertise during incidents, impacting resolution times. 🛠️ **Solution:** Implemented four Tier 2 SME on-call rotations with established escalation paths, onboarding materials, and independently-managed schedules following standardized processes that make adding future rotations straightforward. 🎯 **Impact:** Tier 2 teams handled 52 incidents requiring specialized expertise: Gitaly (34 escalations/7 months), DBO (10 escalations/2 months), AI (2 escalations/1 month), DevOps Rails (5 escalations/2 weeks). This escalation rate validates the program's value in connecting incidents with the right domain experts quickly. 🙏 Shoutouts to the rotation leaders and all engineers participating in these on-call responsibilities! Special thanks to @igorwwwwwwwwwwwwwwwwwwww @kkyrala from the former Ops team for laying the groundwork. 🌟 Ready for closure at the next Grand Review. cc: @alex @sshah @swalker @sabrams and the NIM team 🎉 _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1600#note_2848573573_  | | [FY26 - Q2 - Incident Management Improvements](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1582) ~"group::Networking & Incident Management" | 2025-05-01 | 2025-08-29 | **2025-08-05**: This epic can be closed. We've completed all the work that was contained within and remaining incident.io improvements are going into our incident.io improvements backlog (https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1489/). _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1582#note_2668401501_  | | [Standardize Cloudflare usage across GitLab](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1561) ~"group::Networking & Incident Management" | 2025-05-07 | 2025-09-19 | **2025-09-16**: We've complete the remaining exit criteria for this project over the past week, safely applying changes across `gstg`, `ops` and `gprd`, whilst completing our initial documentation. See our [closing status update](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1561#note_2758792531) for more details. :tada: **achievements:** - :chart_with_upwards_trend: Migrate the WAF configurations for `gstg`, `ops` and `gprd` environments to the `entrypoint` module, completing the internal adoption. - :google_docs: Complete initial operator and user documentation - :checkered_flag: See [closing status update](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1561#note_2758792531) :tada: _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1561#note_2752674212_  | | [FY26 Q3 DR Gamedays](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1607) ~"group::Networking & Incident Management" | 2025-07-01 | 2025-10-24 | **2025-10-21**: **Closing Summary** :tada: We continue to ensure our ability to recover zonal outages of key production components through our quarterly gamedays. Results from this quarter are viewable in the [DR measurements runbooks page](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/disaster-recovery/recovery-measurements.md?ref_type=heads) and this epic is ready to be closed! Some highlights from this quarter: - The HAProxy zonal outage process time [improved by 8 minutes](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/disaster-recovery/recovery-measurements.md?ref_type=heads#gameday-zonal-outage-dr-process-time-2). - CI-Runners and Patroni zonal outage process times remained stable. - The [Gitaly gameday was skipped](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/26947#note_2839130832) due to team capacity disruptions right before the scheduled date which was too late in the quarter to reasonably reassign to another team. We verified this had no compliance impact. [Next quarter](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1752) we are going to assign the gamedays to the teams that own the services as we start to look at the [future of the DR gameday process](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1754). A special shout out to the team members who ran these gamedays: @jarv, @knottos, @stejacks-gitlab, @zbraddock, and @rehab! _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1607#note_2833114074_  | | [Handover of Teleport to InfraSec team](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1681) ~"group::Networking & Incident Management" | 2025-08-18 | 2025-11-07 | **2025-11-04**: :sparkles: **Closing status** :sparkles: **Problem** :telescope: Teleport has had misaligned ownership with SRE groups owning an access system and has been in a mixed state of ownership for the past 2 years, creating risk with no one team having expertise or a stake in maintaining the service. Production SREs have also been responsible for approving Teleport requests, which is similarly misaligned with their roles. This had become a bottleneck and source of frustration from all stakeholders engineering. **Outcomes** :tulip: Infrastructure Security has taken full ownership of Teleport, from vendor contract to the underlying infrastructure. [Access to InfraSec](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/27708) and Production Engineering team members has been updated and all [documentation](https://handbook.gitlab.com/handbook/security/corporate/systems/teleport/) reflects this new ownership. We avoided shifting the problem of approving individual Teleport access requests to a new team by instead [shifting the approval left to engineering managers](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1681#:~:text=EMs%20have%20the%20ability%20to%20approve%20Teleport%20requests). This grows the pool of approvers to help remove the bottleneck engineers were facing while also bringing the approvals to the people who are most aware of the need for access. --- :speaking_head: A special shoutout to @jacobjernigan and @dsalzmann for partnering on this handover. :collaboration: _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1681#note_2864657374_  | | [EOC and IMOC Coordinator Handover](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1685) ~"group::Networking & Incident Management" | 2025-09-01 | 2025-11-07 | **2025-11-04**: :handshake: **Closing status update!** :handshake: The EOC and IMOC Coordinator roles needed to transition from their previous owners to the NIM team as part of the Production Engineering restructuring, ensuring continuity of incident management operations for the IMOC participants and the SRE on-call rotation. :calendar: We successfully completed the handover with no operational disruption. The NIM team now manages both rotations, has conducted monthly schedule updates, and maintained all reporting cadences. Ongoing projects including Tier 2 expansion and the PagerDuty to incident.io migration continue with clear ownership. :success: **2025 Holiday Coverage Planning:** - [EOC Holiday Coverage 2025](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/27688) - [IMOC Holiday Coverage 2025](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/27630) With the handover complete and the NIM team fully operational as coordinators for both EOC and IMOC, we're ready to close this epic in the next Grand Review. :bow: Thank you to the outgoing coordinators for their thorough knowledge transfer! :trophy: _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1685#note_2864657483_  | | [Envoy Phase 0: Discovery](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1643) ~"group::Networking & Incident Management" | 2025-09-16 | 2025-10-08 | **2025-09-30**: :sparkles: **This epic is ready to close** :sparkles: This envoy discovery epic was timeboxed to two weeks to understand the Auth Architecture requirements and expectations of the Infrastructure Platforms group, and also gather initial requirements for what deprecating HAProxy will entail. **How do we deprecate HAProxy?** * We have documented a [HAProxy Functionality Table](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/27149#note_2782343397) * This has identified functionality that is still required, and the components that rely on it (main, ci, registry, etc). * This will form the basis for configuring a replacement for HAProxy, allowing us to iteratively migrate components to Envoy as that functionality is configured. * We will not proceed further with this, until the work from Auth Architecture reaches a later stage. **Partnership with Auth Architecture** * We have established a **path forward** to collaborate with the Auth Architecture group. * We will follow the [Component Ownership Model](https://handbook.gitlab.com/handbook/engineering/infrastructure/production/component-ownership-model/) to pave a path to production for all GitLab platforms. * We have [created an epic](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1710) (confidential) to capture issues coming to Infrastructure from the Auth group * We will utilise this epic for status updates on the progress of the project. * As necessary, and with notice where possible, we will use this epic to bring in other infrastructure teams as necessary (for example: Observability considerations, database concerns, etc). * We will have a recurring sync call with the team to build trust, and address concerns early. * We will continue to review architectural design documents as they arise, such as the [Envoy as a core data plane component](https://gitlab.com/gitlab-org/architecture/auth-architecture/design-doc/-/merge_requests/39) which is currently under review. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1643#note_2787033004_  | | [Cut over EOC and IM to Incident.io](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1648) ~"group::Networking & Incident Management" | 2025-10-01 | 2025-12-05 | **2025-12-02**: :clock1: **total hours spent this week by all contributors**: 28 :tada: **achievements**: - Updated weekly newsletter generator to use incident.io (https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/27908). - Removed unnecessary PagerDuty features from ChatOps (https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/27788) - Deleted the PagerDuty schedules and escalation policies! - This epic can now be closed. As such, [here is a short "demo"](https://www.youtube.com/watch?v=UZFSaomAY5o) where I discuss the project, it's successes, and it's challenges. Enjoy! _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1648#note_2925519228_  | | [Migrate Cloudflare provisioning/deprovisioning to Lumos](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1746) ~"group::Networking & Incident Management" | 2025-10-07 | 2026-02-24 | **2026-02-19**: :clock1: **total hours spent this week by all contributors**: 8 :tada: **achievements**: **Epic closing update** :danceduck: , We have successfully rolled out **Lumos provisioning for FedRAMP sandbox account** , with that we can close this epic and write off a good end to a long journey :railway_track: ps: There's still some assistance we might need to provide to the Corpsec team to deprovision `okta-cloudflare-users` group and some minor documentation updates , however the work is minimal and they do not need an entire epic update. Big thanks @ErikLentz , this would have not been possible without their consistent efforts and patience on making this work :rocket: :collaboration: _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1746#note_3095939825_  | | [Removing Customer Rate-Limit Bypasses](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/374) ~"group::Networking & Incident Management" | 2025-10-21 | 2026-02-06 | **2026-02-03**: :clock1: **total hours spent this week by all contributors**: 8 :handshake: Closing status update Over the past several months, we've made significant progress in strengthening our infrastructure security and rate limiting posture. We began by conducting a comprehensive analysis of all customers with rate limit bypasses, identifying that only two were still exceeding our standard thresholds. This discovery enabled us to safely remove numerous obsolete bypasses, including many that were no longer actively owned and several that covered unnecessarily broad CIDR ranges (one of which represented 2^96 possible IP addresses :scream_cat:). For the two remaining customers with legitimate high-volume needs, we've established ongoing engagement to develop sustainable solutions that will eliminate their bypass requirements. This work has been consolidated under a [follow-up epic ](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1812) to ensure visibility and coordinated progress. **Impact**: This initiative successfully removed more than **120 bypasses**, collectively representing **more than 2^96 IP addresses**. This substantially reduces risk to our infrastructure and helps ensure platform availability for all users. I'd like to extend special recognition to @thisisshreya for their thorough investigative work in identifying and documenting these bypasses, this effort was instrumental to our success :tada: :bow: _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/374#note_3053402897_  | | [FY26 Q4 DR Gamedays](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1752) ~"group::Networking & Incident Management" | 2025-11-01 | 2026-02-06 | **2026-02-03**: **Closing Status Update!** We have completed all 4 game days for FY26 Q4. - :green_circle: [haproxy](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/27781) completed without any serious findings - :green_circle: [runners](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/27780) completed without any serious findings - :yellow_circle: [gitaly](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21087) completed but with some improvements and takeaways - :red_circle: [patroni](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/27779) failed, but found a serious issue with spinning up new nodes, which makes it a successful Game Day _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1752#note_3053402934_  | | [Networking - Keep the Lights on work - Q1 FY27](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1832) ~"group::Networking & Incident Management" | 2026-02-01 | 2026-05-01 | **2026-04-29**: :clock1: **total hours spent this week by all contributors**: 0 No notable items this week _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1832#note_3296944479_  | | [Networking - incoming requests work - Q1 FY27](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1831) ~"group::Networking & Incident Management" | 2026-02-01 | 2026-05-01 | **2026-04-29**: :clock1: **total hours spent this week by all contributors**: 10 **Enterprise AI** - @devin - [Subdomain Delegation for CIO Tooling](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28766) - providing review support **Support** - @thisisshreya - https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28754 - provide analysis for customer following impact of circuit breaker rollout. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1831#note_3296944513_  | | [FCL work items for INC-8324 (Production Incident 21518)](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2004) ~"group::Networking & Incident Management" | 2026-03-12 | 2026-04-14 | **2026-04-01**: :clock1: **total hours spent this week by all contributors:** 8 :sparkles: **Closing Status** :sparkles: This FCL ran for 2 weeks following an [S1 DDoS attack production incident](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21518). We identified and put in motion a variety of protections at the edge that will improve our ability to defend against DDoS traffic and traffic pressure on our platform. :tada: **Here is what we've accomplished:** * [Removed the blanket Git bypass from Cloudflare](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28546) — bypass traffic dropped from \~140k/min to \~5k/min * [Completed full audit of all 42 Cloudflare rate limit rules](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28547), spawning 5 follow-up issues for identified gaps * Cleaned up obsolete log-only Cloudflare rules * [Engaged with Cloudflare](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28551) — key takeaway: our traffic patterns are unique and Cloudflares tools we currently have or could potentially add on (Bot Management, API Shield) won't work for us; they recommend third-party managed IP lists and verified many of the actions we are taking in this FCL were best practices * [Three attempts at enabling Git rate limits](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28578) that have led to a deeper understanding of the complexities of our Git request routing through the application. * [Deployed edge circuit breaker rules](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28556) in log mode :binoculars: **Where do we go from here?** * We are continuing the efforts to enable Git Rate limits and enabling the edge circuit breaker in block mode as part of the the [Code Yellow - Rate Limiting and DDoS](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2008). * Other remaining issues in this FCL have also been moved to the code yellow to be prioritized against other identified code yellow work. :mortar_board: **What we learned** * FCLs for infrastructure teams are difficult, but productive. * Do to the operational nature of our work, we had to say no to a lot of people, but also were unable to say no to all incoming work. * Capacity is a factor - we have team members on call and always need to be ready to respond to incidents when the needs arise. * The nature of some infrastructure work doesn't fit well in a 2 week period. When we roll out changes to production, we need to carefully tune and monitor - that takes time. * Having a period of sprint-like focus felt really good. Things moved forward. There was a collaborative nature to the discussions since everyone was working closely in the same area and on the same topics. Thank you to the entire team for your focused efforts and dedication to upholding GitLab's reliability. :tanuki-colored-heart: _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2004#note_3211163237_  | | [Code Yellow - Rate Limiting and DDos FY27Q2](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2008) ~"group::Networking & Incident Management" | 2026-03-23 | 2026-07-09 | **2026-05-27**: See [&2021](https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/2021) for detailed Rate Limit Simplification progress. :clock1: **total hours spent this week by all contributors**: 16 :tada: **achievements**: * **Cloudflare WAF log rules deployed for Git HTTP traffic.** [#28776](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28776) — Two Cloudflare log rules are now in place: one scoped to `.git` paths broadly, and a second targeting the high-cost `git-upload-pack` operation specifically. Currently in log mode; will be switched to block once the PCL window closes and the rules are validated as non-disruptive. :issue-blocked: **blockers**: * None :arrow_forward: **next**: * Switch Cloudflare Git HTTP WAF rules from log to block mode after PCL window closes ([#28776](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28776)) * Engage Git/Gitaly teams to analyze traffic patterns and determine safe long-term targets for Git HTTPs and LFS rate limits ([#28857](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28857)) _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/2008#note_3387739000_  | </details> ### :x: Cancelled Epics that were cancelled <details> | **Topic** | **Ended** | **Summary** | |-----------| ----------| ------------ | | [[placeholder] Seamless Change Management Lifecycle: Enable teams to operate independently within safe boundaries](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1679) | 2026-05-27 | | </details> ### :rotating_light: Epics that need attention These linked epics are not in the correct state or missing a workflow label <details> | **Topic** | **Links** | **Reason** | |-----------|-----------|-------------| | [Zonal and Regional Outage Resilience](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1505) @cmcfarland @swainaina (+2) group::Networking & Incident Management | Epic has ~"workflow-infra::Done" but is opened | | [Upgrade Cloudflare Terraform provider to v5 in config-mgmt and Terraform modules](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1663) (+0) group::Networking & Incident Management | Labeling problem, epic has ~"workflow-infra::Waiting" | </details>

epic