## DRI @cfeick ## Participants @cfeick @fforster @reprazent ## OKR https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5896 ## Context Latency has been a [primary concern](https://gitlab.com/gitlab-org/gitlab/-/issues/434063#note_1681951804) for the performance of AI Gateway and https://gitlab.com/gitlab-org/gitlab/-/issues/434873+ was raised in order to address this. Regional deployments will also be a benefit to all Runway services where we can position the service closer to the user. This will primarily benefit self-managed users, standalone services (like IDE extensions), or users not using gitlab.com as GitLab multi-tenant SaaS is currently deployed in a single region, meaning that there will always be a hop to that region.  ## Proposal Support multi-region deployments as part of Runway. The solution should: - Be simple to use as part of the `runway.yml` setup - Not require code/configuration duplication ### Example Runway Configuration ```yaml apiVersion: runway/v1.0.0-beta kind: RunwayService spec: image: "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}" regions: - europe-west1 - us-east-1 - asia-east1 ``` ## Exit Criteria - [x] Load Balancing via GCP - [x] Regions configurable via `runway.yml` - [x] Documentation available for how to perform a regional deployment ## Milestones The project milestones are listed below. These milestones are consecutive and are to be delivered as part of this epic: * [x] 1. [Multi-Region Cloud Run Experiment](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/245#tab-issues) * [x] 2. [Multi-Region Cloud Run Beta](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/247#tab-issues) * [x] 3. [Multi-Region Cloud Run GA](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/248#tab-issues) These are internal milestones in support of this project. ## Gateway Locations We will * Deploy 4 additional gateways * Currently 4 already exists * In total we will have 8 gateways by end of FY25Q2 | AI Gateway Regions| Status| |-----------------------------------------------------|--- | | Moncks Corner, South Carolina (us-east1) | Deployed | | Northern Virginia (us-east4) | Deployed | | London, United Kingdom (europe-west2) | Deployed | | Seoul, Korea (asia-northeast3) | Deployed | | Frankfurt, Germany (europe-west3) | Planned | | Tokyo, Japan (asia-northeast1) | Planned | | Paris, France (europe-west9) | Planned | | Los Angeles, California (us-west2) | Planned | ## Development Log <details> <details> <summary>Status 2024-04-24</summary> We have delivered [results](https://console.cloud.google.com/monitoring/metrics-explorer;duration=PT6H?pageState=%7B%22domainObjectDeprecationId%22:%22BD140DAE-031F-41D5-9CB5-4D5CC7E361BF%22,%22xyChart%22:%7B%22constantLines%22:%5B%5D,%22dataSets%22:%5B%7B%22plotType%22:%22LINE%22,%22targetAxis%22:%22Y1%22,%22timeSeriesFilter%22:%7B%22aggregations%22:%5B%7B%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22groupByFields%22:%5B%22resource.label.%5C%22location%5C%22%22%5D,%22perSeriesAligner%22:%22ALIGN_RATE%22%7D%5D,%22apiSource%22:%22DEFAULT_CLOUD%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22filter%22:%22metric.type%3D%5C%22serviceruntime.googleapis.com%2Fapi%2Frequest_count%5C%22%20resource.type%3D%5C%22consumed_api%5C%22%20resource.label.%5C%22project_id%5C%22%3D%5C%22unreview-poc-390200e5%5C%22%20resource.label.%5C%22service%5C%22%3D%5C%22aiplatform.googleapis.com%5C%22%22,%22groupByFields%22:%5B%22resource.label.%5C%22location%5C%22%22%5D,%22minAlignmentPeriod%22:%2260s%22,%22perSeriesAligner%22:%22ALIGN_RATE%22%7D%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22y1Axis%22:%7B%22label%22:%22%22,%22scale%22:%22LINEAR%22%7D%7D%7D&project=unreview-poc-390200e5) by rolling out full end-to-end Multi-Region for both Cloud Run and Vertex AI for AI Gateway in staging environment:  Previously, every Vertex AI request was routed to `us-east4` region. Now, every Vertex AI request is routed to same Cloud Run region. Worth noting: self-managed customers are expected to primarily benefit, while SaaS customers still go through monolith hosted in `us-east` until 17.0 milestone. Scope is not part of this project. We have completed largest remaining tasks for [GA milestone](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/248#tab-issues). We have delivered new `runway_lb` [SLI](https://dashboards.gitlab.net/d/ai-gateway-main/ai-gateway3a-overview?orgId=1&viewPanel=94), which will trigger alerts for unhealthy backends:  Worth noting: all Runway services w/ external LB automatically benefit from these changes with no additional effort. The impact extends beyond context of this work. We delivered regional GCP artifact registries for hosting container images, which will improve availability and disaster recovery in the event of regional outage/incident. Remaining tasks are being evaluated and final pass for any missing issues. Otherwise, we are largely feature complete and still on track to complete project by original target of April 30th. Next steps: we are preparing https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17705 to rollout full end-to-end multi-region support in production and completing documentation for AI Gateway maintainers to fully self-serve future regions. Addendum: Next day after posting previous status update, Test Platforms surfaced issue that AI Gateway does not return 5xx responses when Vertex AI is unavailable. Code Suggestions were not returning responses and silently failing. Blocker tasks have been created in https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2599#note_1880447523, which may impact original plan to rollout in production early next week. </details> <details> <summary>Status 2024-04-17</summary> We met ambitious goal set last week. We have delivered [results](https://dashboards.gitlab.net/d/ai-gateway-regional/ai-gateway3a-regional-detail?orgId=1&from=1713193200000&to=1713366000000&var-PROMETHEUS_DS=PA258B30F88C30650&var-environment=gstg&var-stage=main&viewPanel=3897771623) by rolling out Multi-Region Cloud Run Beta for AI Gateway in [staging](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17704) environment:  Previously, every HTTP request was routed to `us-east1` region. Now, every HTTP request is routed to nearest regional Cloud Run service backend using GCP Global Load Balancer. Initial regions are `us-east1`, `europe-west2`, and `asia-northeast3`. GCP quota increases were requested and approved. Additional [regions](https://gitlab-com.gitlab.io/gl-infra/platform/runway/runwayctl/manifest.schema.html#spec_regions) can be provisioned by Runway service owners using `runway.yml` in CI/CD. Documentation was created to allow Runway service owners to self-serve [multi-region feature](https://runway-docs-4jdf82.runway.gitlab.net/guides/multi-region/) using [service manifest](https://runway-docs-4jdf82.runway.gitlab.net/reference/service-manifest/). As a result, we have essentially completed Multi-Region Cloud Run Beta [milestone](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/247#tab-issues). We have a fully functional and observable Multi-Region Cloud Run deployment that can be used for development and testing by Runway service owners and stakeholders. Next steps: Continuing work on Multi-Region Cloud Run GA [milestone](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/248#tab-issues) to meet production-readiness standards. Expectations for cross-team dependancies on Vertex AI are being discussed by leadership, more up-to-date information will be available in https://gitlab.com/gitlab-com/gl-infra/mstaff/-/issues/315#note_1824428102. </details> <details> <summary>Status 2024-04-10</summary> Due to rapidly developing situation involving customer escalation, we have proposed 3-phase plan dubbed "Region X": 1. https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/226 1. https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/227 1. https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/228 The ambitious goal is to rollout Multi-Region Cloud Run Beta for AI Gateway in staging by next week. We are working directly with stakeholders on action items in related issues and coordinating in [`#scalability-1206-multi-region-for-ai-gateway`](https://gitlab.enterprise.slack.com/archives/C06TMABE1EE) for most update-to-date information. We have increased staffing from 2 participants to 3 participants. </details> <details> <summary>Status 2024-04-03</summary> :timer: Short cycle due to multiple public holidays, unexpected personal time off, and upcoming friends & family day. :octagonal_sign: Blocked on https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/3392. Escalated to ~"team::Scalability-Observability". :arrow_right: https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/698 for routing to Vertex AI region has been initially reviewed by Runway maintainers. Waiting for review by AI Gateway maintainers. :arrow_right: https://gitlab.com/gitlab-com/gl-infra/platform/runway/runwayctl/-/merge_requests/292 for remote writing custom metrics to Mimir has been opened. https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2722 has been picked up as path forward by using global Mimir source in [AI Gateway regional dashboard](https://dashboards.gitlab.net/d/ai-gateway-regional/ai-gateway3a-regional-detail?orgId=1&var-PROMETHEUS_DS=e58c2f51-20f8-4f4b-ad48-2968782ca7d6&var-environment=gprd&var-stage=main). Multi-region project will now be somewhat coupled to Mimir project. :white_check_mark: Related epic https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1222#status-2024-04-03 has added support for dry run deployments, which is note worthy in this update because it will provide Terraform plans to more safely rollout multiple regions for AI Gateway. </details> <details> <summary>Status 2024-03-27</summary> 🎉 We have completed Runway functionality for routing to multiple regions in Cloud Run. https://gitlab.com/gitlab-com/gl-infra/platform/runway/runwayctl/-/merge_requests/280 has been merged and milestone https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/245#tab-issues is complete. We have started work for routing to Vertex AI region based on Cloud Run region. https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/698 has been opened and milestone https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/246#tab-issues is in progress. We still need to work through observability issues in https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/185#note_1824535315 before AI Gateway's Runway version can be updated and milestone https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/247#tab-issues can be completed. Regional detail dashboard now loads default SLIs from Mimir [datasource](https://dashboards.gitlab.net/d/ai-gateway-regional/ai-gateway3a-regional-detail?from=now-6h%2Fm&to=now%2Fm&var-PROMETHEUS_DS=e58c2f51-20f8-4f4b-ad48-2968782ca7d6&var-environment=gprd&orgId=1&var-stage=main), however, custom SLIs are not loaded due to https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2722. Mimir project is not estimated to be completed before multi-region project target date, so we will need to backport regional source recording rules to Thanos as workaround. We have increased staffing from 1 participant to 2 participants. Project remains on track for target date. </details> <details> <summary>Status 2024-03-20</summary> We have safely returned from Summit and project has resumed. We have made progress on two fronts this week: observability and global LB. We have shipped per region SLIs/SLOs: * [Default metrics for Runway](https://dashboards.gitlab.net/d/ai-gateway-main/ai-gateway3a-overview?orgId=1&viewPanel=78) * [Custom metrics for AI Gateway](https://dashboards.gitlab.net/d/ai-gateway-main/ai-gateway3a-overview?orgId=1&viewPanel=138) We are continuing to work w/ ~"team::Scalability-Observability" on final observability components in https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/185#note_1823992806. We have deployed first multi-region test service: * https://console.cloud.google.com/run/detail/us-east1/cfeick-dev-zg7kh0/metrics?project=gitlab-runway-staging * https://console.cloud.google.com/run/detail/us-west1/cfeick-dev-zg7kh0/metrics?project=gitlab-runway-staging * https://console.cloud.google.com/net-services/loadbalancing/details/http/cfeick-dev-zg7kh0-url-map?project=gitlab-runway-staging :tada: We consider this a huge win that has proven feasibility and removed biggest unknowns for global LB routing to closest geographical backend. As a result, **Runway service owners will be able to flexibly configure multi-region deployments entirely in CI/CD**: https://gitlab-com.gitlab.io/gl-infra/platform/runway/runwayctl/manifest.schema.html#spec_regions.  Next steps, we are working on final testing, reviewing, and releasing multi-region resources for Cloud Run. </details> <details> <summary>Status 2024-03-06</summary> We have completed retrofitting observability by adding support for `region` label dimension to default metrics (https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/merge_requests/4467) and custom metrics (https://gitlab.com/gitlab-com/gl-infra/platform/runway/runwayctl/-/merge_requests/238). Based on region metrics, we are in the process of upgrading AI Gateway SLIs/SLOs and saturation monitoring to support regional aggregation ([https://gitlab.com/gitlab-com/runbooks/-/merge_requests/6960,](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/6960) https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/670). Right now, AI Gateway is currently deployed in `us-east1` . By establishing regional baseline for existing dashboards and alerts, we will be able to more safely rollout deployment across multiple regions. Increased granularity will help us gain visibility into per region availability and identify any expected performance gains from multi-region support. In multi-region deployment, global LB will start playing critical component in routing to region closest to user. Next steps, we will be adding new `runway_glb` SLI based on [LB metrics](https://cloud.google.com/load-balancing/docs/metrics), which will complement existing `runway_ingress` SLI for regional Cloud Run services. **Addendum**: Project participants will be in attendance at Summit. Project will resume after Summit on Monday, March 18th, 2024. </details> <details> <summary>Status 2024-02-28</summary> Project milestones have been created to ensure incremental development of multi-region support and safe migration for AI Gateway: 1. [Route Runway to multiple Cloud Run regions](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/245#tab-issues) 1. [Route AIGW to multiple Vertex AI regions](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/246#tab-issues) 1. [Serve 100% of AIGW staging traffic from multi-regions](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/247#tab-issues) 1. [Serve 100% of AIGW production traffic from multi-regions](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/248#tab-issues) When starting draft for provisioning multi-region resources in https://gitlab.com/gitlab-com/gl-infra/platform/runway/runwayctl/-/merge_requests/216, we identified need to refactor rollouts across [GitLab-managed Terraform state](https://docs.gitlab.com/ee/user/infrastructure/iac/terraform_state.html). Next steps will be defining blue/green strategy in https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/205 to ensure availability when gradually rolling out deployments across multiple regions. </details> <details> <summary>Status 2024-02-21</summary> Project has started ~"workflow-infra::In Progress". Highlights: * Completed schema for region configuration (https://gitlab.com/gitlab-com/gl-infra/platform/runway/runwayctl/-/merge_requests/215) * Started provisioning regional resources (https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/181, https://gitlab.com/gitlab-com/gl-infra/platform/runway/team/-/issues/180) </details> <details> <summary>Status 2024-02-14</summary> Technical plan in https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1206#note_1774175341 for ~"workflow-infra::Proposal". Discussion in https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2864 is being used to decide whether multi-region support should be on critical path for Duo Chat GA. </details> <details> <summary>Status 2024-01-22</summary> Epic created for review (~"workflow-infra::Triage") </details> </details> ## Status 2024-05-01 **The project has been completed by estimated due date of April 30th, 2024. The KR progress has achieved 100% completion. Below is closing summary.** **Project problem** Previously, Runway services were deployed to single region in `us-east1` . The problems of single region deployments are latency and availability. The opportunities of multi-region deployments are decreasing latency by serving traffic from nearest regional backend, and increasing availability by supporting region failover during outages. **Project summary** Runway Multi-Region is now Generally Available (GA) and meets production-readiness standards for scalability and observability. Runway provides functionality to provision and serve traffic from multiple regions in GCP Cloud Run. Available locations include up to 40 regions closest to customers. **Project impact** Runway has delivered a stable, observable, and scalable long-term platform solution that powers multiple services deployed on paved road in SaaS Platforms. To reiterate: results are entirely automated and reusable outside context of AI Gateway, including upcoming Cells Topology service. In fact, Runway’s documentation is [multi-region deployment](https://gitlab.com/gitlab-com/gl-infra/platform/runway/docs/-/blob/master/.runway/runway.yml?ref_type=heads#L11-13) used for dogfooding. By rolling out `us-east4` , `europe-west2`, and `asia-northeast3` regions for AI Gateway, Runway has delivered the [first multi-region service in production](https://console.cloud.google.com/run?project=gitlab-runway-production&pageState=(%22cloudRunServicesTable%22:(%22f%22:%22%255B%257B_22k_22_3A_22Name_22_2C_22t_22_3A10_2C_22v_22_3A_22_5C_22ai-gateway_5C_22_22_2C_22s_22_3Atrue_2C_22i_22_3A_22name_22%257D%255D%22))) at GitLab. We've enabled AI Gateway maintainers to self-serve additional regions as service grows without infrastructure being bottleneck. As Stable Counterpart for AI Gateway, we’ve also updated AI Gateway to route Vertex AI requests to nearest region based on Cloud Run region. As a result, AI Gateway has full end-to-end multi-region support for both Cloud Run and Vertex AI. **Project artifacts** To self-serve multi-region deployments, refer to documentation: https://docs.runway.gitlab.com/guides/multi-region/. The platform experience is as simple as adding few lines to `runway.yml` [service manifest](https://gitlab-com.gitlab.io/gl-infra/platform/runway/runwayctl/manifest.schema.html#spec_regions) that will configure [global load balancer](https://dashboards.gitlab.net/d/ai-gateway-main/ai-gateway3a-overview?orgId=1&viewPanel=94) to route HTTP requests to nearest [regional service](https://dashboards.gitlab.net/d/ai-gateway-regional/ai-gateway3a-regional-detail?from=now-6h%2Fm&to=now%2Fm&var-PROMETHEUS_DS=PA258B30F88C30650&var-environment=gprd&orgId=1&viewPanel=3897771623). **Project final status update** The project milestones and exit criteria have been marked as complete. During Grand Review, please close the epic. In the spirit of iteration, GA is just the beginning and we will continue to make investments for service owners by improving and expanding platform capabilities. Thank you to all participants for your contributions. A project of such complexity and urgency could not have achieved customer results so quickly without collaboration and transparency from many team members across Scalability group, SaaS Platforms, Test Platforms, and AI-powered stage.