[GKE RM] shared vs dedicated VPC

Rehab
2026-04-02T15:56:08Z

## Description

Evaluate the trade-offs between using a **shared VPC** vs **dedicated VPC** approach for the new GKE Runner Managers cluster in HRPv5.

### Context

Epic https://gitlab.com/groups/gitlab-com/gl-infra/ci-runners/-/work_items/22 mentions:

> **Network approach:** Use shared VPC to avoid peering complexity and IP conflict management. Trade-off is less isolation, but simplifies operations.

This decision needs validation and documentation of the trade-offs before implementation.

### Historical Context from V4 Platform

The V4 Hosted Runners platform went through significant networking evolution ([source](https://ops.gitlab.net/gitlab-com/gl-infra/ci-runners/hosted-runners-mgmt/-/blob/master/docs/design/summary.md#networking-design)):

> Through the evolution of Hosted Runners platform we've gone from a single subnet in a single VPC, through many subnets within a single VPC, through multiple subnets in multiple VPCs and finally ended with multiple subnets deployed on smaller number of **Shared VPCs** (Shared VPC - a Google provided service).

**Key problems with VPC peering in V4:**

1. **Peering Group size limit**: Hit GCP's ~70 VPC peering limit with 100+ runner managers
2. **CIDR management burden**: All peered subnets must use unique addressing - requires manual [CIDR reservation list](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/ci-runner-networking.md#ephemeral-runners-unique-cidrs-list)
3. **Scaling concerns**: "if we want to scale to a 10x or 100x size we handle today, we will need to create a different networking design"

**Migration to Shared VPC solved these issues:**
- Reduced peering group size (one Shared VPC per shard instead of one VPC per runner manager)
- Simplified firewall management
- Maintained isolation at the application layer

Reference: [V4 Platform Network Design](https://ops.gitlab.net/gitlab-com/gl-infra/ci-runners/hosted-runners-mgmt/-/blob/master/docs/design/summary.md#networking-design)

### V5 Platform Network Design

The V5 platform design specifies a clear network architecture ([source](https://gitlab.com/gitlab-com/gl-infra/ci-runners/hosted-runners-mgmt/-/blob/master/docs/design/design.md#requirement-3-networking-layout)):

**Core principle: Autonomous systems**

Each runner manager should live in a dedicated, totally independent network:

> Each VPC should be entirely independent. There should be **no peering nor direct connections** with any other VPCs or remote networks. This solves one of the biggest known problems we have today: a need to register bigger CIDR blocks when defining environment for every single runner manager. Which is a huge toil.

**Network architecture:**

- **Dedicated VPC per shard replica**: Each runner manager gets its own isolated VPC
- **No VPC peering**: Eliminates peering limits and CIDR management overhead
- **Standardized addressing**: All replicas can use the same CIDR ranges (e.g., `10.0.0.0/16` for ephemeral VMs)
- **Private Service Connect**: Replaces VPC peering for access to `ci-gateway` and other GitLab services

**Subnet design** (identical across all replicas):

| Subnet | CIDR | Purpose |
|--------|------|---------|
| `bastion` | `10.0.0.0/24` | Bastion hosts (if needed) |
| `runner-managers` | `10.0.1.0/24` | Runner manager instances/pods |
| `monitoring` | `10.0.2.0/24` | Prometheus and observability |
| `ephemeral-vms` | `10.1.0.0/16` | Ephemeral VMs or Kubernetes nodes |

**Connectivity approach:**

- **ci-gateway access**: [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) instead of VPC peering
- **Cost**: $0 within same region (us-east1)
- **Cross-region**: Cheaper than routing through CloudFlare even with PSC costs

**Benefits of this approach:**
- ✅ No peering group size limits
- ✅ No CIDR conflict management
- ✅ Simplified horizontal scaling (replicas are identical)
- ✅ True isolation between shards
- ✅ Cost-effective private connectivity via PSC

Reference: [V5 Design - Networking Layout](https://gitlab.com/gitlab-com/gl-infra/ci-runners/hosted-runners-mgmt/-/blob/master/docs/design/design.md#requirement-3-networking-layout)

### Current Infrastructure

The CI runners infrastructure currently uses **Shared VPC** for VM-based runner managers:
- **Host project**: `gitlab-ci-155816`
- **Service projects**: Various ephemeral projects (one per shard)
- **Rationale**: Previously hit GCP's 70 VPC peering limit; shared VPC provides better scalability

Reference: [CI Runner Networking Documentation](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/ci-runner-networking.md#vpc-architecture)

### Experimental GKE Clusters

The experimental clusters (`gitlab-org-experimental`, `gitlab-org-experimental-gvisor`, `gitlab-org-experimental-kata`) use a **dedicated VPC** approach:
- Dedicated `experimental` VPC network
- VPC peering to `gitlab-production/ci-gateway` and `gitlab-staging-1/ci-gateway`
- Separate subnets per cluster/runtime

**Note:** Experimental clusters were built for testing/validation before V5 design was finalized. They use VPC peering rather than Private Service Connect.

Reference: [experimental.tf](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/blob/master/environments/ci/experimental.tf#L5-121)

## Options

### Option 1: Shared VPC (Epic &22 Recommendation)

**Pros:**
- ✅ Consistent with current VM-based runner-managers architecture
- ✅ Simpler initial setup (leverages existing `gitlab-ci-155816` infrastructure)
- ✅ Proven solution that enabled scaling from individual VPCs to 100+ runner managers
- ✅ No new GCP features required (PSC setup complexity avoided initially)

**Cons:**
- ❌ **Conflicts with V5 design**: V5 explicitly requires "no peering nor direct connections"
- ❌ Less network isolation between runner managers and workload clusters
- ❌ Requires coordination with `gitlab-ci-155816` project administrators
- ❌ Shared fate - network issues affect all service projects
- ❌ **Migration required**: Will need to migrate to dedicated VPCs with PSC for V5
- ❌ **CIDR conflicts possible**: If multiple runner manager clusters share VPC

**Implementation:**
- Make `gitlab-ci-155816` the shared VPC host project (already configured)
- Create new GKE cluster as service project attached to shared VPC
- Leverage existing subnets or create new ones in host project
- Plan migration path to V5 dedicated VPC architecture

### Option 2: Dedicated VPC with VPC Peering (Current Experimental Pattern)

**Pros:**
- ✅ Complete network isolation
- ✅ Independent network policy control
- ✅ Easier to tear down/rebuild for testing
- ✅ No dependencies on shared VPC host project

**Cons:**
- ❌ **Conflicts with V5 design**: V5 explicitly avoids VPC peering
- ❌ Requires VPC peering to `ci-gateway` networks (2 peerings: GPRD + GSTG)
- ❌ Additional peering to monitoring networks
- ❌ Risk of CIDR conflicts with existing networks
- ❌ More complex to manage at scale (multiple VPCs to monitor)
- ❌ Contributes to VPC peering limit exhaustion
- ❌ **Historical precedent**: This is the approach V4 migrated AWAY from due to scaling issues
- ❌ **CIDR management burden**: Requires manual coordination via reservation list
- ❌ **Migration required**: Will need to migrate to PSC for V5

**Implementation:**
- Create new VPC in `gitlab-ci-155816` (or new project)
- Set up VPC peering to `gitlab-production/ci-gateway` and `gitlab-staging-1/ci-gateway`
- Define new CIDR ranges that don't conflict
- Add CIDR reservations to [the tracking list](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/ci-runner-networking.md#ephemeral-runners-unique-cidrs-list)

### Option 3: Dedicated VPC with Private Service Connect (V5 Design)

**Pros:**
- ✅ **Aligns with V5 design**: Implements the intended architecture from the start
- ✅ Complete network isolation
- ✅ No VPC peering limits
- ✅ No CIDR conflict management (can reuse same ranges across replicas)
- ✅ Independent network policy control
- ✅ Easier horizontal scaling (all replicas use identical config)
- ✅ No migration required when moving to full V5 platform
- ✅ Cost-effective ($0 within us-east1 region)
- ✅ Modern GCP-native approach

**Cons:**
- ❌ Requires PSC setup (new infrastructure pattern for team)
- ❌ PSC configuration complexity (learning curve)
- ❌ Different from current V4 VM-based architecture
- ❌ May require cross-region costs if not in us-east1

**Implementation:**
- Create dedicated VPC in new GCP project
- Define standard subnet layout per V5 design:
- `runner-managers`: `10.0.1.0/24`
- `monitoring`: `10.0.2.0/24`
- `ephemeral-vms` (GKE nodes): `10.1.0.0/16`
- Configure Private Service Connect endpoint to `ci-gateway`
- Configure PSC endpoint to `vault.gitlab.net` (if needed)
- Set up Prometheus within VPC for metrics collection → Mimir

## Questions to Answer

1. **PSC readiness**: Do we have the expertise/documentation to set up Private Service Connect for `ci-gateway` access?
2. **Migration vs greenfield**: Is it better to start with V5 architecture now, or migrate from Shared VPC later?
3. **Workload cluster connectivity**: Do runner manager pods need direct network access to workload cluster pods, or only to the Kubernetes API server?
4. **Region constraints**: Will the GKE Runner Managers cluster be in us-east1 (for $0 PSC costs)?
5. **Timeline alignment**: When is full V5 platform expected? Does that affect the urgency of implementing V5 network architecture?
6. **Risk tolerance**: Are we willing to implement PSC (new pattern) now, or prefer familiar Shared VPC with planned migration?

## Acceptance Criteria

- [ ] Document decision rationale with pros/cons analysis
- [ ] Identify CIDR ranges needed (if new subnets required)
- [ ] Validate approach doesn't conflict with existing network topology
- [ ] If using PSC: Document PSC setup requirements and costs
- [ ] If using Shared VPC: Document migration plan to V5 dedicated VPC architecture
- [ ] Update epic https://gitlab.com/groups/gitlab-com/gl-infra/ci-runners/-/work_items/23 with network architecture decision
- [ ] Create follow-up issue for implementation

## References

- **V5 Platform Epic**: gitlab-com/gl-infra&1611 ([Unified Hosted Runners Platform](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1611))
- **Parent Epic**: gitlab-com/gl-infra/ci-runners&23 ([GKE cluster for Runner Managers](https://gitlab.com/groups/gitlab-com/gl-infra/ci-runners/-/epics/23))
- **V5 Network Design**: [design.md - Networking Layout](https://gitlab.com/gitlab-com/gl-infra/ci-runners/hosted-runners-mgmt/-/blob/master/docs/design/design.md#requirement-3-networking-layout)
- **V5 Requirements Summary**: [summary.md - Simplify networking design](https://ops.gitlab.net/gitlab-com/gl-infra/ci-runners/hosted-runners-mgmt/-/blob/master/docs/design/summary.md#simplify-networking-design)
- **V4 Platform Design**: [summary.md - Network Design Section](https://ops.gitlab.net/gitlab-com/gl-infra/ci-runners/hosted-runners-mgmt/-/blob/master/docs/design/summary.md#networking-design)
- **Network documentation**: [ci-runner-networking.md](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/ci-runner-networking.md)
- **Current shared VPC config**: [network_vpc.tf](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/blob/master/environments/ci/network_vpc.tf)
- **Experimental VPC example**: [experimental.tf](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/blob/master/environments/ci/experimental.tf)
- **Private Service Connect**: [GCP Documentation](https://cloud.google.com/vpc/docs/private-service-connect)