**Purpose:** Begin the incremental rollout to production by enabling steps on internal runners — progressing from lowest-risk to highest-risk targets. **Background:** We have several internal runner shards beyond just the private runner managers (PRM). Rather than jumping straight to the highest-volume runner, we should work through these systematically — starting with shards where breakage has minimal impact and building confidence before touching the runners that serve the monolith. The methodology follows the same ring-based approach we outlined for the [shared runner rollout](https://gitlab.com/groups/gitlab-org/-/work_items/21125): minimize blast radius early, validate on real workloads, expand as confidence grows. **Rollout Strategy:** ### Ring 0 — Canary (near-zero blast radius) | Runner | Shard | Limit | Rationale | |---|---|---|---| | `tamland` default | tamland | 10 | Canary shard running unstable runner builds. 20 global concurrent. If this breaks, nobody notices except us. Uses `docker` executor (not `docker+machine`) — validates a second code path early. | ### Ring 1 — Dev instance and Charts | Runner | Shard | Limit | Rationale | |---|---|---|---| | `dev-gitlab-org` | private | 200 | dev.gitlab.org is internal. Low-risk. | | `dev-gitlab-org-medium` | private | 50 | Same instance, medium size. | | `com-charts` | private | 50 | Charts group. | ### Ring 2 — Internal infra groups on gitlab.com | Runner | Shard | Limit | Rationale | |---|---|---|---| | `com-gitlab-com` | private | 150 | gitlab-com group (infra, handbook). Internal teams with direct support access. | | `ops-gitlab-net` | private | 150 | Ops instance. Slightly higher risk — infra automation runs here — but teams are reachable. | ### Ring 3 — High-volume gitlab-org | Runner | Shard | Limit | Rationale | |---|---|---|---| | `com-gitlab-org` | private | 1125 | General gitlab-org projects. High volume = good signal. Start with 2/16 managers (12.5%), then expand. | | `shared-gitlab-org` default | shared-gitlab-org | 900 | Same workloads, different shard. Expand after private is stable. | ### Ring 4 — Medium/large runners | Runner | Shard | Limit | Rationale | |---|---|---|---| | `com-gitlab-org-medium` | private | 625 | Bigger instances, same workloads. | | `shared-gitlab-org` default-medium | shared-gitlab-org | 200 | | | `qa-runners` dind | qa-runners | 1000 | High volume but only QA. MaxBuilds=1. | ### Ring 5 — Restricted / monolith CI (highest risk) | Runner | Shard | Limit | Rationale | |---|---|---|---| | `staging-gitlab-com` | private | 100 | Staging env. MaxBuilds=1, IdleCount=0. High impact because is part of gitlab release. | | `com-gitlab-org-large` | private | 100 | **Restricted to gitlab-org/gitlab, security/gitlab, gitlab-foss.** Breaking this blocks monolith MR pipelines. | | `dev-gitlab-org-large` | private | 100 | Restricted to gitlab-ee, gitlabhq on dev. | ### Deferred (different architectures / platforms) | Runner | Rationale | |---|---| | `distribution-*` (com + dev) | ARM64 + n1-highcpu-32. MaxBuilds=1. Blocking omnibus builds blocks releases. Different failure modes. | | `windows-ci` | Windows platform, 4 concurrent total. Not worth the complexity yet. | | `shared-gitlab-org` dind | Docker-in-Docker, MaxBuilds=1. Specialized workload. | **Configuration Changes:** Feature flags are set per-runner-registration via environment variables (per Tomasz's recommendation in !7097). E.g. for tamland: ```json "environment": [ "FF_USE_NATIVE_STEPS=true", "FF_SCRIPT_TO_STEP_MIGRATION=true" ] ``` Note: `FF_EXPORT_HIGH_CARDINALITY_METRICS` is under separate discussion with Runner Platform — not included by default. **Coordination:** This rollout requires coordination with the Runner Platform team (@tmaczukin): - Chef configuration changes to runner-manager instances - Integration with existing production Grafana dashboards (#318 metrics) - Monitoring by both Functions team and Runner Platform SREs **Prerequisites:** - Pilot deployment (#311) is successful and validated - Monitoring framework (#312) is operational - Step-specific metrics (#318) are implemented and exposed - Documentation (#321) is complete for SRE troubleshooting - docker+machine executor compatibility (#317) is verified - Team has analyzed pilot data and identified no blocking issues **Success Criteria (per ring):** - Feature flags enabled on target runners - Jobs execute via steps on affected runners - Adoption metrics show expected percentage - No increase in job failure rates - Runner Platform team comfortable with stability before advancing to next ring **Resources:** - [chef-repo runners-manager-private role](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/blob/master/roles/runners-manager-private.json) (Base configuration for private runners) - [Runbooks: Linux runners README](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/linux/README.md) (Runner architecture and deployment process) - [Runbooks: Blue-green deployment](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/ci-runners/linux/blue-green-deployment.md) (Deployment procedure) - [#317 - docker+machine compatibility](https://gitlab.com/gitlab-org/step-runner/-/issues/317) - [#318 - Step-specific metrics](https://gitlab.com/gitlab-org/step-runner/-/issues/318) - [Shared runner rollout methodology](https://gitlab.com/groups/gitlab-org/-/work_items/21125)