## Strategy and Timeline (revised 2024-07-08) - We are fully supporting the GitLab maintained Docker Machine fork for autoscaling runner on VM's on the significant public providers at a minimum through **FY27 Q2 (MAY-JUNE 2026).** - **Architecture**: Refer to the [Next Runner Auto-scaling Architecture blueprint document](https://docs.gitlab.com/ee/architecture/blueprints/runner_scaling/index.html) for additional details. - The new Runner Autoscaling solution is based on the Next Runner Architecture and so enables the development of autoscaling plugins for other cloud platforms by the wider community. | Public Cloud | Feature | Milestone | |--------------|---------|-----------| | AWS | [GitLab Runner AWS EC2 Fleeting Plugin - Experimental](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29219 "GitLab Runner Auto-scaling: Fleeting plugin for AWS EC2 instances - Experimental") | 15.11 | | AWS | [GitLab Runner AWS EC2 Fleeting Plugin - Beta](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29404 "GitLab Runner Fleeting plugin for AWS EC2 instances (BETA)") | 16.5 | | AWS | [GitLab Runner AWS EC2 Fleeting Plugin GA](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29222 "GitLab Runner Fleeting plugin for AWS EC2 instances (GA)") | 17.2 (JULY 2024) | | GCP | [GitLab Runner GCP Fleeting Plugin - Experimental](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29217 "GitLab Runner Auto-scaling: Fleeting plugin for Google Compute Engine - Experiment.") | 16.0 | | GCP | [GitLab Runner GCP Fleeting Plugin - Beta](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29409 "GitLab Runner Fleeting plugin for GCP Compute Engine - Beta") | 16.6 | | GCP | [GitLab Runner GCP Fleeting Plugin - GA](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29221 "GitLab Runner Fleeting plugin for GCP Compute Engine - GA") |17.1 (JUNE 2024) | | Azure | [GitLab Runner Azure VMs Fleeting Plugin - Experimental](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29410 "GitLab Runner Fleeting plugin for Azure Virtual Machines - Experimental") | 16.1 | | Azure | [GitLab Runner Azure VMs Fleeting Plugin - Beta](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29220 "GitLab Runner Fleeting plugin for Azure Virtual Machines - BETA") | 16.7 | | Azure | [GitLab Runner Azure VMs Fleeting Plugin - GA](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29223 "GitLab Runner Fleeting plugin for Azure Virtual Machines - (GA)") |17.3 (AUG 2024)| ## Summary of the plan to replace Docker Machine GitLab's [next Runner Auto-scaling Architecture](https://docs.gitlab.com/ee/architecture/blueprints/runner_scaling/#next-runner-auto-scaling-architecture) is the technical blueprint used to develop the replacement for the current Docker Machine-based autoscaler. The Taskscaler and Fleeting libraries are used in the GitLab Runner binary in a new “autoscaler” executor. ### [Taskscaler](https://gitlab.com/gitlab-org/fleeting/taskscaler) - is the new GitLab developed autoscaling component that replaces the Docker Machine technology. - uses a new library called [Fleeting](https://gitlab.com/gitlab-org/fleeting), which provides an “instance group” (IG) abstraction. - is used by providing credentials and the name of the autoscaling group (AWS ASG, GCP). On your cloud provider, you configure your instance type and instance size, and in GitLab Runner, you only reference the autoscaling group by name. ### [Fleeting...](https://gitlab.com/gitlab-org/fleeting) - is an abstraction for cloud providers' instance groups. It allows for the provisioning of multiple identical instances with a minimal API focused on just creation, connection and deletion. - is the library used by Taskscaler ## Fleeting plugin - a concrete implementation of a fleeting instance group representing a specific Google Compute Instance Group Manager (IGM) or Amazon Web Services Auto Scaling Group (ASG) - when initalized. ### Cloud Provider plugins Individual plugin binaries are created for each cloud provider via the [Hashicorp go-plugin](https://github.com/hashicorp/go-plugin). - [AWS plug-in](https://gitlab.com/gitlab-org/fleeting/fleeting-plugin-aws) - creates AWS instances using ASG's. - [GCP plug-in](https://gitlab.com/gitlab-org/fleeting/fleeting-plugin-googlecompute) - creates GCP instances using IGM's ## How to use the new solution? 1. [GitLab Runner Autoscaling](https://docs.gitlab.com/runner/runner_autoscale/) 2. [Docker Autoscaler](https://docs.gitlab.com/runner/executors/docker_autoscaler.html) 3. [Instance executor](https://docs.gitlab.com/runner/executors/instance.html) ### Reference links - [Machine blueprint terms merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98717) - [Initial Taskscaler merge request](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3617) - [Taskscaler 101 overview video](https://youtu.be/GU81g2oOtAQ) - [Code repository - Taskscaler, Fleeting, Cloud provider plugins](https://gitlab.com/gitlab-org/fleeting) - https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3617+ ## Overview and Context ## Why we need to migrate away from Docker Machine 1. The official project is in [maintenance mode](https://github.com/docker/machine/issues/4537) 1. Doesn't support new features from cloud providers 1. Doesn't support new availability zones for cloud providers 1. New bugs show up with the newer version of Docker such as https://github.com/docker/machine/issues/4858 1. Maintaining our own [fork](https://gitlab.com/gitlab-org/ci-cd/docker-machine) is expensive from an engineering perspective, requires us to be experts on cloud provider API. 1. Know bugs on a cloud provider such as Azure spot instances 1. Maintain [20+ drivers](https://docs.docker.com/machine/drivers/) requires us to know all of those drivers. 1. We are a small team that can't handle the large workload it requires to maintain `docker-machie` especially since it's not our main goal at GitLab. 1. No support for Windows 1. Little to no support for arm64 ## Problem to solve: [Autoscaling](https://docs.gitlab.com/runner/configuration/autoscale.html) of GitLab Runner on Virtual Machines hosted on the major cloud platforms is done today with Docker+Machine which is in [maintenance mode](https://github.com/docker/machine/issues/4537). Below is an overview of what needs to happen - Provide a solution for our existing users who are using `docker+machine` executor to autoscale on cloud providers. - Propose a rollout plan for users on how to migrate away from `docker+machine` executor. - Deprecate `docker+machine` executor, freezing any new feature requests and bug fixes. ### Why the runner group need to handle autoscaling / Providing reference architectures/templates This sounds like a lot of work and it is! So why the runner team should handle this? If the runner team does this work the problems below will be solved or in a better situation. #### It helps us build a better GitLab Runner In the past, the runner team sometimes said no to handling something, for example, helm chart, Kubernetes operator just because of bandwidth reasons. Which is fine and maybe was the right call at the time. However, the problem we face at the moment is these solutions are provided by other team members, which is awesome to see, but the runner team is either not aware of the effort or we (the runner team) didn't take into consideration their pain points and made it difficult for them whilst developing a new feature. It was also hard to prioritize their features/fixes they needed because we had competing priorities or we didn't have the knowledge of how GitLab Runner was being used. We end up being ignorant of most of the pain points of deploying the runner on that platform. We then took ownership of these platforms because we wanted them to be first-class citizens and it also helps the runner team develop a better runner, for example [config templates](https://docs.gitlab.com/runner/register/#runners-configuration-template-file) was developed to solve a pain point with the helm chart because we understood how hard it was to add configuration, we also discussed if we should change the configuration to YAML because of this. #### Toil work and pulled in multiple directions We also get constant questions from Professional Services/Support/Sales on how to best deploy GitLab Runner. The engineers sometimes also get pulled in for emergency support to help them with their own snowflake deployment. We had cases where engineers had to drop everything to help other teams sort out the deployment method for our customers. #### Snowflake deployments One of the hardest things about GitLab Runner is that it can be deployed on a lot of platforms and on a different kind of infrastructure. We provide no/little guidance on how to deploy and scale a GitLab Runner fleet, when they ask us how we do it we just say chef, and it's all internal so we can't really share it but most of the time it ends up with an engineer/PM talking to the customer to guide them through it. This also leads to hard to debug problems because everyone has their own twist on their infrastucture. ## Deployment methods In https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27061 we did some investigation on how our current users are using the `docker+machine` executor and any other autoscaling they might be using for `gitlab-runner`. There is a[summary presentation](https://docs.google.com/presentation/d/1Vi6kkIAdfIWez8JgUFq47WeozQz7Ih-cs-9zPHffKjM/edit?usp=sharing) and a [recording of the presentation](https://youtu.be/5zzqx2E4qtQ) of that presentation to bring you up to speed. We've identified different deployment methods of the runner and how they are using `docker+machine` or anything else. Terms: - **Self-hosted**: Users running their own `gitlab-runner` fleet. - **GitLab.com**: Users using the [shared runner fleet](https://about.gitlab.com/blog/2016/04/05/shared-runners/) provided by GitLab Inc. ### Self-hosted - Using their datacenter - Usually, air-gapped environments - Using hypervisor to manage VMS - Running trusted code, only company code. - Can be using OpenStack/Nomad for scheduling workloads. ### Self-hosted - Kubernetes - Can be used in two setups; Using cloud provider kubernetes offering, or using bare metal servers in their datacenter. - Autoscaling is provided by Kubernetes itself. - Kubernetes expertise differs from user to user. - Autoscaling happens both on `gitlab-runner` and the cluster that runs jobs. - Running trusted code, no isolation concerns. ### Self-hosted - Cloud provider VMs - Using cloud providers like AWS/GCP to host their VMs. - Ability to use the cloud provider [autoscaling](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27061#cloud-provider-autoscaling-compared-to-docker-machine) capabilities. - Running trusted code, no isolation concerns. <br/> - AWS: https://gitlab.com/groups/gitlab-org/-/epics/5223 ### GitLab.com - Unprivileged containers - Run a job without managing infrastructure. - Can specify their own Docker image. - Running untrusted code. - Containers aren't running in privileged mode, so users can't build docker images with `dind`. - What the user has locally should run in CI without any hiccups. ### GitLab.com - Privileged containers - All from [unprivileged containers](https://gitlab.com/groups/gitlab-org/-/epics/2502#gitlabcom-unprivileged-containers) - Running containers in a privileged mode so users can build Docker containers. ## Design requirements ### North stars - **Performance**: Performance for the running CI jobs should be the same if not better than what users currently have. - **Security**: The platform is secure, locked-down, observable, and allows the administrators to react quickly to a security incident. - **Cost**: It should not cost users any more than they are currently spending on running their CI fleet. - **Reliability**: Jobs shouldn't fail because of the underlying infrastructure and GitLab.com it sticks to the SLOs defined. ### Usability - Easy to set up, with a few commands, then allow users to grow depending on the scale - Self-hosted and GitLab.com shared runners should use something similar with some extra security layers for ~Dogfooding - Container-based system to provide maximum portability and allow customization. - Cost-effective on running `gitlab-runner`. - Cost-effective running user job. - Provide an easy upgrade path for the runner fleet. - HA out of the box. - Provide frequent cleanup process/hooks to prevent filling up the disk. - Don't pick up jobs if we don't have resources to run the jobs. ### Security and reliability - Able to stop bitcoin miners in an automated fashion - CPU/Memory Limit - Network/Bandwidth Limit - Each job is isolated both on a process and network-level (depending on multi-tenancy requirements) - Multi-tenant, able to run untrusted code from multiple users. - Based on a locked-down host OS to reduce security footprint. - Not susceptible to noisy neighbors. - Doesn't have a large security footprint when compared to the current solutions. - Observability - Able to tell what binaries/syscall the user is executing - Able to tell what kind of network activity is happening for the user job - Fail closed, any unexpected behavior, failure terminate the job. - Multiple layers of security to have defence in depth. - Have privileged and unprivileged containers in a separate set up since they need different requirements and will also reduce the blast radius if we have to mitigate some issues with privileged containers. - Ability to roll out kernel fixes and security patches in a fast and reliable manner. - Ephemeral `gitlab-runner` managers to prevent long-running processes. ## Docker machine driver replacments | [Docker Machine Drivers](https://docs.docker.com/machine/drivers/) | GitLab Runner Autoscaling Recommendation | |--------------------------------------------------------------------| -----------------------------------------| | Amazon Web Services | [AWS Autos Scaling groups](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) / Kubernetes executor [(EKS)](https://aws.amazon.com/eks/?whats-new-cards.sort-by=item.additionalFields.postDateTime&whats-new-cards.sort-order=desc) | | Microsoft Azure | [Azure virtual machine scale sets](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/overview) / Kubernetes executor [(AKS)](https://docs.microsoft.com/en-us/azure/aks/) | | DigitalOcean | [DigialOcean Kubernetes cluster autoscaling](https://www.digitalocean.com/docs/kubernetes/how-to/autoscale/) with Kubernetes executor | | Exoscale | Not validted [autoscaling](https://www.exoscale.com/syslog/autoscaling-with-grafana-and-prometheus/) feature from cloud provider | | Generic | N/A | | Google Compute Engine | [Autoscaling managed instance groups](https://cloud.google.com/compute/docs/autoscaler/) / Kubernetes executor [(GKE)](https://cloud.google.com/kubernetes-engine/) | | Linode (unofficial plugin, not supported by Docker) | Kubernetes executor [(LKE)](https://www.linode.com/products/kubernetes/) | Microsoft Hyper-V | N/A | | OpenStack | [Autoscaling](https://docs.openstack.org/auto-scaling-sig/latest/theory-of-auto-scaling.html) for example using [Heat](https://docs.openstack.org/senlin/latest/scenarios/autoscaling_heat.html) | Rackspace | N/A | | IBM Softlayer | IBM Cloud [autoscaling](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-about-auto-scale) / Kubernetes executor with [autoscaling](https://cloud.ibm.com/docs/containers?topic=containers-ca) | Oracle VirtualBox | N/A | | VMware vCloud Air | N/A | | VMware Fusion | N/A | | VMware vSphere | [Autoscaling VMs](https://blogs.vmware.com/management/2017/06/configure-auto-scaling-private-cloud.html) / Kubernetes executor [(Tanzu autoscaler)](https://docs.vmware.com/en/VMware-Tanzu-Service-Mesh/services/service-autoscaling-with-tsm-user-guide/GUID-7F848C15-6DC4-466B-996C-BD4356EA4ADE.html) | | VMware Workstation (unofficial plugin, not supported by Docker) | N/A | | Grid 5000 (unofficial plugin, not supported by Docker) | N/A | | Scaleway (unofficial plugin, not supported by Docker) | Kubernetes executor [(Kubernetes autoscaling)](https://blog.scaleway.com/understanding-kubernetes-autoscaling/) | | Hetzner Cloud (unofficial plugin, not supported by Docker) | N/A | | ArvanCloud (unofficial plugin, not supported by Docker) | N/A | ## Competitors CI autoscaling 1. **Jenkins-x**: Creates a pod for each build according to https://jenkins-x.io/docs/reference/components/pod-templates/. This is very similar to our Kubernetes executor 1. **Drone**: They have their own home grown [autoscaler](https://autoscale.drone.io/) with [available source code](https://github.com/drone/autoscaler) which is very similar to what is done with `docker-machine` where it integrates with [each supported cloud provider](https://github.com/drone/autoscaler/tree/v1.7.5/drivers) and creates a VM for the provider which exposes a [Docker daemon](https://github.com/drone/autoscaler/blob/v1.7.5/engine/docker.go#L23-L47) to run CI jobs on. 1. **Buildkite**: Uses AWS ASG to [scale in/out](https://buildkite.com/screencasts/elastic-ci-stack-for-aws) their agents 1. **CircleCI**: Not much information, but reading https://circleci.com/blog/intelligent-ci-cd-with-circleci-building-our-autoscaler/ it seems like they scale up/down custom VMs depending on job queue. 1. **GithHub Actions**: For example, https://github.blog/2020-08-04-github-actions-self-hosted-runners-on-google-cloud/ provides multiple types of autoscaling but all of them revolve in autoscaling their runner to multiple instances and pick up jobs and run that job on that machine. ## DISCLAIMER <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.* <!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION --> This epic will group all issues that are related to implementing solutions for autoscaling the Runner on various platforms and deployment methods.