Direct support for AWS autoscaling groups and spot instances
Status Update (2022-08-10)
The intent of this feature request is being addressed here.
Description
Include problem, use cases, benefits, and/or goals
GitLab Runner can be configured to auto scale runners on Amazon Web Services (AWS) via the docker+machine executor. Using Docker Machine, Docker hosts can be created and removed as needed in AWS backed by on-demand or spot instances.
This configuration provides a cost-efficient way to scale GitLab runners, and generally works well. However, there is currently no way to do things like:
- Specify more than one instance type.
- Specify multiple availability zones or subnets (or conversely, not specify a specific availability zone).
- Fall back to on-demand instances if spot instances are not available.
Since the docker+machine executor doesn't actually interact with AWS and spot instance directly, such features would need to be implemented in Docker Machine. But, Docker Machine is now in maintenance mode, and will no longer incorporate non-bugfix changes.
Proposal
I propose creating a new executor that directly uses EC2 auto scaling groups to instantiate on-demand and spot instances as runners. A sort of docker+ec2asg
executor like the current docker+machine
executor.
Links to related issues and merge requests / references
Please paste a link of the related issues or/and merge requests