Autoscaling GitLab CI on AWS CodeBuild
Please support the following workflow (It's almost exactly like Autoscaling GitLab CI on AWS Fargate @ https://docs.gitlab.com/runner/configuration/runner_autoscale_aws_fargate/) but we're swapping AWS Fargate for AWS CodeBuild.
Here are the big wins with this approach
- AWS CodeBuild supports Privileged mode which lets you run DIND (docker in docker)
- AWS CodeBuild has a lot of overrides, with the most important one being
timeoutInMinutesOverride
so we can have the build killed when it exceeds the pipelines timeout in gitlab (I had a gitlab ci-container fargate container running for 1 month :^( ) - AWS CodeBuild has native support for windows (no implicit EC2 provisioning required like Fargate)
- We can run our docker containers w/ 72 vCPU and 145GB of memory (build.general1.2xlarge)
The only downside is there's limited build compute types (https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-compute-types.html) and they cost A LOT more (E.x.: linux general1.large is $0.02 in CodeBuild and the Fargate equivalent is $0.0065)
I also came across https://about.gitlab.com/blog/2020/07/31/aws-fargate-codebuild-build-containers-gitlab-runner/ and I don't like it because it's a double hop that we're being double billed for (paying for both a fargate container and a aws code build project...)
What I'm proposing is you create a "CodeBuild" driver for gitlab-runner and it will do the following
- start a build using the project defined in the TOML with the following build inputs (see https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codebuild.html#CodeBuild.Client.start_build)
-
privilegedModeOverride set to True -
artifactsOverride will be set to NO_ARTIFACTS -
serviceRoleOverride will come from the runner's TOML config file (Make this optional) -
sourceTypeOverride will be set to NO_SOURCE -
buildspecOverride will be in inline build spec that's created at runtime. It will do the same thing that AWS Fargate container does... create a SSH server to connect to the ci-cordinator and run whatever the SSH client sends. (NOTE: keep in mind that this could be a windows container) -
projectName will come from the runner's TOML config file -
environmentVariablesOverride will be set at runtime by the gitlab-runner. -
timeoutInMinutesOverride will be set at runtime by gitlab-runner based on the gitlab CI/CD pipeline's timeout -
environmentTypeOverrideset, imageOverride, computeTypeOverride will come from the runner's TOML config file
- Wait for the build to complete (basically keep calling BatchGetBuilds at some interval until the build status is no longer IN_PROGRESS).
- Succeed or fail based on the build status
The VPC settings cannot be overridden, so we need to set that up when creating the AWS CodeBuild project.