Migration from docker+machine on docker-autoscaler problems.

Introduction

With the recent release of the docker-autoscaler with AWS fleeting plugin, we have embarked on a proof of concept to assess its suitability as a replacement for our existing docker+machine executor within the GitLab Runner configuration.

Summary

In general, the new solution works when we tested it on a small scale (several jobs to be performed for individual runners). However, in a higher load situation, we started to observe performance problems or the implementation itself not managing the autoscaling group on AWS in a proper way.

Problems observed from the metrics:

  1. The docker-autoscaler can reduce the desired capacity of the autoscaling group despite having hundreds of pending jobs. This results in only part of the available machine pool being in use and queuing times for jobs becoming unexpectedly long.
  2. The docker-autoscaler can completely saturate the machine's CPU even for relatively large instances (m7a.4xlarge). In the case of docker+machine we could use much smaller machines as controllers.
  3. The docker-autoscaler can scale in its autoscaling group to 0 machines even though it has hundreds of pending tasks in the queue.
  4. If at a certain point in time there appears a large number of jobs, e.g. 200-300 and a small number of machines are still alive as idle, e.g. 15, then the docker-autoscaler is not able to scale up, but tries to carry out tasks with the machines it has available. And to make matters worse, it generally scales down all the time.

Steps to reproduce:

Runner's configuration (executor `docker-autoscaler`) 
concurrent = 100
log_format = "json"
log_level = "debug"
listen_address = ":9252"
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
session_timeout = 1_800

[[runners]]
url = "https://gitlab.our.domain/"
executor = "docker-autoscaler"
token = "<gitlab-runner-token>"

  [runners.custom_build_dir]
  enabled = true

  [runners.docker]
  privileged = false
  disable_cache = true
  runtime = "runc"
  shm_size = 2_147_483_648
  tls_verify = false
  disable_entrypoint_overwrite = false
  oom_kill_disable = false
  network_mtu = 0
  pull_policy = "if-not-present"
  helper_image = "gitlab/gitlab-runner-helper:x86_64-v17.4.2"
  image = "alpine:3.14"

  [runners.cache]
  Type = "s3"
  Shared = true
  MaxUploadedArchiveSize = 0

    [runners.cache.s3]
    ServerAddress = "s3.amazonaws.com"
    BucketLocation = "eu-west-1"
    BucketName = "<bucket-name>"

  [runners.autoscaler]
  capacity_per_instance = 1
  plugin = "aws:1.0.0"
  delete_instances_on_shutdown = true
  instance_ready_command = "cloud-init status --wait"
  max_instances = 200
  max_use_count = 50

    [runners.autoscaler.plugin_config]
    name = "<autoscaling-group-name>"

    [runners.autoscaler.connector_config]
    username = "ubuntu"
    use_external_addr = false
    keepalive = 300

    [[runners.autoscaler.policy]]
    idle_count = 0
    idle_time = "5m"
Example test pipelines 
image: busybox:1.37.0

variables:
  TAG: autoscaler-test-runner

test-many:
  tags: [$TAG]
  rules:
    - if: $CI_PIPELINE_SOURCE == "web"
    - if: $CI_PIPELINE_SOURCE == "api"
    - when: never
  parallel:
    matrix:
      - X: [A, B, C, D, E, F, G, H, I, J]
        Y: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  script:
    - echo "Running test for $X $Y"
    - echo "DONE"

test-few:
  tags: [$TAG]
  rules:
    - if: $CI_PIPELINE_SOURCE == "web"
    - if: $CI_PIPELINE_SOURCE == "api"
    - when: never
  parallel:
    matrix:
      - X: [A, B, C, D, E]
        Y: [1, 2, 3, 4, 5]
  script:
    - echo "Running test for $X $Y"
    - echo "DONE"

First scenario:

  1. Run pipeline with small number of jobs (25)
  2. After all jobs will be finished wait a minute
  3. Run pipelines with large number of jobs (200)
  4. Observe behaviour of desired capacity of related autoscaling group on AWS console.

Actual behaviour

In the scenario described, the number of machines in the ASG will not exceed 25 even for a while and, despite queuing tasks for several minutes, is systematically reduced. The queuing time of the jobs becomes very long.

Screenshot_2024-11-06_154851

Expected behaviour

When new jobs come up for execution (particularly in numbers beyond the capabilities of the gitlab-runner controller), the ASG should be scaled up to the maximum possible number of machines and no scaling down should take place until the last of the pending jobs has been taken for execution by the controller. (This is a current behaviour of docker+machine executor)

Second scenario:

  1. Run a huge number of jobs (1000) when limit of runner is much less (100)
  2. Observe behaviour of desired capacity of related autoscaling group and CPU usage of controller instance on AWS console.

Actual behaviour

In the beginning, when there are a large number of jobs to be done, it happens that the ASG will be scaled up to the maximum number of jobs allowed by the gitlab-runner, but instead of maintaining this state until all the jobs have been done despite the many queued jobs waiting to be completed, the autoscaler is able to scale the ASG down.

Screenshot_2024-11-06_154956

Expected behaviour

When there are a large number of jobs to be done as quickly as possible, the controller should scale the ASG to the maximum number of machines and keep them alive and reused until all the jobs are done. (This is a current behaviour of docker+machine executor)

Reference runner's configuration (executor `docker+machine`) 
concurrent = 100
log_format = "json"
listen_address = ":9252"
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[[runners]]
url = "https://gitlab.our.domain/"
executor = "docker+machine"
token = "<gitlab-runner-token>"
limit = 100
name = "eu-test-long-a"

  [runners.machine]
  MachineOptions = [
  "amazonec2-ssh-user=ubuntu",
  "amazonec2-use-private-address=true",
  "amazonec2-private-address-only=true",
  "amazonec2-retries=25",
  "amazonec2-volume-type=gp2",
  "amazonec2-root-size=48",
  "amazonec2-region=eu-west-1",
  "amazonec2-metadata-token=required",
  "amazonec2-metadata-token-response-hop-limit=1",
  "amazonec2-userdata=/worker-userdata/userdata-noatm",
  "amazonec2-subnet-id=<subnet-id>",
  "amazonec2-zone=a",
  "amazonec2-instance-type=c6a.large"
]
  MachineDriver = "amazonec2"
  MaxBuilds = 50
  IdleCount = 0
  IdleCountMin = 0
  IdleScaleFactor = 0
  IdleTime = 300
  MachineName = "eu-test-long-%s"

  [runners.custom_build_dir]
  enabled = true

  [runners.docker]
  privileged = false
  disable_cache = true
  runtime = "runc"
  shm_size = 2_147_483_648
  tls_verify = false
  disable_entrypoint_overwrite = false
  oom_kill_disable = false
  network_mtu = 0
  image = "alpine:3.14"

  [runners.cache]
  Type = "s3"
  Shared = true
  MaxUploadedArchiveSize = 0

    [runners.cache.s3]
    ServerAddress = "s3.amazonaws.com"
    BucketLocation = "eu-west-1"
    BucketName = "<bucket-name>"

[session_server]
session_timeout = 1_800

Environment description

GitLab: Self-managed premium v17.4.3-ee

Runner: 17.4.2

Fleeting plugin: AWS 1.0.0

Edited by Paweł Bochniak