gitlab docker-machine runner fails to remove created ec2 instances

Status update: 2025-02-03

We do not plan to resolve bugs for the docker-machine executor. We are asking customers and users to adopt the new runner autoscaling solution using Fleeting + Taskclaer.

Summary

Runner in docker-machine configuration fails to terminate aws instances after idle timeout

Steps to reproduce

I can personally reproduce this on every run, I'm not sure how I can produce this in a way that you will be able to.

.gitlab-ci.yml

before_script:
  - apt-get update -qq && apt-get install -y -qq sqlite3 libsqlite3-dev nodejs
  - ruby -v
  - which ruby
  - gem install bundler --no-document
  - bundle install --jobs $(nproc)  "${FLAGS[@]}"

rspec:
  script:
    - bundle exec rspec

rubocop:
  script:
    - bundle exec rubocop

Actual behavior

AWS runner is created from docker-machine manager. It goes idle and fails to remove itself

Expected behavior

Machines should be removed after idle timeout

Relevant logs and/or screenshots

--debug run

Successfully removed *runner-name*
ERROR: Problem while reading command output error=READ |0 file already closed

docker-machine ls
no output

sudo ls /root/.docker/machine/machines
*runner-name*

Environment description

docker: 18.09.6 docker-machine: 0.16

config.toml contents

concurrent = 10
check_interval = 0

[[runners]]
  name = "gitlab-aws-autoscaler"
  url = "omitted"
  token = "omitted"
  executor = "docker+machine"
  limit = 10
  [runners.docker]
    image = "alpine:latest"
    privileged = true
    disable_cache = true
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "omitted"
      BucketLocation = "us-east-1"
  [runners.machine]
    IdleCount = 1
    IdleTime = 1800
    MaxBuilds = 100
    OffPeakPeriods = [
      "* * 0-10,20-23 * * mon-fri *",
      "* * * * * sat,sun *"
    ]
    OffPeakIdleCount = 0
    OffPeakIdleTime = 1200
    MachineDriver = "amazonec2"
    MachineName = "gitlab-docker-machine-%s"
    MachineOptions = [
      "amazonec2-region=us-east-1",
      "amazonec2-vpc-id=omitted",
      "amazonec2-subnet-id=omitted",
      "amazonec2-use-private-address=true",
      "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true",
      "amazonec2-security-group=docker-machine-scaler",
      "amazonec2-instance-type=t2.large",
    ]

Used GitLab Runner version

gitlab-runner --version
Version: 10.5.0
Git revision: 10.5.0
Git branch: HEAD
GO version: go1.10
Built: unknown
OS/Arch: linux/amd64

Following documentation here. https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/