Add suspend/resume support to the Docker Autoscaler executor

Summary

Add suspend/resume support to the Docker Autoscaler executor. This is a followup to Add suspend/resume support for instance autosca... (!6570 - merged) which introduced this for Instance Autoscaler executor.

A job can now opt in to preserving its environment when it completes, so a subsequent build resumes onto the preserved state instead of cold-starting from scratch. Useful for agent sessions and human-in- the-loop workflows where recreating the working directory, services, and installed dependencies dominates job runtime.

On suspend, the build container, helper, services, named volumes, and the build network are preserved on the VM; the runner remembers only enough state to route a resumed dispatch back to the same place. On resume, the runner re-attaches to that preserved state and fails if any piece is missing, rather than silently spinning up a fresh empty environment that would lose the suspended job's data.

The feature is gated behind FF_SUSPENDABLE_ENVIRONMENTS; cold-start behavior is unchanged when the flag is off or the job does not opt in.

Testing

1. Build the fleeting-plugin-aws

cd fleeting-plugin-aws
make build

Add to your PATH (or copy to a directory already in PATH):

export PATH="$PWD/out:$PATH"

# Verify
fleeting-plugin-aws --version

2. Build gitlab-runner

cd gitlab-runner
go build -o gitlab-runner ./cmd/gitlab-runner

3. Configure the runner

Create config.toml (Update the token and SANDBOX_PUBLIC_IP):

concurrent = 10
check_interval = 0
connection_max_age = "15m0s"
log_level = "info"
shutdown_timeout = 0

[[runners]]
  name = "vtak-local-suspendable-environments-docker"
  url = "http://gdk.test:3000"
  id = 34
  token = "glrt-REDACTED"
  token_obtained_at = 2026-03-24T08:44:45Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker-autoscaler"
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "alpine:3.23"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0
    extra_hosts = ["gdk.test:SANDBOX_PUBLIC_IP"]
  [runners.autoscaler]
    plugin = "fleeting-plugin-aws"
    capacity_per_instance = 2
    max_instances = 20
    instance_ready_command = "timeout 120 bash -c 'until which step-runner; do sleep 2; done'"
    reservation_throttling = false
    [[runners.autoscaler.policy]]
      preemptive_mode = false
    [runners.autoscaler.plugin_config]
      name          = "fleeting-plugin-aws-asg"
      profile       = "eng-dev-sandbox-vtak"
      region        = "ap-south-1"
      scale_in_termination = true
    [runners.autoscaler.connector_config]
      username = "ubuntu"
      key_path = "/Users/vtak/.ssh/eng-dev-sandbox-vtak-ap-south-1-key-pair.pem"
      use_external_addr = true
      dial_timeout = "2m0s"

4. Network: Autoscaler instances to GDK

If GDK runs locally on your laptop, autoscaler instances (AWS) can't resolve gdk.test. Route through a sandbox relay.

4.1. Autoscaler instances

Add to the ASG launch template user data:

echo "<sandbox-public-ip> gdk.test" >> /etc/hosts

4.2. Sandbox (one-time setup)

Enable remote port forwarding:

grep '^GatewayPorts' /etc/ssh/sshd_config || echo 'GatewayPorts yes' | sudo tee -a /etc/ssh/sshd_config
sudo systemctl restart ssh

Ensure the security group allows inbound TCP 3000 from the autoscaler VPC CIDR.

4.3. Your laptop

Open the tunnel (keep this terminal open):

ssh -R 0.0.0.0:3000:gdk.test:3000 ubuntu@<sandbox-public-ip>

4.4. Verify

From the sandbox:

curl -s -o /dev/null -w '%{http_code}' http://localhost:3000
# Should return 200 or 302

5. Start the runner

./gitlab-runner run --config config.toml

6. One-time Rails console setup

Feature.disable(:duo_runner_restrictions)

7. Trigger a suspend job

user = User.find_by_username("root")
project = Project.find_by_path("gitlab-test")

suspend_d = Ci::Workloads::WorkloadDefinition.new do |d|
  d.image = "alpine:3.23"
  d.add_service("nginx:alpine")
  d.add_service("redis:alpine")
  d.commands = [
    'ls -larth',
    'echo "hello from suspended workload at $(date)" >> "${PWD}/hello.log"',
    'apk add --no-cache redis',
    'wget -qO- http://nginx/ | head -1',
    'redis-cli -h redis SET mykey "hello-from-suspend"',
    'redis-cli -h redis GET mykey',
    'echo "BOTH_SERVICES_OK" >> "${PWD}/hello.log"',
    'cat "${PWD}/hello.log"'
  ]
  d.tags = ["vtak-local-suspendable-environments-docker"]
  d.add_variable("FF_SUSPENDABLE_ENVIRONMENTS", "true")
  d.add_variable("FF_NETWORK_PER_BUILD", "false")
  d.suspend_on_success = true
  d.suspend_on_failure = true
end

suspend_workload = Ci::Workloads::RunWorkloadService.new(
  project: project, current_user: user,
  source: :duo_workflow, workload_definition: suspend_d
).execute

8. Verify build options

suspend_pipeline_id = suspend_workload.payload.pipeline_id
suspend_build = Ci::Pipeline.find(suspend_pipeline_id).builds.first

suspend_build.options
# Should include :suspend_on_success => true

presenter = Ci::BuildRunnerPresenter.new(suspend_build)
presenter.suspend_options
# Should return { suspend_on_success: true, suspend_on_failure: true }

9. Get the environment key

After the job completes, the runner suspends the environment and logs the environment key:

Job environment suspended: 123/s_456/acquisition-key=ABC&build-container-id=DEF&helper-id=GHI&service-ids=JKL%2CMNO

Copy the full environment key value for the next step.

10. Trigger a resume job

Replace the environment_key with the environment key from the runner logs:

resume_d = Ci::Workloads::WorkloadDefinition.new do |d|
  d.image = "dummy"
  d.commands = [
    'ls -larth',
    'apk add --no-cache redis',
    'cat "${PWD}/hello.log"',
    'grep BOTH_SERVICES_OK "${PWD}/hello.log"',
    'wget -qO- http://nginx/ | head -1 && echo "NGINX_OK"',
    'redis-cli -h redis GET mykey && echo "REDIS_OK"'
  ]
  d.tags = ["vtak-local-suspendable-environments-docker"]
  d.add_variable("FF_SUSPENDABLE_ENVIRONMENTS", "true")
  d.add_variable("FF_NETWORK_PER_BUILD", "false")
  d.environment_key = 'UPDATE_ME'
end

resume_workload = Ci::Workloads::RunWorkloadService.new(
  project: project, current_user: user,
  source: :duo_workflow, workload_definition: resume_d
).execute

11. Verify build options

resume_pipeline_id = resume_workload.payload.pipeline_id
resume_build = Ci::Pipeline.find(resume_pipeline_id).builds.first

resume_build.options
# Should include :environment_key => "..."

presenter = Ci::BuildRunnerPresenter.new(resume_build)
presenter.suspend_options
# Should return { suspend_on_success: false, suspend_on_failure: false, environment_key: "X5eakR52n/..." }

Screenshots

NOTE

• Since the runner config has capacity_per_instance set to 2, the first 2 jobs are scheduled on the same instance. The third job is scheduled on a different instance.
• The resumed environment is always on the same instance on which it was suspended on with preserved containers. It inherits the entire filesystem of the suspended job as well.

1. Suspend Job

   

2. Resume Job

   

Related

• Epic: Resumable Jobs for CI and Agent Sessions (gitlab-org#21159)
• Blueprint: Blueprint - Resumable Jobs for CI and Agent Ses... (gitlab#593314 - closed)
• Issue: gitlab-runner: Add suspend/resume support to th... (#39462 - closed)