## Summary When using the `docker-autoscaler` executor with the AWS fleeting plugin, **automatic runner token expiration** (configured via **Admin > Settings > CI/CD > Runners > Runners expiration time**), a new authentication token is fetched upon expiration. This causes the gitlab-runner manager to appear as a **second logical runner** (new token). This new runner treats existing instances as "pre-existing" and prunes them (or marks them as exceeding `idle_time`) while jobs are still running, resulting in **SSH tunnel EOF errors** and **job system failures**. This bug was observed and troubleshoot'd with AWS and AutoScaling groups but it probably other Cloud vendors. ## Steps to reproduce 1. Configure a GitLab Runner manager with: - Executor: `docker-autoscaler` - Fleeting AWS plugin managing one or more ASGs 2. In GitLab **Admin > Settings > CI/CD > Runners**, set a non-empty **Runners expiration time** value so that runner authentication tokens are automatically reset by GitLab on a fixed cadence. 3. Run CI pipelines that: - Use the `docker-autoscaler` runner - Have jobs long enough that some instances remain running across the token reset boundary 5. Wait for the next automatic runner token reset event or manually reset the token via `gitlab-runner reset-token`. Around that time, observe: - Runner logs showing `Resetting runner authentication token... succeeded` for the existing runner ID - Shortly after, logs showing a **new runner ID** initialized on the same manager host, for the same ASG(s) - The new runner discovering existing instances as `preexisted` and marking them for removal with: - `err=no data on pre-existing instance so removing for safety`, or - `reason=instance exceeded max idle time` 6. Correlate with CI job logs that were executing on those instances at that time and see SSH tunnel EOF errors on the job log. ## Expected behavior VM's should be allowed to finish the jobs before being gracefully shutdown. Also, the logs **should not** list the VM removal reason as exceeding `idle_time`. ## Relevant logs and/or screenshots ### Example Job Logs ```text Post "http://internal.tunnel.invalid/v1.44/images/create?fromImage=registry.gitlab.com%2Fgitlab-org%2Fgitlab-runner%2Fgitlab-runner-helper&tag=x86_64-v18.7.2": ssh tunnel: EOF (%!s(<nil>)) (manager.go:237:0s) ERROR: Failed to remove container b019ad... ERROR: Failed to remove container f2c81e... ERROR: Failed to remove container 75cdf0...377 ERROR: Failed to cleanup volumes ERROR: Job failed (system failure): waiting for container: error during connect: Post "http://internal.tunnel.invalid/v1.44/containers/b019ad...e48/wait?condition=not-running": ssh tunnel: EOF (%!s(<nil>)) ``` These failures occur while the jobs are still within their normal runtime and **shorter than the configured `idle_time`**. ### Runner Logs Showing Instance Termination Mid-Job Example where an instance is pruned for `max idle time` while jobs are running: ```text # Instance initially healthy and used for jobs ... instance discovery ... ... instance is ready instance=<instance-a> runner=<runner-id-1> subsystem=taskscaler ... Mar 27 14:47:54 <runner-manager-host-2> gitlab-runner[23717]: health success failure-count=0 instance=<instance-a> runner=<runner-id-1> subsystem=taskscaler # Shortly after, a different runner ID marks the same instance for removal Mar 27 14:49:01 <runner-manager-host-2> gitlab-runner[23717]: instance marked for removal instance=<instance-a> reason=instance exceeded max idle time runner=<runner-id-2> subsystem=taskscaler Mar 27 14:49:43 <runner-manager-host-2> gitlab-runner[23717]: instance pruned group=<ASG_NAME_2> id=<instance-a> runner=<runner-id-2> subsystem=taskscaler ``` Correlated CI job logs for that instance show the SSH tunnel EOF pattern described above. ### Runner Logs Showing Two Logical Runners Managing the Same ASG Example: ASG `<ASG_NAME_2>` with runners `COMgS5nb1` and `CiRsZ_JSU`: ```text # Runner COMgS5nb1 running jobs on <instance-b> Mar 27 14:47:17 <runner-manager-host-2> gitlab-runner[23717]: Dialing instance <instance-b>... job=734937 runner=COMgS5nb1 ... Mar 27 14:47:18 ...: Instance <instance-b> connected job=734909 runner=COMgS5nb1 ... Mar 27 14:48:28 ...: Instance <instance-b> connected job=734938 runner=COMgS5nb1 ... # Runner CiRsZ_JSU pruning the same instance Mar 27 14:49:01 <runner-manager-host-2> gitlab-runner[23717]: instance marked for removal instance=<instance-b> reason=instance exceeded max idle time runner=CiRsZ_JSU subsystem=taskscaler Mar 27 14:49:43 <runner-manager-host-2> gitlab-runner[23717]: instance pruned group=<ASG_NAME_2> id=<instance-b> runner=CiRsZ_JSU subsystem=taskscaler ``` Example: ASG `<ASG_NAME_1>` with runners `NQ6CvQ8vD` and `x_RJQWSsa`: ```text # Runner NQ6CvQ8vD actively using <instance-c> Mar 27 14:41:12 <runner-manager-host-1> gitlab-runner[24416]: health success failure-count=0 instance=<instance-c> runner=NQ6CvQ8vD subsystem=taskscaler Mar 27 14:41:12 ...: Instance <instance-c> connected job=<job-id-1> runner=NQ6CvQ8vD ... Mar 27 14:42:55 ...: Instance <instance-c> connected job=<job-id-2> runner=NQ6CvQ8vD ... # Second runner for same ASG discovers and removes it as "preexisting" Mar 27 14:43:12 ...: instance discovery cause=preexisted group=<ASG_NAME_1> id=<instance-c> runner=x_RJQWSsa state=running subsystem=taskscaler Mar 27 14:43:12 ...: ERROR: ready up preparation failed err=no data on pre-existing instance so removing for safety instance=<instance-c> runner=x_RJQWSsa subsystem=taskscaler Mar 27 14:43:12 ...: instance marked for removal instance=<instance-c> reason=instance ready up preparation failed runner=x_RJQWSsa subsystem=taskscaler Mar 27 14:43:57 ...: instance pruned group=<ASG_NAME_1> id=<instance-c> runner=NQ6CvQ8vD subsystem=taskscaler ERROR: instance unexpectedly removed instance=<instance-c> runner=NQ6CvQ8vD ... ``` ### Runner Logs Showing Periodic Token Reset and New Runner Creation Example (manager `<runner-manager-host-3>`, logical runners `rekcBkU97` and `1woPYW8do`): ```text # First runner active Apr 01 19:19:50 <runner-manager-host-3> gitlab-runner[22141]: instance rate limiting burst=10 limit=100 runner=rekcBkU97 subsystem=taskscaler # ~18 hours later: token reset and second runner appears Apr 02 13:19:48 <runner-manager-host-3> gitlab-runner[22141]: Resetting runner authentication token... succeeded correlation_id=01KN75K6KKYDZ6GQ9T1CTSP1EH runner=rekcBkU97 ... Apr 02 13:19:50 <runner-manager-host-3> gitlab-runner[22141]: instance rate limiting burst=10 limit=100 runner=1woPYW8do subsystem=taskscaler Apr 02 13:19:50 <runner-manager-host-3> gitlab-runner[22141]: plugin initialized ... runner=1woPYW8do ... version=v1.2.0 Apr 02 13:19:50 <runner-manager-host-3> gitlab-runner[22141]: instance discovery cause=preexisted group=<ASG_NAME_1> id=<instance-d> runner=1woPYW8do state=running ... Apr 02 13:19:50 <runner-manager-host-3> gitlab-runner[22141]: ERROR: ready up preparation failed err=no data on pre-existing instance so removing for safety instance=<instance-d> runner=1woPYW8do ... Apr 02 13:19:50 ...: instance marked for removal instance=<instance-d> reason=instance ready up preparation failed runner=1woPYW8do ... Apr 02 13:19:57 ...: ERROR: out-of-sync capacity actual=0 expected=1 name=<ASG_NAME_1> region=us-east-2 runner=rekcBkU97 subsystem=fleeting-plugin.plugin ``` ## Environment description - GitLab: `18.8.5-ee` (self-managed, Omnibus) - GitLab Runner: `18.7.2` - Executor: `docker-autoscaler` - Fleeting plugin: AWS - AWS: - EC2 Auto Scaling Groups (ASGs) - Scale-in protection **enabled** - Auto-scaling **disabled** (`min_size = 0`, `desired_capacity = 0`) - Instances: On-demand EC2 (no spot instances) ## Workaround **Clear/unset** the **Runners expiration time** value, so that runner authentication tokens are **not** automatically reset by GitLab, but this has security concerns.