GitLab 18.6.6 instance executor fatal: unable to get password from user
Summary
Self hosted gitlab-runner 18.6.6 on Ubuntu 24.04. I'm using the instance executor set to one job per instance. I am using the aws-fleeting-plugin and the jobs passed from the Linux runner are executed on ephemeral Windows machines running Windows Server 2025. git-lfs/3.6.1 (GitHub; windows amd64; go 1.23.3)
The last working gitlab-runner version that we are able to use without issue in production is 18.5.0 . I have tested just about every version from 18.6.0 to 18.6.6 and I continue to get the error fatal: unable to get password from user in my job log.
Steps to reproduce
gitlab-runner 18.6.x with instance executor. One job per instance with the aws-fleeting-plugin similar to this. Exact config.toml will be below.
.gitlab-ci.yml
A trivial example which still doesn't work
stages:
- example-stage
example-job:
stage: example-stage
tags:
- example-tag
script:
- |
Write-Output "Hello World!"Actual behavior
Running with gitlab-runner 18.6.6 (xx00xxxx)
on ip-00-000-000-00 xxxxxxxxx, system ID: x_x00xx000xx00
feature flags: FF_USE_POWERSHELL_PATH_RESOLVER:true
Resolving secrets
Preparing the "instance" executor
Preparing instance...
Dialing instance i-000x00xx0000000xx...
Instance i-000x00xx0000000xx connected
Preparing environment
Running on EC2AMAZ-xxxxx0x via
ip-00-000-000-00...
Getting source from Git repository
Gitaly correlation ID: 00xxxxxxxx00xxxxx00xxxxx0x
Fetching changes with git depth set to 20...
Initialized empty Git repository in $path_to_some_build_dir/.git/
Created fresh repository.
fatal: unable to get password from user
Retrying in 5s
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 128Expected behavior
Running with gitlab-runner 18.5.0 (xxx00000)
on ip-00-000-000-00 xxxxxxxxx, system ID: x_x00xx000xx00
feature flags: FF_USE_POWERSHELL_PATH_RESOLVER:true
Resolving secrets
Preparing the "instance" executor
Preparing instance...
Dialing instance i-000x00xx0000000xx...
Instance i-000x00xx0000000xx connected
Preparing environment
Running on EC2AMAZ-xxxxx0x via
i-000x00xx0000000xx...
Getting source from Git repository
Gitaly correlation ID: 00xxxxxxxx00xxxxx00xxxxx0x
Fetching changes with git depth set to 20...
Initialized empty Git repository in $path_to_some_build_dir/.git/
Created fresh repository.
Checking out xxx00000 as detached HEAD (ref is main)...
git-lfs/3.6.1 (GitHub; windows amd64; go 1.23.3; git xx00x00x)
Skipping Git submodules setup
Executing "step_script" stage of the job script
$ Write-Output "Hello World!" # collapsed multi-line command
Cleaning up project directory and file based variables
Job succeededRelevant logs and/or screenshots
job log
Dec 16 23:09:22 [REDACTED] gitlab-runner[6230]: time="2025-12-16T23:09:22.556176813Z" level=debug msg="fetching password data" instance=[REDACTED] name=[REDACTED] region=[REDACTED] runner=[REDACTED] subsystem=fleeting-plugin.fleeting-plugin-aws timestamp="2025-12-16T23:09:22.556Z" try=10
Dec 16 23:10:11 [REDACTED] gitlab-runner[6230]: time="2025-12-16T23:10:11.346920272Z" level=warning msg="Job failed: exit code 128" duration_s=420.342117369 error="exit code 128" exit_code=128 failure_reason= gitlab_user_id=[REDACTED] job=8478737 job-status=failed namespace_id=[REDACTED] organization_id=[REDACTED] project=[REDACTED] project_full_path=[REDACTED] root_namespace_id=[REDACTED] runner=[REDACTED]
Dec 16 23:10:12 [REDACTED] gitlab-runner[6230]: time="2025-12-16T23:10:12.767207284Z" level=warning msg="Submitting job to coordinator... job failed" bytesize=1527 checksum="crc32:[REDACTED]" code=200 correlation_id=[REDACTED] job=8478737 job-status=failed runner=[REDACTED] status="200 OK" update-interval=0s
Dec 16 23:10:12 [REDACTED] gitlab-runner[6230]: time="2025-12-16T23:10:12.767359937Z" level=info msg="Removed job from processing list" builds=0 gitlab_user_id=[REDACTED] job=8478737 max_builds=100 namespace_id=[REDACTED] organization_id=[REDACTED] project=[REDACTED] project_full_path=[REDACTED] queue_depth=1 queue_size=1 repo_url="[REDACTED]" root_namespace_id=[REDACTED] runner=[REDACTED] time_in_queue_seconds=105
Dec 16 23:10:12 [REDACTED] gitlab-runner[6230]: time="2025-12-16T23:10:12.767400405Z" level=info msg="instance marked for removal" instance=[REDACTED] reason="instance reached max use count" runner=[REDACTED] subsystem=taskscaler
Dec 16 23:10:12 [REDACTED] gitlab-runner[6230]: time="2025-12-16T23:10:12.767438116Z" level=debug msg="Failed to process runner" builds=0 error="exit code 128" executor=instance max_builds=100 runner=[REDACTED]Environment description
Customized installation using the instance executor
root@ip-xx-xxx-xxx-xxx:~# gitlab-runner -v
Version: 18.6.6
Git revision: df85dadf
Git branch: 18-6-stable
GO version: go1.24.11 X:cacheprog
Built: 2025-12-09T21:36:46Z4
OS/Arch: linux/amd64config.toml contents
concurrent = ${CONCURRENT}
log_level = "debug"
connection_max_age = "15m0s"
check_interval = 15
log_format = "text"
[session_server]
session_timeout = 1800
[[runners]]
name = "$HOSTNAME"
url = "SELF HOSTED INSTANCE URL" # this is an actual URL in my configuration
token = "$TOKEN"
executor = "instance"
environment = ["FF_USE_POWERSHELL_PATH_RESOLVER=1"]
shell = "powershell"
[runners.feature_flags]
FF_USE_ADAPTIVE_REQUEST_CONCURRENCY = true
[runners.instance]
[runners.autoscaler]
capacity_per_instance = ${CAPACITY_PER_INSTANCE}
max_instances = ${AUTOSCALING_MAX}
max_use_count = 1
plugin = "fleeting-plugin-aws"
[runners.autoscaler.plugin_config]
config_file = "/home/gitlab-runner/.aws/config"
name = "${AUTOSCALING_GROUP_NAME}"
profile = "default"
[runners.autoscaler.connector_config]
os = "windows"
protocol = "winrm"
username = "Administrator"
key_path = "${SOME_PATH_TO_A_KEY_FILE}"
use_static_credentials = false
keepalive = "10m0s"
timeout = "10m0s"
use_external_addr = false
[[runners.autoscaler.policy]]
idle_count = 0
preemptive_mode = false
idle_time = "20m0s"
scale_factor = 0.0
scale_factor_limit = 0Used GitLab Runner version
root@ip-xx-xxx-xxx-xxx:~# gitlab-runner -v
Version: 18.6.6
Git revision: df85dadf
Git branch: 18-6-stable
GO version: go1.24.11 X:cacheprog
Built: 2025-12-09T21:36:46Z4
OS/Arch: linux/amd64Possible fixes
I will link some issues and MRs that I know are related, but haven't had the time to dive into
Edited by Ayub Mohamud