Healthchecks fail with Docker 29
Summary
After upgrading our runner hosts to Docker 29, health checks on at least mysql/mariadb containers (and probably others) constantly fail with results in intermitting job failures based on whether the dependency was fully started soon enough.
Steps to reproduce
Upgrade host to Docker 29.
.gitlab-ci.yml
doctrine-mapping-check:
stage: tests
image: "<custom php image>"
interruptible: true
services:
- "mysql:${MYSQL_VERSION}"
needs:
- install-dependencies
cache:
<<: *composer_cache
before_script:
- "sed -i \"s/DATABASE_URL=.*/DATABASE_URL=mysql:\\/\\/root:$MYSQL_ROOT_PASSWORD@mysql:3306\\/db/g\" .env" # Set database
- "sed -i \"s/APP_SECRET=.*/APP_SECRET=SuperSecret/g\" .env" # Set app secret
script:
- php bin/console doctrine:schema:validate --skip-sync
- php bin/console doctrine:migrations:migrate
- php bin/console doctrine:schema:update --dump-sql
- php bin/console doctrine:schema:validate --skip-mapping
tags:
- drenso-docker
rules:
- if: $CI_PIPELINE_SOURCE != "schedule" && $CI_COMMIT_REF_NAME != "production"
Actual behavior
Waiting for services to be up and running (timeout 60 seconds)...
*** WARNING: Service runner-khy4ww1er-project-29-concurrent-5-2b3753e9b131dc3b-mysql-0 probably didn't start properly.
Health check error:
service "runner-khy4ww1er-project-29-concurrent-5-2b3753e9b131dc3b-mysql-0-wait-for-service" health check: exit code 1
Health check container logs:
2025-11-21T16:00:40.688568793Z FATAL: No HOST or PORT found
Service container logs:
2025-11-21T16:00:39.356688222Z 2025-11-21 16:00:39+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4 started.
2025-11-21T16:00:39.790756049Z 2025-11-21 16:00:39+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2025-11-21T16:00:39.805380351Z 2025-11-21 16:00:39+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4 started.
2025-11-21T16:00:40.223058194Z 2025-11-21 16:00:40+00:00 [Note] [Entrypoint]: Initializing database files
2025-11-21T16:00:40.253343120Z 2025-11-21T16:00:40.248029Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2025-11-21T16:00:40.253370749Z 2025-11-21T16:00:40.250329Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.4.7) initializing of server in progress as process 80
2025-11-21T16:00:40.503784786Z 2025-11-21T16:00:40.503523Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-11-21T16:00:57.774753389Z 2025-11-21T16:00:57.774519Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
*********
The error is shown in red in the Gitlab job log UI.
2025-11-21T16:00:40.688568793Z FATAL: No HOST or PORT found
Expected behavior
The health check should complete correctly. Downgrading to Docker 28 solves the issue and removes the error shown above.
Environment description
Self managed runner on an up-to-date Debian Trixie host.
config.toml contents
concurrent = 8
check_interval = 0
user = "gitlab-runner"
connection_max_age = "15m0s"
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-runner"
url = "https://gitlab.drenso.dev"
id = 0
token = "<snip>"
token_obtained_at = 0001-01-01T00:00:00Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
environment = ["COMPOSER_CACHE_DIR=/cache/composer", "COMPOSER_HTACCESS_PROTECT=0", "YARN_CACHE_FOLDER=/cache/yarn"]
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/opt/gitlab-runner/cache:/cache", "/opt/gitlab-runner/builds:/builds"]
pull_policy = ["if-not-present"]
shm_size = 0
network_mtu = 0
Used GitLab Runner version
bobv@gitlab-runner:~$ gitlab-runner --version
Version: 18.6.1
Git revision: b5e9c6d0
Git branch: 18-6-stable
GO version: go1.24.6 X:cacheprog
Built: 2025-11-21T06:34:41Z
OS/Arch: linux/amd64
Possible fixes
I'm going to reference #39129 (closed) here as it is related to Docker 29, but I'm not seeing a relevant ticket for this particular issue yet.