Docker service health check may test wrong port if service exposes multiple ports
When linking a Docker service container which exposes multiple ports (for example using the official MySQL images), the health check seems to test the wrong port, and reports a timeout.
Steps to reproduce
I created a small demo repository.
stages: - test job: stage: test services: - mysql:5.7.25 variables: MYSQL_ROOT_PASSWORD: totallysecurepassword script: echo "OK"
The job reports a warning in the job output:
Waiting for services to be up and running... *** WARNING: Service runner-ed2dce3a-project-11796843-concurrent-0-mysql-0 probably didn't start properly. Health check error: service "runner-ed2dce3a-project-11796843-concurrent-0-mysql-0-wait-for-service" timeout
Afterwards the job succeeds.
No warning should be shown.
Relevant logs and/or screenshots
The full job output can be found here.
Using the shared Runner on GitLab.com, with the Docker executor.
Used GitLab Runner version
Running with gitlab-runner 11.9.0-rc2 (227934c0) on docker-auto-scale ed2dce3a
As far as I can see, the health check reads the IP address and the port of the service from the environment variables (see source).
This will create the following environment inside the job's container (given the service container name is
MYSQL_ENV_GOSU_VERSION=1.7 MYSQL_ENV_MYSQL_MAJOR=5.7 MYSQL_ENV_MYSQL_ROOT_PASSWORD=... MYSQL_ENV_MYSQL_VERSION=5.7.25-1debian9 MYSQL_NAME=/serene_ptolemy/mysql MYSQL_PORT=tcp://172.17.0.2:3306 MYSQL_PORT_33060_TCP=tcp://172.17.0.2:33060 MYSQL_PORT_33060_TCP_ADDR=172.17.0.2 MYSQL_PORT_33060_TCP_PORT=33060 MYSQL_PORT_33060_TCP_PROTO=tcp MYSQL_PORT_3306_TCP=tcp://172.17.0.2:3306 MYSQL_PORT_3306_TCP_ADDR=172.17.0.2 MYSQL_PORT_3306_TCP_PORT=3306 MYSQL_PORT_3306_TCP_PROTO=tcp
So to my understanding the health check script will take the service IP address from
MYSQL_PORT_33060_TCP_ADDR and the port from
MYSQL_PORT_33060_TCP_PORT, and try to establish a TCP connection.
Unfortunately this is the wrong port: while MySQL
EXPOSESs the two ports, the MySQL service is actually is actually bound to the port 3306.
Is it possible to check which ports are actually "in use" by the service container?
Or can we add a configuration to the service definition to define the port to be checked? Something like this:
job: stage: test services: - name: mysql:5.7.25 health_check_port: 3306
We created an own Docker image built "on top" of the official MySQL image, to expose only one port.
The Dockerfile to build this custom MySQL image for version 5.7 looks like this:
FROM mysql:5.7 as upstream FROM scratch COPY --from=upstream / / ENV GOSU_VERSION 1.7 ENV MYSQL_MAJOR 5.7 ENV MYSQL_VERSION 5.7.25-1debian9 VOLUME /var/lib/mysql ENTRYPOINT ["docker-entrypoint.sh"] # only expose port 3306, but not 33060, # as the upstream Dockerfile does EXPOSE 3306 CMD ["mysqld"]
After building this image and pushing it to our GitLab registry, we can use it in our GitLab CI jobs like this:
job: services: - name: <our registry server here>/docker/mysql-ci:5.7 alias: mysql