Docker service health check may test wrong port if service exposes multiple ports
Summary
When linking a Docker service container which exposes multiple ports (for example using the official MySQL images), the health check seems to test the wrong port, and reports a timeout.
Steps to reproduce
I created a small demo repository.
.gitlab-ci.yml
stages:
- test
job:
stage: test
services:
- mysql:5.7.25
variables:
MYSQL_ROOT_PASSWORD: totallysecurepassword
script: echo "OK"
Actual behavior
The job reports a warning in the job output:
Waiting for services to be up and running...
*** WARNING: Service runner-ed2dce3a-project-11796843-concurrent-0-mysql-0 probably didn't start properly.
Health check error:
service "runner-ed2dce3a-project-11796843-concurrent-0-mysql-0-wait-for-service" timeout
Afterwards the job succeeds.
Expected behavior
No warning should be shown.
Relevant logs and/or screenshots
The full job output can be found here.
Environment description
Using the shared Runner on GitLab.com, with the Docker executor.
Used GitLab Runner version
Running with gitlab-runner 11.9.0-rc2 (227934c0)
on docker-auto-scale ed2dce3a
Thougts
As far as I can see, the health check reads the IP address and the port of the service from the environment variables (see source).
The official MySQL images (in version 5.7 and 8.0) expose multiple ports, namely 3306 and 33060 (see Dockerfile for 5.7 or Dockerfile for 8.0).
This will create the following environment inside the job's container (given the service container name is mysql
):
MYSQL_ENV_GOSU_VERSION=1.7
MYSQL_ENV_MYSQL_MAJOR=5.7
MYSQL_ENV_MYSQL_ROOT_PASSWORD=...
MYSQL_ENV_MYSQL_VERSION=5.7.25-1debian9
MYSQL_NAME=/serene_ptolemy/mysql
MYSQL_PORT=tcp://172.17.0.2:3306
MYSQL_PORT_33060_TCP=tcp://172.17.0.2:33060
MYSQL_PORT_33060_TCP_ADDR=172.17.0.2
MYSQL_PORT_33060_TCP_PORT=33060
MYSQL_PORT_33060_TCP_PROTO=tcp
MYSQL_PORT_3306_TCP=tcp://172.17.0.2:3306
MYSQL_PORT_3306_TCP_ADDR=172.17.0.2
MYSQL_PORT_3306_TCP_PORT=3306
MYSQL_PORT_3306_TCP_PROTO=tcp
So to my understanding the health check script will take the service IP address from MYSQL_PORT_33060_TCP_ADDR
and the port from MYSQL_PORT_33060_TCP_PORT
, and try to establish a TCP connection.
Unfortunately this is the wrong port: while MySQL EXPOSES
s the two ports, the MySQL service is actually is actually bound to the port 3306.
Ideas
Is it possible to check which ports are actually "in use" by the service container?
Or can we add a configuration to the service definition to define the port to be checked? Something like this:
job:
stage: test
services:
- name: mysql:5.7.25
health_check_port: 3306
Workaround
We created an own Docker image built "on top" of the official MySQL image, to expose only one port.
The Dockerfile to build this custom MySQL image for version 5.7 looks like this:
FROM mysql:5.7 as upstream
FROM scratch
COPY --from=upstream / /
ENV GOSU_VERSION 1.7
ENV MYSQL_MAJOR 5.7
ENV MYSQL_VERSION 5.7.25-1debian9
VOLUME /var/lib/mysql
ENTRYPOINT ["docker-entrypoint.sh"]
# only expose port 3306, but not 33060,
# as the upstream Dockerfile does
EXPOSE 3306
CMD ["mysqld"]
After building this image and pushing it to our GitLab registry, we can use it in our GitLab CI jobs like this:
job:
services:
- name: <our registry server here>/docker/mysql-ci:5.7
alias: mysql