Gitlab CI does not know how to wait for services
Summary
The documentation for the wait-for-services behavior says:
[...] the Runner:
- checks which ports are exposed from the container by default
- starts a special container that waits for these ports to be accessible
In practice Gitlab CI seems to wait until the first port is available to assume the service running.
Steps to reproduce
Here is the test part of my pipeline which will fail if i dind't add a sleep 120
before the curl command.
Note: In my case I use a derived Gitlab image as service, where I have pre-provisioned example projects. Just in case someone is asking what I am trying to achive there :)
test:
stage: test
image: thomass/curl
services:
- name: gitlab/gitlab-ce:11.0.1-ce.0
alias: gitlab-test
script: curl -fsSl http://gitlab-test:80
Example Project
Here is my project or what I try to achieve:
In the following I show the port behavior of a starting service and what happens in the test job of the example project. For this example I have chosen Gitlab as this service itself. On the port behavior I can explain how Gitlab CI would determine a service running.
First start a Gitlab test instance:
sudo docker run -it --rm gitlab/gitlab-ce:11.0.1-ce.0 bash
Whithin this instance start Gitlab and inspect when ports become available
# start Gitlab
/assets/wrapper &>/dev/null &
# wait for ports become available
while true; do \
< /dev/tcp/127.0.0.1/22 \
< /dev/tcp/127.0.0.1/80 \
< /dev/tcp/127.0.0.1/443 \
echo "" \
sleep 2 \
done
With the (shortened) output I can explain the error prone behavior of gitlab:
# first all three ports are obviously not available
bash: /dev/tcp/127.0.0.1/22: Connection refused
bash: /dev/tcp/127.0.0.1/80: Connection refused
bash: /dev/tcp/127.0.0.1/443: Connection refused
# ... after seconds ..
# the ssh port becomes available - this is where gitlab ci would assume the service to be running
bash: /dev/tcp/127.0.0.1/80: Connection refused
bash: /dev/tcp/127.0.0.1/443: Connection refused
# ... after about a minute ...
# even if Gitlab would wait for all ports (like documented)
# Gitlab would timeout because this port will never become active
bash: /dev/tcp/127.0.0.1/443: Connection refused
What is the current bug behavior?
Gitlab CI assumes the service running as soon as the first port (ssh) becomes available.
What is the expected correct behavior?
Gitlab CI would let me select for which port Gitlab CI should waiting for or if it should wait for the Docker internal healthcheck command.
Possible fixes
test:
stage: test
image: thomass/curl
services:
- name: gitlab/gitlab-ce:11.0.1-ce.0
alias: gitlab-test
wait-for:
- port 80
- docker-healthcheck
script: curl -fsSl http://gitlab-test:80