Skip to content

Gitlab CI does not know how to wait for services

Summary

The documentation for the wait-for-services behavior says:

[...] the Runner:

  1. checks which ports are exposed from the container by default
  2. starts a special container that waits for these ports to be accessible

In practice Gitlab CI seems to wait until the first port is available to assume the service running.

Steps to reproduce

Here is the test part of my pipeline which will fail if i dind't add a sleep 120 before the curl command.

Note: In my case I use a derived Gitlab image as service, where I have pre-provisioned example projects. Just in case someone is asking what I am trying to achive there :)

test:
  stage: test
  image: thomass/curl
  services:
    - name: gitlab/gitlab-ce:11.0.1-ce.0
      alias: gitlab-test
  script: curl -fsSl http://gitlab-test:80

Example Project

Here is my project or what I try to achieve:

https://gitlab.xarif.de/docker/gitlab-testinstance/blob/6def53a37de3349fe34b31c8b1f51f527667883a/.gitlab-ci.yml#L38

In the following I show the port behavior of a starting service and what happens in the test job of the example project. For this example I have chosen Gitlab as this service itself. On the port behavior I can explain how Gitlab CI would determine a service running.

First start a Gitlab test instance:

sudo docker run -it --rm gitlab/gitlab-ce:11.0.1-ce.0 bash

Whithin this instance start Gitlab and inspect when ports become available

# start Gitlab
/assets/wrapper &>/dev/null &
# wait for ports become available
while true; do \
 < /dev/tcp/127.0.0.1/22 \
 < /dev/tcp/127.0.0.1/80 \
 < /dev/tcp/127.0.0.1/443 \
 echo "" \
 sleep 2 \
done

With the (shortened) output I can explain the error prone behavior of gitlab:

# first all three ports are obviously not available
bash: /dev/tcp/127.0.0.1/22: Connection refused
bash: /dev/tcp/127.0.0.1/80: Connection refused
bash: /dev/tcp/127.0.0.1/443: Connection refused

# ... after seconds ..

# the ssh port becomes available - this is where gitlab ci would assume the service to be running
bash: /dev/tcp/127.0.0.1/80: Connection refused
bash: /dev/tcp/127.0.0.1/443: Connection refused

# ... after about a minute ...

# even if Gitlab would wait for all ports (like documented)
# Gitlab would timeout because this port will never become active
bash: /dev/tcp/127.0.0.1/443: Connection refused

What is the current bug behavior?

Gitlab CI assumes the service running as soon as the first port (ssh) becomes available.

What is the expected correct behavior?

Gitlab CI would let me select for which port Gitlab CI should waiting for or if it should wait for the Docker internal healthcheck command.

Possible fixes

test:
  stage: test
  image: thomass/curl
  services:
    - name: gitlab/gitlab-ce:11.0.1-ce.0
      alias: gitlab-test
      wait-for:
        - port 80
        - docker-healthcheck
  script: curl -fsSl http://gitlab-test:80