Skip to content

GitLab Runner 12.10 - network per build - healtheck for services time out

Summary

I’m having trouble with healthcheck for services in GitLab CI when using Network per build 1, I have a bunch of services which start just fine and work as expected, but the healthcheck always end in timeout, which adds significant time to build duration. This was working previously when not using the Network per Build feature. I have all the appropriate ports exposed in my Dockerfiles.

You can see repo with reproducible example here

Pipeline with FF_NETWORK_PER_BUILD = 1 is here - there are warnings

Pipeline with no FF_NETWORK_PER_BUILD is here - there are no warnings

Steps to reproduce

  • Create .gitlab-ci.yml
  • Set FF_NETWORK_PER_BUILD flag to 1
  • Define some services
  • Run the pipeline
  • See warning about service timeout even if services did start correctly and have exposed ports
.gitlab-ci.yml 
variables:
  FF_NETWORK_PER_BUILD: 1

stages:
  - test-services

test services:
  stage: test-services
  services:
    - name: registry.gitlab.com/benke.lukas/network-per-build-services-bug/apache:1.0
      alias: apache
    - name: registry.gitlab.com/benke.lukas/network-per-build-services-bug/php-fpm:1.0
      alias: php-fpm
  script:
    - echo "running"

Actual behavior

There is warning about service timeout

Expected behavior

There should be no warning because services did not time out and have ports exposed

Relevant logs and/or screenshots

job log 
Using Docker executor with image ruby:2.5 ...
 Starting service registry.gitlab.com/benke.lukas/network-per-build-services-bug/apache:1.0 ...
 Authenticating with credentials from job payload (GitLab Registry)
 Pulling docker image registry.gitlab.com/benke.lukas/network-per-build-services-bug/apache:1.0 ...
 Using docker image sha256:338898637748ba6e4b671d7d886aecf7ea1b87e762a49b1d726f0f507adf6288 for registry.gitlab.com/benke.lukas/network-per-build-services-bug/apache:1.0 ...
 Starting service registry.gitlab.com/benke.lukas/network-per-build-services-bug/php-fpm:1.0 ...
 Authenticating with credentials from job payload (GitLab Registry)
 Pulling docker image registry.gitlab.com/benke.lukas/network-per-build-services-bug/php-fpm:1.0 ...
 Using docker image sha256:06d675fcecb89f4d9d4b67e5d297ae439c3a20f20241e7696b5a48116b82443d for registry.gitlab.com/benke.lukas/network-per-build-services-bug/php-fpm:1.0 ...
 Waiting for services to be up and running...
 *** WARNING: Service runner-0277ea0f-project-18758248-concurrent-0-de1098fb1beb825f-registry.gitlab.com__benke.lukas__network-per-build-services-bug__php-fpm-1 probably didn't start properly.
 Health check error:
 service "runner-0277ea0f-project-18758248-concurrent-0-de1098fb1beb825f-registry.gitlab.com__benke.lukas__network-per-build-services-bug__php-fpm-1-wait-for-service" timeout
 Health check container logs:
 Service container logs:
 2020-05-13T15:25:34.123799659Z [13-May-2020 15:25:34] NOTICE: fpm is running, pid 1
 2020-05-13T15:25:34.124263435Z [13-May-2020 15:25:34] NOTICE: ready to handle connections
 *********
 *** WARNING: Service runner-0277ea0f-project-18758248-concurrent-0-de1098fb1beb825f-registry.gitlab.com__benke.lukas__network-per-build-services-bug__apache-0 probably didn't start properly.
 Health check error:
 service "runner-0277ea0f-project-18758248-concurrent-0-de1098fb1beb825f-registry.gitlab.com__benke.lukas__network-per-build-services-bug__apache-0-wait-for-service" timeout
 Health check container logs:
 Service container logs:
 2020-05-13T15:25:30.096389342Z AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.18.0.2. Set the 'ServerName' directive globally to suppress this message
 2020-05-13T15:25:30.098185714Z AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.18.0.2. Set the 'ServerName' directive globally to suppress this message
 2020-05-13T15:25:30.100540737Z [Wed May 13 15:25:30.100439 2020] [mpm_event:notice] [pid 1:tid 140183277788488] AH00489: Apache/2.4.43 (Unix) configured -- resuming normal operations
 2020-05-13T15:25:30.100619092Z [Wed May 13 15:25:30.100571 2020] [core:notice] [pid 1:tid 140183277788488] AH00094: Command line: 'httpd -D FOREGROUND'
 *********
 Pulling docker image ruby:2.5 ...
 Using docker image sha256:a98425292e846f475eabbae2d96922efe869b66538fa84efaaaef7b5125bd86a for ruby:2.5 ...

Environment description

GitLab.com but happens exactly the same on private install

config.toml contents 
I do not have this available

Used GitLab Runner version

On shared runners:

 Running with gitlab-runner 12.10.0-rc2 (6c8c540f)
   on docker-auto-scale 0277ea0f

On private runners there is 12.10

Possible fixes

Unfortunately I have no idea :)