GitLab Runner 12.10 - network per build - healtheck for services time out
Summary
I’m having trouble with healthcheck for services in GitLab CI when using Network per build 1, I have a bunch of services which start just fine and work as expected, but the healthcheck always end in timeout, which adds significant time to build duration. This was working previously when not using the Network per Build feature. I have all the appropriate ports exposed in my Dockerfiles.
You can see repo with reproducible example here
Pipeline with FF_NETWORK_PER_BUILD = 1 is here - there are warnings
Pipeline with no FF_NETWORK_PER_BUILD is here - there are no warnings
Steps to reproduce
- Create .gitlab-ci.yml
- Set FF_NETWORK_PER_BUILD flag to 1
- Define some services
- Run the pipeline
- See warning about service timeout even if services did start correctly and have exposed ports
.gitlab-ci.yml
variables:
FF_NETWORK_PER_BUILD: 1
stages:
- test-services
test services:
stage: test-services
services:
- name: registry.gitlab.com/benke.lukas/network-per-build-services-bug/apache:1.0
alias: apache
- name: registry.gitlab.com/benke.lukas/network-per-build-services-bug/php-fpm:1.0
alias: php-fpm
script:
- echo "running"
Actual behavior
There is warning about service timeout
Expected behavior
There should be no warning because services did not time out and have ports exposed
Relevant logs and/or screenshots
job log
Using Docker executor with image ruby:2.5 ...
Starting service registry.gitlab.com/benke.lukas/network-per-build-services-bug/apache:1.0 ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/benke.lukas/network-per-build-services-bug/apache:1.0 ...
Using docker image sha256:338898637748ba6e4b671d7d886aecf7ea1b87e762a49b1d726f0f507adf6288 for registry.gitlab.com/benke.lukas/network-per-build-services-bug/apache:1.0 ...
Starting service registry.gitlab.com/benke.lukas/network-per-build-services-bug/php-fpm:1.0 ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/benke.lukas/network-per-build-services-bug/php-fpm:1.0 ...
Using docker image sha256:06d675fcecb89f4d9d4b67e5d297ae439c3a20f20241e7696b5a48116b82443d for registry.gitlab.com/benke.lukas/network-per-build-services-bug/php-fpm:1.0 ...
Waiting for services to be up and running...
*** WARNING: Service runner-0277ea0f-project-18758248-concurrent-0-de1098fb1beb825f-registry.gitlab.com__benke.lukas__network-per-build-services-bug__php-fpm-1 probably didn't start properly.
Health check error:
service "runner-0277ea0f-project-18758248-concurrent-0-de1098fb1beb825f-registry.gitlab.com__benke.lukas__network-per-build-services-bug__php-fpm-1-wait-for-service" timeout
Health check container logs:
Service container logs:
2020-05-13T15:25:34.123799659Z [13-May-2020 15:25:34] NOTICE: fpm is running, pid 1
2020-05-13T15:25:34.124263435Z [13-May-2020 15:25:34] NOTICE: ready to handle connections
*********
*** WARNING: Service runner-0277ea0f-project-18758248-concurrent-0-de1098fb1beb825f-registry.gitlab.com__benke.lukas__network-per-build-services-bug__apache-0 probably didn't start properly.
Health check error:
service "runner-0277ea0f-project-18758248-concurrent-0-de1098fb1beb825f-registry.gitlab.com__benke.lukas__network-per-build-services-bug__apache-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2020-05-13T15:25:30.096389342Z AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.18.0.2. Set the 'ServerName' directive globally to suppress this message
2020-05-13T15:25:30.098185714Z AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.18.0.2. Set the 'ServerName' directive globally to suppress this message
2020-05-13T15:25:30.100540737Z [Wed May 13 15:25:30.100439 2020] [mpm_event:notice] [pid 1:tid 140183277788488] AH00489: Apache/2.4.43 (Unix) configured -- resuming normal operations
2020-05-13T15:25:30.100619092Z [Wed May 13 15:25:30.100571 2020] [core:notice] [pid 1:tid 140183277788488] AH00094: Command line: 'httpd -D FOREGROUND'
*********
Pulling docker image ruby:2.5 ...
Using docker image sha256:a98425292e846f475eabbae2d96922efe869b66538fa84efaaaef7b5125bd86a for ruby:2.5 ...
Environment description
GitLab.com but happens exactly the same on private install
config.toml contents
I do not have this available
Used GitLab Runner version
On shared runners:
Running with gitlab-runner 12.10.0-rc2 (6c8c540f)
on docker-auto-scale 0277ea0f
On private runners there is 12.10
Possible fixes
Unfortunately I have no idea :)