Kubernetes Runner TCP establishment timeouts when building docker images

Summary

When building a docker image with the integrated Kubernetes (k3s) Cluster Runner deployed with the Kubernetes integration by GitLab, it sometimes happens that TCP/UDP timeouts just happen and the build process aborts.

Steps to reproduce

Deploy a k3s cluster (without traefik) with a node which has a public IP address and another node which is in a NAT network. The master has to be directly connected over a second interface to the NAT. Integrate this cluster in GitLab and install a GitLab Runner on it. Create a project with a Dockerfile and activate Auto Devops.

.gitlab-ci.yml

Auto DevOps was used.

Actual behaviour

The docker image build process crashes due to TCP connection establishment fail or unknown DNS host error.

Expected behaviour

The docker image build process is running successfully.

Relevant logs and/or screenshots

job log

[0KRunning with gitlab-runner 12.8.0 (1b659122)
[0;m[0K  on runner-gitlab-runner-8647bb9795-xs24r stFWxQ4t
[0;msection_start:1586798706:prepare_executor
[0K[0KUsing Kubernetes namespace: gitlab-managed-apps
[0;m[0KUsing Kubernetes executor with image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image/master:stable ...
[0;msection_end:1586798706:prepare_executor
[0Ksection_start:1586798706:prepare_script
[0KWaiting for pod gitlab-managed-apps/runner-stfwxq4t-project-54-concurrent-08zjkf to be running, status is Pending
Running on runner-stfwxq4t-project-54-concurrent-08zjkf via runner-gitlab-runner-8647bb9795-xs24r...
section_end:1586798709:prepare_script
[0Ksection_start:1586798709:get_sources
[0K[32;1mFetching changes with git depth set to 50...[0;m
Initialized empty Git repository in /builds/siewert-systems/xxxx/bootserver-documentation/.git/
[32;1mCreated fresh repository.[0;m
From https://xxxx/siewert-systems/xxxx/bootserver-documentation
 * [new ref]         refs/pipelines/216 -> refs/pipelines/216
 * [new branch]      master             -> origin/master
[32;1mChecking out 0bb32813 as master...[0;m

[32;1mSkipping Git submodules setup[0;m
section_end:1586798710:get_sources
[0Ksection_start:1586798710:restore_cache
[0Ksection_end:1586798710:restore_cache
[0Ksection_start:1586798710:download_artifacts
[0Ksection_end:1586798711:download_artifacts
[0Ksection_start:1586798711:build_script
[0K[32;1m$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command[0;m
[32;1m$ /build/build.sh[0;m
Logging to GitLab Container Registry with CI credentials...
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
Building Dockerfile-based application...
Error response from daemon: manifest for xxxx/siewert-systems/xxxx/bootserver-documentation/master:0000000000000000000000000000000000000000 not found: manifest unknown: manifest unknown
Error response from daemon: manifest for xxxx/siewert-systems/xxxx/bootserver-documentation/master:latest not found: manifest unknown: manifest unknown
Sending build context to Docker daemon  69.63kB

Step 1/11 : FROM ruby:2.6.5
2.6.5: Pulling from library/ruby
f15005b0235f: Pulling fs layer
41ebfd3d2fd0: Pulling fs layer
b998346ba308: Pulling fs layer
f01ec562c947: Pulling fs layer
2447a2c11907: Pulling fs layer
1915e6344d7f: Pulling fs layer
6ea7f301ab45: Pulling fs layer
0c79fcaf0109: Pulling fs layer
2447a2c11907: Waiting
1915e6344d7f: Waiting
0c79fcaf0109: Waiting
6ea7f301ab45: Waiting
f01ec562c947: Waiting
41ebfd3d2fd0: Verifying Checksum
41ebfd3d2fd0: Download complete
b998346ba308: Verifying Checksum
b998346ba308: Download complete
f01ec562c947: Verifying Checksum
f01ec562c947: Download complete
1915e6344d7f: Verifying Checksum
1915e6344d7f: Download complete
f15005b0235f: Download complete
0c79fcaf0109: Download complete
6ea7f301ab45: Verifying Checksum
6ea7f301ab45: Download complete
2447a2c11907: Verifying Checksum
f15005b0235f: Pull complete
41ebfd3d2fd0: Pull complete
b998346ba308: Pull complete
f01ec562c947: Pull complete
2447a2c11907: Pull complete
1915e6344d7f: Pull complete
6ea7f301ab45: Pull complete
0c79fcaf0109: Pull complete
Digest: sha256:651078e89471c30567685dce4caa321adf1f846b353e05c327b55d76a84acc50
Status: Downloaded newer image for ruby:2.6.5
 ---> ad10dfbc638b
Step 2/11 : RUN apt-get update && apt-get install -y nodejs && apt-get clean && rm -rf /var/lib/apt/lists/*
 ---> Running in 9cf87401ef25
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://security.debian.org/debian-security buster/updates/main amd64 Packages [187 kB]
Get:3 http://deb.debian.org/debian buster InRelease [122 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [7380 B]
Fetched 8337 kB in 46s (179 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libc-ares2 libnode64 libuv1 nodejs-doc
Suggested packages:
  npm
The following NEW packages will be installed:
  libc-ares2 libnode64 libuv1 nodejs nodejs-doc
0 upgraded, 5 newly installed, 0 to remove and 5 not upgraded.
Need to get 6753 kB of archives.
After this operation, 30.4 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 libc-ares2 amd64 1.14.0-1 [85.8 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libuv1 amd64 1.24.1-1 [110 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libnode64 amd64 10.15.2~dfsg-2 [5521 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 nodejs amd64 10.15.2~dfsg-2 [86.2 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 nodejs-doc all 10.15.2~dfsg-2 [950 kB]
[91mdebconf: delaying package configuration, since apt-utils is not installed
[0mFetched 6753 kB in 0s (21.7 MB/s)
Selecting previously unselected package libc-ares2:amd64.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 23968 files and directories currently installed.)
Preparing to unpack .../libc-ares2_1.14.0-1_amd64.deb ...
Unpacking libc-ares2:amd64 (1.14.0-1) ...
Selecting previously unselected package libuv1:amd64.
Preparing to unpack .../libuv1_1.24.1-1_amd64.deb ...
Unpacking libuv1:amd64 (1.24.1-1) ...
Selecting previously unselected package libnode64:amd64.
Preparing to unpack .../libnode64_10.15.2~dfsg-2_amd64.deb ...
Unpacking libnode64:amd64 (10.15.2~dfsg-2) ...
Selecting previously unselected package nodejs.
Preparing to unpack .../nodejs_10.15.2~dfsg-2_amd64.deb ...
Unpacking nodejs (10.15.2~dfsg-2) ...
Selecting previously unselected package nodejs-doc.
Preparing to unpack .../nodejs-doc_10.15.2~dfsg-2_all.deb ...
Unpacking nodejs-doc (10.15.2~dfsg-2) ...
Setting up libc-ares2:amd64 (1.14.0-1) ...
Setting up libuv1:amd64 (1.24.1-1) ...
Setting up libnode64:amd64 (10.15.2~dfsg-2) ...
Setting up nodejs-doc (10.15.2~dfsg-2) ...
Setting up nodejs (10.15.2~dfsg-2) ...
update-alternatives: using /usr/bin/nodejs to provide /usr/bin/js (js) in auto mode
Processing triggers for libc-bin (2.28-10) ...
Removing intermediate container 9cf87401ef25
 ---> 209b6c95960e
Step 3/11 : COPY ./Gemfile /usr/src/app/
 ---> 9ef8b5c10d92
Step 4/11 : COPY ./Gemfile.lock /usr/src/app/
 ---> 1a567a28af79
Step 5/11 : WORKDIR /usr/src/app
 ---> Running in ffaab652e54f
Removing intermediate container ffaab652e54f
 ---> ab73e1ff0cae
Step 6/11 : RUN gem install bundler --version '2.0.2'
 ---> Running in 212a70f884aa
Successfully installed bundler-2.0.2
1 gem installed
Removing intermediate container 212a70f884aa
 ---> 41f799328b1f
Step 7/11 : RUN bundle install
 ---> Running in fb43a811c887
Fetching gem metadata from https://rubygems.org/.
[91mRetrying dependency api due to error (2/4): Bundler::HTTPError Network error while fetching https://index.rubygems.org/api/v1/dependencies?gems=activesupport%2Caddressable%2Cautoprefixer-rails%2Cbackports%2Ccoffee-script%2Ccoffee-script-source%2Cconcurrent-ruby%2Ccontracts%2Cdotenv%2Cerubis%2Cexecjs%2Cfast_blank%2Cfastimage%2Cffi%2Chaml%2Chamster%2Chashie%2Ci18n%2Ckramdown%2Clisten%2Cmemoist%2Cmiddleman%2Cmiddleman-autoprefixer%2Cmiddleman-cli%2Cmiddleman-core%2Cmiddleman-sprockets%2Cmiddleman-syntax%2Cmini_portile2%2Cminitest%2Cnokogiri%2Cpadrino-helpers%2Cpadrino-support%2Cparallel%2Cpublic_suffix%2Crack%2Crake%2Crb-fsevent%2Crb-inotify%2Credcarpet%2Crouge%2Csass-listen%2Csassc%2Cservolux%2Csprockets%2Ctemple%2Cthor%2Cthread_safe%2Ctilt%2Ctzinfo%2Cuglifier (too many connection resets (due to Net::ReadTimeout with #<TCPSocket:(closed)> - Net::ReadTimeout) after 1 requests on 46972847544920, last used 20.064525591 seconds ago)
[0m............
Using rake 12.3.2
Fetching concurrent-ruby 1.1.5
Installing concurrent-ruby 1.1.5
Fetching i18n 0.9.5

[91mRetrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/i18n-0.9.5.gem)
[0mInstalling i18n 0.9.5
Using minitest 5.11.3
Fetching thread_safe 0.3.6
Installing thread_safe 0.3.6
Fetching tzinfo 1.2.5
Installing tzinfo 1.2.5
Fetching activesupport 5.0.7.2
Installing activesupport 5.0.7.2
Fetching public_suffix 3.1.0
Installing public_suffix 3.1.0
Fetching addressable 2.6.0
Installing addressable 2.6.0
Fetching execjs 2.7.0
Installing execjs 2.7.0
Fetching autoprefixer-rails 9.5.1.1
Installing autoprefixer-rails 9.5.1.1
Fetching backports 3.15.0

[91mRetrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/backports-3.15.0.gem)
[0mInstalling backports 3.15.0
Using bundler 2.0.2
Fetching coffee-script-source 1.12.2
Installing coffee-script-source 1.12.2
Fetching coffee-script 2.4.1
Installing coffee-script 2.4.1
Fetching contracts 0.13.0

[91mRetrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/contracts-0.13.0.gem)
[0mInstalling contracts 0.13.0
Fetching dotenv 2.7.2
Installing dotenv 2.7.2
Fetching erubis 2.7.0

[91mRetrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/erubis-2.7.0.gem)
[0mInstalling erubis 2.7.0
Fetching fast_blank 1.0.0
Installing fast_blank 1.0.0 with native extensions
Fetching fastimage 2.1.5
Installing fastimage 2.1.5
Fetching ffi 1.11.1

[91mRetrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/ffi-1.11.1.gem)
[0mInstalling ffi 1.11.1 with native extensions
Fetching temple 0.8.1
Installing temple 0.8.1
Fetching tilt 2.0.9
[91mRetrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/tilt-2.0.9.gem)
[0m
Installing tilt 2.0.9
Fetching haml 5.1.1

[91mRetrying download gem from https://rubygems.org/ due to error (2/4): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/haml-5.1.1.gem)
[0m
[91mRetrying download gem from https://rubygems.org/ due to error (3/4): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/haml-5.1.1.gem)
[0m
[91mRetrying download gem from https://rubygems.org/ due to error (4/4): Gem::RemoteFetcher::UnknownHostError timed out (https://rubygems.org/gems/haml-5.1.1.gem)
[0m
[91mGem::RemoteFetcher::UnknownHostError: timed out
(https://rubygems.org/gems/haml-5.1.1.gem)
An error occurred while installing haml (5.1.1), and Bundler cannot continue.
Make sure that `gem install haml -v '5.1.1' --source 'https://rubygems.org/'`
succeeds before bundling.

In Gemfile:
  middleman was resolved to 4.3.4, which depends on
    haml
[0mThe command '/bin/sh -c bundle install' returned a non-zero code: 5
section_end:1586799515:build_script
[0Ksection_start:1586799515:after_script
[0Ksection_end:1586799516:after_script
[0Ksection_start:1586799516:upload_artifacts_on_failure
[0Ksection_end:1586799516:upload_artifacts_on_failure
[0K[31;1mERROR: Job failed: command terminated with exit code 1
[0;m

Environment description

One node as k8s master (k3s master) with an public IP address where e.g. the ingress controller exposes, plus two nodes in a NAT network which are directly connected to the master (to NAT).

Used GitLab Runner version

Running with gitlab-runner 12.8.0 (1b659122)