Skip to content

Kubernetes executor docker build networking problem

Summary

Using Helm, kubernetes executor, and the AutoDevOps. When attempting to build a docker container any network requests will time out/stall. It appears that the networking in the build container is failing to find the bridge.

time="2018-11-05T22:37:20.993144913Z" level=info msg="Loading containers: start."
time="2018-11-05T22:37:21.013845539Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'\nbridge                146976  1 br_netfilter\nstp                    12976  1 bridge\nllc                    14552  2 bridge,stp\nip: can't find device 'br_netfilter'\nbr_netfilter           22256  0 \nbridge                146976  1 br_netfilter\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"
time="2018-11-05T22:37:21.024768705Z" level=warning msg="Running modprobe nf_nat failed with message: `ip: can't find device 'nf_nat'\nnf_nat_ipv6            14131  1 ip6table_nat\nnf_nat_masquerade_ipv4    13412  1 ipt_MASQUERADE\nnf_nat_ipv4            14115  1 iptable_nat\nnf_nat                 26787  4 nf_nat_ipv6,xt_nat,nf_nat_masquerade_ipv4,nf_nat_ipv4\nnf_conntrack          133053  9 nf_conntrack_ipv6,nf_nat_ipv6,nf_conntrack_netlink,ip_vs,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat_ipv4,xt_conntrack,nf_nat\nlibcrc32c              12644  4 ip_vs,nf_nat,nf_conntrack,xfs\nmodprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1"
time="2018-11-05T22:37:21.035582270Z" level=warning msg="Running modprobe xt_conntrack failed with message: `ip: can't find device 'xt_conntrack'\nxt_conntrack           12760  5 \nnf_conntrack          133053  9 nf_conntrack_ipv6,nf_nat_ipv6,nf_conntrack_netlink,ip_vs,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat_ipv4,xt_conntrack,nf_nat\nmodprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1"
time="2018-11-05T22:37:21.122102131Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"

Reproduce / Behavior

Reproduced both with .gitlab-ci.yml and the autodevops templates. (Simplified)

build:
  script: 
    - docker build .

Output shows timeout/stalling of network requests. (seen on wget/apk/apt-get)

Step 4/12 : RUN wget http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz 
---> Running in ee6860282c30 
Connecting to dl-cdn.alpinelinux.org (151.101.200.249:80) 
APKINDEX.tar.gz 0% | | 0 --:--:-- ETA 
APKINDEX.tar.gz 0% | | 0 --:--:-- ETA

Workaround

Successfully able to complete builds by adding the network to the build command --network host

build:
  script: 
    - docker build --network host .

This requires a .gitlab-ci.yml and excludes autodevops.

Relevant logs and/or screenshots

Full output from the container logs: output.txt

Used GitLab Runner version

11.3.6-ee

Seems related to other reported kubernetes/networking issues: https://github.com/gliderlabs/docker-alpine/issues/307

Internal ZD: https://gitlab.zendesk.com/agent/tickets/107364

Possible fixes

We can set hostNetworking to true when we create the pod on kubernetes, as https://github.com/gliderlabs/docker-alpine/issues/307#issuecomment-427514462 but I am not sure what are the implications if there are any. Looking at the comments it specifies that we need specify the ports as well, which there might be no way of knowing which port we need to update, unless it's just my lack of knowledge on how it work.

Edited by Steve Xuereb