Gitlab Runner getting Connection timed out error when execute the Job (Only If the host firewall is enabled)
Summary
Hi, I have successfully installed the Gitlab Runner on my rancher kubernetes cluster, using the Gitlab Integration. The runner accepts the Job, but the job fails when the host firewall is on. It only executes the Job successfully when i turn-off firewall. The Gitlab runner executor is kubernetes.
Already opened Ports on firewall: 80, 443, 2376, 6443, 4567, 8006, 8443, 9252 (akl necessary ports are opened). And internet available to the pod...
Steps to reproduce
.gitlab-ci.yml
image: java:8
variables: GRADLE_OPTS: "-Dorg.gradle.daemon=false" CONTAINER_TEST_IMAGE: mydomain.com:$CI_BUILD_REF_NAME CONTAINER_RELEASE_IMAGE: mydomain.com:4567/test/myproject:latest DOCKER_HOST: tcp://localhost:2375
before_script: - chmod +x gradlew
stages:
- assembling
- testing
- building
assembling: stage: assembling script: - ./gradlew -g /cache/.gradle clean assemble allow_failure: false
testing: stage: testing script: - ./gradlew -g /cache/.gradle check
building: stage: building script: - ./gradlew -g /cache/.gradle buildUberJar - ./gradlew -g /cache/.gradle uploadArchives artifacts: untracked: true
Actual behavior
The gitlab-runner is failed to execute and finish the job when the host firewall is on.
Expected behavior
The gitlab-runner need to be successfully execute and finish the job with host firewall is on.
Relevant logs and/or screenshots
job log
Waiting for pod gitlab-managed-apps/runner-whn-hyvl-project-62-concurrent-0zpm7j to be running, status is Pending
Running on runner-whn-hyvl-project-62-concurrent-0zpm7j via runner-gitlab-runner-6d79955494-hwxgg...
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/test/myproject/.git/
Created fresh repository.
From https://gitlab.mydomain.com/test/myproject
* [new ref] refs/pipelines/419 -> refs/pipelines/419
* [new branch] master -> origin/master
Checking out 2fe79d9b as master...
Skipping Git submodules setup
$ chmod +x gradlew
$ ./gradlew -g /cache/.gradle clean assemble
Downloading https://services.gradle.org/distributions/gradle-4.10.3-all.zip
Exception in thread "main" java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at org.gradle.wrapper.Download.downloadInternal(Download.java:66)
at org.gradle.wrapper.Download.download(Download.java:51)
at org.gradle.wrapper.Install$1.call(Install.java:62)
at org.gradle.wrapper.Install$1.call(Install.java:48)
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:69)
at org.gradle.wrapper.Install.createDist(Install.java:48)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:107)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
ERROR: Job failed: command terminated with exit code 1
Environment description
Deployed as an instance cluster in the gitlab, and the runner-whn-hyvl-project-62-concurrent-0zpm7j is resolving to the host and to the internet (I can ping inside the pod).
The Runner 'runner-whn-hyvl-project-62-concurrent-0zpm7j' is resolving to the host, even the firewall is enabled...
I am able ping the host from this runner container...
config.toml contents
listen_address = "[::]:9252"
concurrent = 4
check_interval = 3
log_level = "info"
[session_server]
session_timeout = 1800
[[runners]]
name = "runner-gitlab-runner-6d79955494-hwxgg"
request_concurrency = 1
url = "https://gitlab.mydomain.com/"
token = "my-token"
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = false
image = "ubuntu:16.04"
namespace = "gitlab-managed-apps"
namespace_overwrite_allowed = ""
privileged = true
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.pod_security_context]
[runners.kubernetes.volumes]