Gitlab runner - could not resolve host
## Summary
Hey guys! I'm having a problem with my runner up in Kubernetes and I'm already getting furious for not finding a solution. Sometimes, when a job is started I run into an error right at the beginning:
```bash
fatal: unable to access https://git.exemple.io: Could not resolve host: git.exemple.io
```
## Potential workaround
- Use a recent version of Runner.
- Try adjusting Kubernetes dns config options:
```toml
[[runners.kubernetes.dns_config.options]]
name = "ndots"
value = "1"
```
- If pods take a while for the network to be available, try either:
- Using a `pre_get_sources_script` (https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6644#note_2173852179):
```
pre_get_sources_script = "for i in $(seq 1 5);do getent hosts $CI_SERVER_HOST && break || sleep 1;done"`
```
- Setting the job environment variable `GET_SOURCES_ATTEMPTS` to a greater number than 1. This will retry fetching git sources should there be a failure. (https://docs.gitlab.com/ee/ci/runners/configure_runners.html#job-stages-attempts)
## Steps to reproduce
Implement the catalog via Rancher and apply some customizations in answers.yml of the Helm Chart.
```
---
image: gitlab/gitlab-runner:alpine-v12.1.0
checkInterval: "90"
concurrent: "25"
gitlabUrl: "https://git.exemple.io/"
rbac:
create: "true"
resources:
limits:
cpu: "200m"
memory: "256Mi"
requests:
cpu: "150m"
memory: "128Mi"
runnerRegistrationToken: "********************"
runners:
pollTimeout: "720"
privileged: "true"
requestConcurrency: "30"
runUntagged: "true"
tags: "devops-prod"
image: "ubuntu:18.04"
cloneUrl: "https://git.exemple.io"
builds:
cpuLimit: "400m"
memoryLimit: "512Mi"
cpuRequests: "400m"
memoryRequests: "512Mi"
services:
cpuLimit: "400m"
memoryLimit: "512Mi"
cpuRequests: "400m"
memoryRequests: "512Mi"
terminationGracePeriodSeconds: "12000"
```
<!--
Please add the definition of the job from `.gitlab-ci.yml` that is failing
inside of the code blocks (```) below.
-->
<details>
<summary> .gitlab-ci.yml </summary>
```yml
# =============================================================================
# INCLUDE TEMPLATES
# =============================================================================
include:
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-audit.yml"
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-base.yml"
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-build.yml"
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-dast.yml"
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-deploy.yml"
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-global.yml"
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-scanning.yml"
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-sonarqube.yml"
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-web-performance.yml"
- project: "shared/devops/pipeline"
file: "/ci/.gitlab-ci-rules.yml"
# =============================================================================
# DEFAULT IMAGE
# =============================================================================
image: docker:19.03.0
# =============================================================================
# SERVICES
# =============================================================================
services:
- docker:19.03.0-dind
# =============================================================================
# STAGES
# =============================================================================
stages: [ reports, build, scanning, deploy, web-performance, dast ]
# =============================================================================
# WORKFLOW RULES
# =============================================================================
workflow:
rules:
- if: $PIPELINE_DISABLE == "true"
when: never
- if: $CI_MERGE_REQUEST_ID
when: never
- when: always
# =============================================================================
# GLOBAL VARIABLES
# =============================================================================
variables:
DOCKER_HOST: tcp://localhost:2375
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
DOCKER_BUILDKIT: 1
GIT_DEPTH: 1
GIT_SUBMODULE_STRATEGY: none
GIT_STRATEGY: none
```
</details>
## Actual behavior
The job breaks.
## Expected behavior
The job should be successful.
## Relevant logs and/or screenshots
<!--
Paste the job logs inside of the code blocks (```) below so it would be
easier to read.
-->
<details>
<summary> job log </summary>
```sh
1 Running with gitlab-runner 12.7.1 (003fe500)
2 on gitlab-runner-all-develop-gitlab-runner-ff99c489d-7br7w B7HYCxmV
3
Using Kubernetes namespace: gitlab-runner-all-develop
00:00
4 Using Kubernetes executor with image docker:19.03.0 ...
6
Waiting for pod gitlab-runner-all-develop/runner-b7hycxmv-project-80-concurrent-08vcld to be running, status is Pending
00:03
7 Running on runner-b7hycxmv-project-80-concurrent-08vcld via gitlab-runner-all-develop-gitlab-runner-ff99c489d-7br7w...
9
Fetching changes with git depth set to 1...
00:06
10 Initialized empty Git repository in /builds/ads/signo/api/cache/.git/
11 Created fresh repository.
12 fatal: unable to access 'https://git.stefanini.io/ads/signo/api/cache.git/': Could not resolve host: git.stefanini.io
15 ERROR: Job failed: command terminated with exit code 1
```
</details>
## Environment description
My gitlab server is in Kubernetes as is my gitlab runner. Yes, we are in a customized environment.
<!--
Please add the contents of `config.toml` inside of the code blocks (```)
below, remember to remove any secret tokens!
-->
<details>
<summary> config.toml contents </summary>
```toml
listen_address = "[::]:9252"
concurrent = 25
check_interval = 90
log_level = "info"
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-runner-all-develop-gitlab-runner-ff99c489d-7br7w"
output_limit = 4096
request_concurrency = 30
url = "https://git.exemple.io/"
token = "*********************"
executor = "kubernetes"
clone_url = "https://git.exemple.io"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = false
image = "ubuntu:18.04"
namespace = "gitlab-runner-all-develop"
namespace_overwrite_allowed = ""
privileged = true
cpu_limit = "400m"
memory_limit = "512Mi"
service_cpu_limit = "400m"
service_memory_limit = "512Mi"
cpu_request = "400m"
memory_request = "512Mi"
service_cpu_request = "400m"
service_memory_request = "512Mi"
poll_timeout = 720
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.pod_security_context]
[runners.kubernetes.volumes]
```
</details>
issue