Gitlab runner lookup fh-gitlab.company.com on 10.96.0.10:53: no such host

Hi, I tried to use the gitlab (Community Edition 13.1.0; omnibus install) managed helm to install gitlab runner (gitlab/gitlab-runner:alpine-v13.0.1) on kubernetes (v1.18.0).

But somehow the runner-gitlab-runner deployment is printing below errors:

WARNING: Checking for jobs... failed runner=eG9znkY1 status=couldn't execute POST against https://fh-gitlab.mycompany.com/api/v4/jobs/request: Post https://fh-gitlab.mycompany.com/api/v4/jobs/request: dial tcp: lookup fh-gitlab.mycompany.com on 10.96.0.10:53: no such host

I tried to dig fh-gitlab.mycompany.com on both host machine and dnsutil pod (https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#create-a-simple-pod-to-use-as-a-test-environment) on gitlab-managed-apps namespace, both are getting through:

# on host machine

# dig @10.96.0.10 fh-gitlab.mycompany.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> @10.96.0.10 fh-gitlab.mycompany.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<< opcode: QUERY, status: NXDOMAIN, id: 22966
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;fh-gitlab.mycompany.com.	IN	A

;; AUTHORITY SECTION:
mycompany.com.		3	IN	SOA	usfhktpxyp01v.mycompany.com. root.mycompany.com. 1 86400 3600 604800 3600

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Thu Nov 12 19:29:11 UTC 2020
;; MSG SIZE  rcvd: 150

# on dnsutil pod

# cat /etc/resolv.conf 
nameserver 10.96.0.10
search gitlab-managed-apps.svc.cluster.local svc.cluster.local cluster.local mycompany.com
options ndots:5

# dig fh-gitlab.mycompany.com

; <<>> DiG 9.11.6-P1 <<>> fh-gitlab.mycompany.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<< opcode: QUERY, status: NXDOMAIN, id: 61825
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 1234a629cbc06ec1 (echoed)
;; QUESTION SECTION:
;fh-gitlab.mycompany.com.	IN	A

;; AUTHORITY SECTION:
mycompany.com.		7	IN	SOA	usfhktpxyp01v.mycompany.com. root.mycompany.com. 1 86400 3600 604800 3600

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Thu Nov 12 19:26:07 UTC 2020
;; MSG SIZE  rcvd: 162

What is happening here? Why is the runner reporting dns resolve issue when the record is clearly there?