E2E test CNG orchestrator cannot connect to registries when using Rancher Desktop

E2E test CNG orchestrator cannot connect to registries to pull images when using Rancher Desktop. The goal here would be to get the orchestrator to work on MacOS somehow, it doesn't really matter if it is Rancher Desktop or another open source solution.

Kind clusters fail to resolve external domains (like registry.gitlab.com) when running on Rancher Desktop on macOS, causing container image pulls to timeout. Root Cause: Rancher Desktop on macOS runs in a VM, creating multiple networking layers (macOS → Rancher Desktop VM → Kind cluster → containers) where DNS resolution gets lost or times out. The issue doesn't occur on Linux with Docker.

Key Error:

dial tcp: lookup registry.gitlab.com on 192.168.5.2:53: read udp 172.19.0.2:54620->192.168.5.2:53: i/o timeout

╰─❯ bundle exec orchestrator create deployment kind --host-ssh-port 2222                                                                                                         ─╯
Creating cluster 'gitlab'
[✔] performing cluster creation ... done
Creating cluster "gitlab" ...
 • Ensuring node image (kindest/node:v1.32.0) 🖼  ...
 ✓ Ensuring node image (kindest/node:v1.32.0) 🖼
 • Preparing nodes 📦   ...
 ✓ Preparing nodes 📦
 • Writing configuration 📜  ...
 ✓ Writing configuration 📜
 • Starting control-plane 🕹️  ...
 ✓ Starting control-plane 🕹️
 • Installing CNI 🔌  ...
 ✓ Installing CNI 🔌
 • Installing StorageClass 💾  ...
 ✓ Installing StorageClass 💾
 • Waiting ≤ 30s for control-plane = Ready ⏳  ...
 ✓ Waiting ≤ 30s for control-plane = Ready ⏳
 • Ready after 16s 💚
Set kubectl context to "kind-gitlab"
You can now use your cluster with:

kubectl cluster-info --context kind-gitlab

Thanks for using kind! 😊
[✖] installing metrics server ... failed
Adding helm chart 'https://kubernetes-sigs.github.io/metrics-server/'
helm chart repo already exists, updating
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "metrics-server" chart repository
Update Complete. ⎈Happy Helming!⎈
Upgrading helm release 'metrics-server' in namespace 'kube-system'

=== block 'installing metrics server' error ===
Command 'helm upgrade --install metrics-server metrics-server/metrics-server --namespace kube-system --timeout 1m --values - --wait --atomic --version ^3.12' failed!
Release "metrics-server" does not exist. Installing it now.
Error: release metrics-server failed, and has been uninstalled due to atomic being set: context deadline exceeded
=== block 'installing metrics server' error ===
Cluster 'gitlab' created
Creating CNG deployment 'gitlab'
[✔] running pre-deployment setup ... done
Adding helm chart 'https://charts.gitlab.io'
helm chart repo already exists, updating
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "gitlab" chart repository
Update Complete. ⎈Happy Helming!⎈
Creating namespace 'gitlab'
namespace/gitlab created
Creating gitlab license secret
`QA_EE_LICENSE|EE_LICENSE` variable is not set, skipping
Creating admin user initial password secret
secret/gitlab-initial-root-password created
Creating pre-receive hook
configmap/pre-receive-hook created
[✖] running helm deployment ... failed
Upgrading helm release 'gitlab' in namespace 'gitlab'
Helm deployment failed!
For more information on troubleshooting failures, see: 'https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/gems/gitlab-orchestrator?ref_type=heads#troubleshooting'
Following events of Warning type present in cluster:
[
  {
    "type": "Warning",
    "reason": "Failed",
    "name": "Pod/gitlab-shared-secrets-20c3865-m8zx5",
    "message": "Failed to pull image \"registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.0.2\": failed to pull and unpack image \"registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.0.2\": failed to resolve reference \"registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.0.2\": failed to do request: Head \"https://registry.gitlab.com/v2/gitlab-org/build/cng/kubectl/manifests/v18.0.2\": dial tcp: lookup registry.gitlab.com on 192.168.5.2:53: read udp 172.19.0.2:54620->192.168.5.2:53: i/o timeout"
  },
  {
    "type": "Warning",
    "reason": "Failed",
    "name": "Pod/gitlab-shared-secrets-20c3865-m8zx5",
    "message": "Failed to pull image \"registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.0.2\": failed to pull and unpack image \"registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.0.2\": failed to resolve reference \"registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.0.2\": failed to do request: Head \"https://registry.gitlab.com/v2/gitlab-org/build/cng/kubectl/manifests/v18.0.2\": dial tcp: lookup registry.gitlab.com on 192.168.5.2:53: read udp 172.19.0.2:33219->192.168.5.2:53: i/o timeout"
  },
  {
    "type": "Warning",
    "reason": "Failed",
    "name": "Pod/gitlab-shared-secrets-20c3865-m8zx5",
    "message": "Error: ErrImagePull"
  },
  {
    "type": "Warning",
    "reason": "Failed",
    "name": "Pod/gitlab-shared-secrets-20c3865-m8zx5",
    "message": "Failed to pull image \"registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.0.2\": failed to pull and unpack image \"registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.0.2\": failed to resolve reference \"registry.gitlab.com/gitlab-org/build/cng/kubectl:v18.0.2\": failed to do request: Head \"https://registry.gitlab.com/v2/gitlab-org/build/cng/kubectl/manifests/v18.0.2\": dial tcp: lookup registry.gitlab.com on 192.168.5.2:53: server misbehaving"
  },
  {
    "type": "Warning",
    "reason": "Failed",
    "name": "Pod/gitlab-shared-secrets-20c3865-m8zx5",
    "message": "Error: ImagePullBackOff"
  }
]

I suspect it's related to the /etc/resolv.conf file

kubectl -n kube-system edit configmap coredns:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30 {
           disable success cluster.local
           disable denial cluster.local
        }
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2025-06-17T01:27:16Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "253"
  uid: 7e52a139-ba7a-46d9-9f66-2c2da171c558

Potential workaround: use Colima (yet to be verified) https://docs.gitlab.com/charts/development/kind/#installing-dependencies

Note: Docker Engine - Community can’t be used on mac without docker desktop which requires a licence: https://docs.docker.com/engine/#licensing , that is why i am not using Docker Desktop

Edited by Jay McCure