Unable to connect a kubernetes cluster in 3k GitLab architecture(multiple nodes)

On the 3k architecture of GitLab, we are not able to register kubernetes cluster.

I follow the docs to install Gitlab Agent, but in the step after creating an agent configuration file (.gitlab/agents/my-agent/config.yaml) I have got a failure to register our own kubernetes cluster. I go to Operate > Kubernetes clusters and I could no agents to select. If I click on Connect a Kibernetes cluster via new cluster, a pop-up tells "failed to register an agent" and main cause is related the protocol "GRPC::Unavailable"

In the /etc/gitlab/gitlab.rb on all three gitlab-rails nodes and four sidekiq nodes, I have set

gitlab_kas['enable'] = true

and gitlab-ctl status shows gitlab-kas component is attached(operating), but not working just with a standard configuration properly.

down: gitlab-kas: 1s, normally up, want up; run: log: (pid 1385) 4046256s

We have an external Haproxy node in the frontend network, that is basically introduced by the 3k architecture in GET. I have figured out that I have to configure further in order to enable the agent server on multiple nodes (KAS enable on multiple nodes). I follow this configuration:

gitlab_kas_external_url 'wss://kas.my-gitlab.org/'
gitlab_kas['api_secret_key'] = ''
gitlab_kas['private_api_secret_key'] = ''
gitlab_kas['private_api_listen_address'] = '0.0.0.0:8155'
gitlab_kas['env'] = {
  'no_proxy' => "0,localhost,127.0.0.1",
  'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/", 
  'OWN_PRIVATE_API_URL' => 'grpc://my-gitlab:8155' 
}
gitlab_rails['gitlab_kas_external_url'] = 'wss://my-gitlab.org/-/kubernetes-agent/'
gitlab_rails['gitlab_kas_internal_url'] = 'grpc://localhost:8153'
gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://my-gitlab.org/-/kubernetes-agent/k8s-proxy/'

I have replaced internal api address with localhost rather than address recommended in the docs grpc://kas.internal.gitlab.example.com'

gitlab_rails['gitlab_kas_internal_url'] = 'grpc://localhost:8153'  # or 127.0.0.1:8135

I've found some reports for the no_proxy setup ( #7707), but this approach is not working for us.

gitlab_rails['env'] = { "no_proxy" => "0,127.0.1.1,127.0.0.1,localhost,169.254.169.254" }

Unfortunately none of abvoe approaches help us. gitalb-kas could not be running up with above setting.

Socket approach is a remaining option, but that is not recommended for the multiple nodes as like 3k architecture in the docs page. however, would it be an option somehow?

Where do I have to touch or fix to get a proper setting for gitlab-kas?

From single gitlab omnibus installation, I can register my Kubernetes cluster without any complex configuration. (gitlba_kas['enable'] = true should be enough to have runnnig KAS for single instance case.)