gitlab.com kubernetes integration CI runner can't contact cluster

Summary

kubernetes cluster that is integrated with gitlab.com, with one click install gitlab-runner. the gitlab runner then can't contact the ip of the cluster API

Steps to reproduce

create a on premise cluster with metallb+calico+kubeadm, join the cluster to gitlab.com, then run a CICD task that deploy to a environment (How one can reproduce the issue - this is very important)

Example Project

https://gitlab.com/impertrix-solutions/urlshortener

What is the current bug behavior?

contact http://2.2.2.2:6443 io timeout, can't contact kubernetes cluster

What is the expected correct behavior?

successfully deploying the manifest to the namespace

Relevant logs and/or screenshots

none really, just can't ping when execed into container