Unable to fix "try setting KUBERNETES_MASTER environment variable" error
I have setup two VMs that I am using throughout my journey of educating myself in CI/CD, GitLab, Kubernetes, Cloud Computing in general and so on. Both VMs have Ubuntu 22.04 Server as a host.
VM1 - MicroK8s Kubernetes cluster (192.168.122.73
)
Most of the setup is "default". Since I'm not really that knowledgeable, I have only configured two pods and their respective services - one with PostGIS and the other one with GeoServer. My intent is to add a third pod, which is the deployment of a app that I a have in VM2 and that will communicate with the GeoServer in order to provide a simple map web service (Leaflet + Django). All pods are exposed both within the cluster via internal IPs as well as externally (externalIp
).
I have also installed two GitLab-related components here:
- GitLab Runner with Kubernetes as executor
- GitLab Kubernetes Agent
In VM2 both are visible as connected. Both have been installed as per GitLab's documentation (AMD64 DEB package for the runner, Helm chart for the agent).
VM2 - GitLab (192.168.122.82
)
Here is where GitLab (default installation, latest version) runs. In the configuration (/etc/gitlab/gitlab.rb
) I have enabled the agent server.
The /etc/gitlab-runner/config.toml
looks as follows:
concurrent = 1
check_interval = 0
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "ginonline-k8s-runner"
url = "http://192.168.122.82/" # <----- VM2
id = 1
token = "@@@@@@@@@@@@@@@@@@@@@q"
token_obtained_at = 2023-02-14T22:27:28Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "python:3.11.2-slim"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[[runners]]
name = "ginonline-k8s-runner"
url = "http://192.168.122.82/" # <----- VM2
id = 2
token = ""@@@@@@@@@@@@@@@@@@@@@"
token_obtained_at = 2023-02-14T23:08:42Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "kubernetes"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.kubernetes]
host = ""
bearer_token_overwrite_allowed = false
image = "python:3.11.2-slim"
namespace = "gitlab-runner"
namespace_overwrite_allowed = ""
node_selector_overwrite_allowed = ""
pod_labels_overwrite_allowed = ""
service_account_overwrite_allowed = ""
pod_annotations_overwrite_allowed = ""
[runners.kubernetes.affinity]
[runners.kubernetes.pod_security_context]
[runners.kubernetes.init_permissions_container_security_context]
[runners.kubernetes.init_permissions_container_security_context.capabilities]
[runners.kubernetes.build_container_security_context]
[runners.kubernetes.build_container_security_context.capabilities]
[runners.kubernetes.helper_container_security_context]
[runners.kubernetes.helper_container_security_context.capabilities]
[runners.kubernetes.service_container_security_context]
[runners.kubernetes.service_container_security_context.capabilities]
[runners.kubernetes.volumes]
[runners.kubernetes.dns_config]
[runners.kubernetes.container_lifecycle]
Initially I had the runner in VM1 configured to have Docker as executor. I had not issues with that. However then I thought it would be nice to try out running the runner inside the cluster so that everything is capsuled (using the internal cluster IPs without further configuration and exposing the VM's operating system).
Both the runner and agent are showing as connected but running a pseudo-CI/CD pipeline (the one provided by GitLab, where you have build, test and deploy stages with each consisting of a simple echo and waiting for a few seconds) returns the following error:
Running with gitlab-runner 15.8.2 (4d1ca121)
on testcluster-k8s-runner Hko2pDKZ, system ID: s_072d6d140cfe
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
ERROR: Preparation failed: getting Kubernetes config: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Will be retried in 3s ...
Using Kubernetes namespace: gitlab-runner
ERROR: Preparation failed: getting Kubernetes config: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Will be retried in 3s ...
Using Kubernetes namespace: gitlab-runner
ERROR: Preparation failed: getting Kubernetes config: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Will be retried in 3s ...
ERROR: Job failed (system failure): getting Kubernetes config: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
I am unable to find any information regarding KUBERNETES_MASTER
except in issue tickets (GitLab) and questions (SO and other Q&A platforms), where people are complaining about this error without any solution. I have no idea what it is, where to set it. My guess would be it belongs in the runner's configuration on VM1 or at least the environment of the gitlab-runner
(the user that contains the runner's userspace with its respective /home/gitlab-runner
directory).
The only one solution I have found so far (and it doesn't work for me) is to create the .kube
directory from the user which uses kubectl
(in my case microk8s kubectl
since I use MicroK8s) to the home
directory of the GitLab runner. I didn't see anything special in this directory (no hidden files) except for a cache subdirectory, hence my decision to simply create it at /home/gitlab-runner/.kube
, which didn't change a thing. I also tried with cache
inside .kube
with the same result.