management cluster deployment fails due to cluster-reachable failure
Summary
Hi all, not sure if this is a bug or more like a config issue. I'm trying to test Sylva on OpenStack, so I created a VM with the specs written in the Installation chapter and then followed the entire section to get a management cluster on Docker using kind. So, the steps executed so far are:
- Clone the repository
- Create the docker kind network
if ! docker network inspect kind > /dev/null 2>&1; then
echo "Docker network 'kind' doesn't exist. Creating the network..."
docker network create kind
fi
# Export docker network "kind" adress
KIND_PREFIX=$(docker network inspect kind -f '{{ (index .IPAM.Config 0).Subnet }}')
CLUSTER_IP=$(echo $KIND_PREFIX | awk -F"." '{print $1"."$2"."$3".100"}')
echo $CLUSTER_IP
- Prepare CAPD deployment values
cp -r environment-values/kubeadm-capd/ environment-values/my-kubeadm-capd
vi environment-values/my-kubeadm-capd/values.yaml
# file content
---
cluster:
k8s_version: v1.27.3
capi_providers:
infra_provider: capd
bootstrap_provider: cabpk
# CAPD only supports 1 CP machine
control_plane_replicas: 1
capd_docker_host: unix:///var/run/docker.sock # dynamically replaced in CI
cluster_virtual_ip: 172.18.0.100
proxies:
# put your own proxy settings here if you need
http_proxy: http://<proxy_ip>:3128/
https_proxy: http://<proxy_ip>:3128/
no_proxy: localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
- Added the execution rights to all the
.sh
scripts:
find . -iname "*.sh" 2>/dev/null | xargs -tn1 chmod +x
- Executed the bootstrap script:
./bootstrap.sh environment-values/my-kubeadm-capd
But somehow the bootstrap script fails because of the cluster-reachable kustomization, I assume it's not getting authenticated towards the management cluster APIs.
Warning ReconciliationFailed 86s (x8 over 4m57s) kustomize-controller ConfigMap/default/dummy-deps-cluster-reachable dry-run failed: failed to get API group resources: unable to retrieve the complete list of server APIs: v1: Get "https://172.18.0.3:6443/api/v1?timeout=30s": Forbidden
I took the kubeconfig from the secret management-cluster-kubeconfig
, extracted both certificate and key from there and executed the follwing curl that is working from both VM and Bootstrap container node:
sylva@sylva-mgmt-server:~/sylva-core$ curl -k --cert mgmt-cert.crt --key mgmt-key.key https://172.18.0.3:6443/api/v1?timeout=30s
{
"kind": "APIResourceList",
"groupVersion": "v1",
"resources": [
...
]
}
Steps to reproduce
- Clone the repository
- Create the docker kind network
- Prepare CAPD deployment values
- Executed the bootstrap script
What is the current bug behavior?
The bootstrap script fails due to the cluster-reachable Kustomization failure that is reporting the output Forbidden
.
What is the expected correct behavior?
cluster-reachable Kustomization should be able to access the management cluster API using the data contained in the management-cluster-kubeconfig.
Relevant logs and/or screenshots
cluster-reachable Kustomization:
Status:
Conditions:
Last Transition Time: 2024-03-12T10:45:45Z
Message: Detecting drift for revision sha1:c865c67eeb3dff992a2db62135ed177080f4247b with a timeout of 30s
Observed Generation: 1
Reason: ProgressingWithRetry
Status: True
Type: Reconciling
Last Transition Time: 2024-03-12T10:45:45Z
Message: ConfigMap/default/dummy-deps-cluster-reachable dry-run failed: failed to get API group resources: unable to retrieve the complete list of server APIs: v1: Get "https://172.18.0.3:6443/api/v1?timeout=30s": Forbidden
management cluster resource:
root@sylva-control-plane:/# kubectl get cluster.cluster -n sylva-system
NAME CLUSTERCLASS PHASE AGE VERSION
management-cluster Provisioned 23m
root@sylva-control-plane:/# kubectl get cluster.cluster -n sylva-system management-cluster -o yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
annotations:
helm.sh/resource-policy: keep
meta.helm.sh/release-name: cluster
meta.helm.sh/release-namespace: sylva-system
creationTimestamp: "2024-03-12T10:40:28Z"
finalizers:
- cluster.cluster.x-k8s.io
generation: 2
labels:
app.kubernetes.io/managed-by: Helm
cluster.x-k8s.io/cluster-name: management-cluster
helm.toolkit.fluxcd.io/name: cluster
helm.toolkit.fluxcd.io/namespace: sylva-system
name: management-cluster
namespace: sylva-system
resourceVersion: "2940"
uid: 68abf798-9764-4009-ae17-87b2de6badd3
spec:
clusterNetwork:
pods:
cidrBlocks:
- 100.72.0.0/16
serviceDomain: cluster.local
services:
cidrBlocks:
- 100.73.0.0/16
controlPlaneEndpoint:
host: 172.18.0.3
port: 6443
controlPlaneRef:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
name: management-cluster-control-plane
namespace: sylva-system
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerCluster
name: management-cluster
namespace: sylva-system
status:
conditions:
- lastTransitionTime: "2024-03-12T10:41:02Z"
status: "True"
type: Ready
- lastTransitionTime: "2024-03-12T10:40:58Z"
status: "True"
type: ControlPlaneInitialized
- lastTransitionTime: "2024-03-12T10:41:02Z"
status: "True"
type: ControlPlaneReady
- lastTransitionTime: "2024-03-12T10:40:30Z"
status: "True"
type: InfrastructureReady
infrastructureReady: true
observedGeneration: 2
phase: Provisioned
Possible fixes
//