On workload cluster, 'Cluster Member' default role does not permit 'network-attachment-definitions' use
on Sylva version 1.0.0, I allocated one workload cluster on Openstack VMs fitted with Multus Unit and secondary network as one ProviderNet. I allocated a specific user & group in Keycloak and I set the '' Cluster Member' role to this 'WC/Cluster and Project Members/Cluster Membership'. Thks to this, this 'davidalles' user succeeded in login into Rancher and get its kubeconfig file and to allocate in rancher a first 'project and related namespace (i.e. 'davidappli').
Issue: By default, the 'Cluster Member' does not grant the permission on 'network-attachment-definitions' api usage which is required for Multus configuration
$ more netatt.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-confprivate
namespace: davidappli
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "ens4",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "172.20.60.240/29",
"rangeStart": "172.20.60.242",
"rangeEnd": "172.20.60.244",
"gateway": "172.20.60.241"
}
}'
$ k apply -f netatt.yaml -n davidappli --kubeconfig ../wc-local-ders-davidalles-kubeconfig.yaml
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "k8s.cni.cncf.io/v1, Resource=network-attachment-definitions", GroupVersionKind: "k8s.cni.cncf.io/v1, Kind=NetworkAttachmentDefinition"
Name: "macvlan-confprivate", Namespace: "davidappli"
from server for: "netatt.yaml": network-attachment-definitions.k8s.cni.cncf.io "macvlan-confprivate" is forbidden: User "u-zpfgkp3chc" cannot get resource "network-attachment-definitions" in API group "k8s.cni.cncf.io" in the namespace "davidappli"
A workaround is possible (Thanks to @alain.thioliere): In Rancher, logged as 'Sylva-admin'
- In 'Users & Authentication/Role Templates/Project/Namespace' allocate a new role template named 'netattachmanager', configured with 'AllVerbs', resources='network-attachment-definitions', apiGroups='k8s.cni.cncf.io'
- In 'Cluster and Project Members/Project Membership/Add' to add this new project membership for the related user's group onto this 'netattachmanager'
Then the deployment is possible
$ k api-resources --kubeconfig ../wc-local-ders-kubeconfig |grep k8s.cn
network-attachment-definitions net-attach-def k8s.cni.cncf.io/v1 true NetworkAttachmentDefinition
$ k auth can-i -n davidappli --kubeconfig ../wc-local-ders-davidalles-kubeconfig.yaml --list |grep netw
°°°
network-attachment-definitions.k8s.cni.cncf.io [] [] [create delete get list patch update watch]
°°°
$ k apply -f netatt.yaml -n davidappli --kubeconfig ../wc-local-ders-davidalles-kubeconfig_bis.yaml
networkattachmentdefinition.k8s.cni.cncf.io/macvlan-confprivate created
$ k apply -f pod_multus.yaml -n davidappli --kubeconfig ../wc-local-ders-davidalles-kubeconfig_bis.yaml
pod/mpod1 created
$ k describe pod mpod1 -n davidappli --kubeconfig ../wc-local-ders-davidalles-kubeconfig_bis.yaml
Name: mpod1
Namespace: davidappli
Priority: 0
Service Account: default
Node: cluster-md-md0-54d3d73947-fpxkn/192.168.1.187
Start Time: Thu, 29 Feb 2024 10:27:27 +0000
Labels: <none>
Annotations: cni.projectcalico.org/containerID: 4c8c68ffbb65d7cf3fb1ed2c88f40798e0754955c14212016465946a61438e6b
cni.projectcalico.org/podIP: 100.72.127.187/32
cni.projectcalico.org/podIPs: 100.72.127.187/32
k8s.v1.cni.cncf.io/network-status:
[{
"name": "k8s-pod-network",
"ips": [
"100.72.127.187"
],
"default": true,
"dns": {}
},{
"name": "davidappli/macvlan-confprivate",
"interface": "net1",
"ips": [
"172.20.60.243"
],
"mac": "2e:3a:12:12:6e:d2",
"dns": {}
}]
Edited by Gurvan Moal