Use kyverno to define namespaces with specific PodSecurityAdmission levels

Summary

With RKE2, the default configuration is to deploy with a CIS profile leading to set a PSA level restricted at cluster level. I can happen that some projects will require to deploy more privileged pods. Sometimes the namespace can be created manually with labels corresponding to the desired PSA level, before deploying the actual CNF. In other times, the user cannot control the creation of the namespaces. One such example is a test suite that creates namespaces with random names with a specific prefix, and deploys privileged pods in it.

Kyverno can be used in this situation to mutate the namespaces matching certain criteria.

related references

https://kyverno.io/blog/2023/06/12/using-kyverno-with-pod-security-admission/ https://kubernetes.io/docs/tasks/configure-pod-container/migrate-from-psp/

Details

The static and strict nature of the PSA has the consequence that a policy engine is needed when more fine-grained policies are required (and policies on attributes not covered by the PSA).
Kyverno is already deployed by default on the management clusters. It should also be deployed by default on the workload clusters, with the possibility to opt-out on some clusters if need be, for example if another policy engine like OPA gatekeeper is prefered on a workload cluster.

In order to be able to add PSA labels on namespaces, the kyverno backend needs the permission to update the namespaces, and with rancher to use the update-psa verb:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: background-controller
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/part-of: kyverno
  name: kyverno:update-namespace
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: background-controller
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/part-of: kyverno
  name: kyverno:update-psa
rules:
- apiGroups:
  - management.cattle.io
  resources:
  - projects
  verbs:
  - updatepsa

Example of policy that add a privileged label to the namespaces starting with the prefix "test-":

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-privileged-test-namespaces
  annotations:
    policies.kyverno.io/title: Add Privileged Label to Existing Namespaces starting with the test prefix
    policies.kyverno.io/category: Pod Security Admission
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Namespace
    kyverno.io/kyverno-version: 1.8.0
    policies.kyverno.io/minversion: 1.7.0
    kyverno.io/kubernetes-version: "1.24"
    policies.kyverno.io/description: >-
      This policy add the label `pod-security.kubernetes.io/enforce: privileged` to the new and existing namespaces whose names start with the string test-
spec:
  mutateExistingOnPolicyUpdate: true
  rules:
  - name: label-privileged-namespaces
    match:
      any:
      - resources:
          kinds:
          - Namespace
    mutate:
      targets:
        - apiVersion: v1
          kind: Namespace
      patchStrategicMerge:
        metadata:
          <(name): "test-*"
          labels:
            pod-security.kubernetes.io/enforce: privileged

cc: @corinne.saintjalme