Add the medik8s operator (!8047) · Merge requests · Sylva-projects / sylva-core

What does this MR do and why?

This MR aims to introduce Medik8s (more specifically, NHC and SNR) as an optional unit targeted for workload clusters, thus addressing the requirements for automated and safe node failure detections and recovery.

The two components which are introduced by this unit for a workload cluster are:

NHC (NodeHealthcheck Operator) - v.0.12.0 - monitors node conditions and triggers remediations
SNR (Self Node Remediation) - v.0.13.0 - fences unhealthy nodes

Responds to issue #3892 (closed)

Test coverage

In order to test the behavior of this component, 3 distinct tests have been used which are going to be described as it follows.

N.B. In order to test the functionalities which are provided by NHC and SNR, I have kept the testing on a single RKE2 cluster which is composed by 3 Control Plane Nodes and 3 Worker Nodes, deployed as a Management cluster with the Medik8s unit placed in the general values.yaml. Prior to this MR, the unit has been moved to workload-cluster.values.yaml.

The cluster architecture:

Scenario 1 ---- Description: This scenario aims to show how NHC and SNR are working when a node gets into a NotReady state in order to be able to reduce the downtime of a service and protect the workload. The expected workflow: detection -> fencing -> eviction -> cleanup

For all the three scenarios, the same deployment with 3 pods have been used with each pod on a different worker node:

We have picked one of the worker nodes in order to stop its kubelet and in order to see how NHC and SNR are acting:

The worker node is being marked as NotReady :

NHC creats a SelfNodeRemediation CR

out-of-service taint is added and pods are evicted

===============================

Fencing is completed, taints removed, SNR CR is cleaned up. The detection-fencing-eviction cycle can be seen in the following logs:

Total recovery time was around 4 minutes (kubelet stopped -> pod on a healthy node).

Scenario 2 ---- Description: Storm protection - This scenario is proving how minHealthy can be tweaked in order to adjust the behavior of SNC according to the user's needs. Basically, stopped the kubelet on two worker nodes at the same time and see if the minHealthy set to 51% is reacting correctly.

MinHealthy from the node-healtcheck.yaml from the medik8s-operators unit

Same configuration for the demonstrative workload as in Scenario 1

rke2-agent stopped on two nodes at the same time

the 2 nodes are now marked as NotReady

One of the pods is getting into Terminating state, but there is no fencing or SelfNodeRemediation CR

Same for the second pod/node

Logs which are proving that NHC is not going to spawn any SelfRemediationNode :

This is the expected behavior and it acts as a safeguard for the cases in which most of the worker nodes get unhealthy and NHC assumes a wider issue (something like a network partition or an API server problem) and prevents cascading reboots that would escalate the situation.

Scenario 3 ---- Description: Here is simulated a pause in the SNC in order to test the possibility of quickly stopping it in case of operations such as an update.

Same configuration for the demonstrative workload as in both Scenario 1 and Scenario 2

NodeHealthCheck paused

rke2-agent stopped on a single worker node

Node gets into NotReady state

Even though the pod is getting into Terminating state, no SelfNodeRemediation is spawned

Set spec.pauseRequests on the NodeHealthCheck CR, stop the rke2-agent on a worker. NHC us going to observe the node as unhealthy, but it is not going to spawn any SNR. Removing the pause request is going to trigger the same behavior as in Scenario 1. Ideal for operations.

Known issue: RBAC denials for event creation

Both NHC and SNR controllers attempt to create K8s Events in the default namespace. This is actually an upstream issue in the operators RBAC scope - The ClusterRole grants event creation cluster-wide, but the controllers target the wrong namespace when emitting events for cluster wide Node objects.

[git:tibm-issue-3892-medik8s]root@vbmh4-feper:sylva-core# kubectl -n medik8s logs deploy/self-node-remediation-controller-manager -c manager 2>&1 | grep -c 'events.*forbidden.*"default"'
33

Overall impact - none on functionality. The errors are not retried (as can be seen in the previous logs). The only adversive effect is just log noise during remediations events.

Additional operators that may be added in the future:

Fence Agents Remediation (FAR)

https://github.com/medik8s/fence-agents-remediation Hardware-level fencing via BMC/IPMI/Redfish - direct power control of the affected nodes.

Machine Deletion Remediation (MDR)

https://github.com/medik8s/machine-deletion-remediation Deletes CAPI machine objects, triggering full re-provisioning by the infrastructure provider.

Node Maintenance Operator (NMO)

https://github.com/medik8s/node-maintenance-operator Declarative cordon and drain via NodeMaintenance CRs.

They may all be tested for integration with Sylva, but I propose to keep them for a further MR after we validate the usage and the potential of the SNR and NHC operators for the workload-cluster.

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	`capd`, `capo`, `capm3`
🚀	Bootstrap Provider	`kubeadm` (alias `kadm`), `rke2`, `okd`, `ck8s`
🐧	Node OS	`ubuntu`, `suse`, `na`, `leapmicro`
🛠️	Deployment Options	Deployment option list and description
🎬	Pipeline Scenarios	Available scenario list and description
🟢	Enabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🔴	Disabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🏗️	Target platform	Can be used to select specific deployment environment Available platform list and description
⚡	Pipeline control	`autorun`, `manual` or `blocking`. Can be used to override global config and start a deployment pipeline the required way

Global config for deployment pipelines

autorun pipelines
allow failure on pipelines
record sylvactl events

Notes:

Enabling autorun will make deployment pipelines to be run automatically without human interaction
Disabling allow failure will make deployment pipelines mandatory for pipeline success.
if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited Aug 31, 2026 by Tiberiu Mihai

Add the medik8s operator