Keycloak legacy cleanup inventory (!9134) · Merge requests · Sylva-projects / sylva-core

What does this MR do and why?

Root problem:

On some scenarios from upgrade-from-release-1.7 , keycloak-base-resources never becomes ready and blocks everything downstream.

From Job update-management-cluster (#16414822060) · Jobs · Sylva-projects / sylva-core · GitLab

❌ 🔻🔻🔻🔻🔻 sylvactl watch conclusion... 🔎
00:00
unit Kustomization/keycloak-base-resources did not become ready after 5m0s
Details on Kustomization/keycloak-base-resources and related resources:
IDENTIFIER                                         STATUS     REASON               MESSAGE
Kustomization/sylva-system/keycloak-base-resources InProgress                      Kustomization generation is 3, but latest observed generation is 2
╰┄╴┬┄┄[Conditions]
   ├┄╴Reconciling                                  True       ProgressingWithRetry Detecting drift for revision 0.0.0-git-edec7cfc@sha256:abbe61e2b5591f6ad988980950cd472033cb26bacb228434dde0daa218d1a49d with a timeout of 30s
   ├┄╴Ready                                        False      PruneFailed          delete failed, errors: KeycloakRealm/keycloak/sylva query failed: no matches for kind "KeycloakRealm" in version "legacy.k8s.keycloak.org/v1alpha1";
   ╰┄╴Healthy                                      True       Succeeded            Health check passed in 228.613942ms
❌ 🔺🔺🔺🔺🔺 🔎

Same on jobs:

Root Cause:

All cluster outputs from below come from the job 16414822060.

On release-1.7, keycloak-base-resources created a KeycloakRealm for the legacy keycloak-realm-operator, whose CRDs same from the keycloak-legacy-operator unit:

git grep -n 'KeycloakRealm' origin/release-1.7 -- kustomize-units/keycloak-base-resources; git show origin/release-1.7:charts/sylva-units/values.yaml | grep -n -A3 '^  keycloak-legacy-operator:$'
origin/release-1.7:kustomize-units/keycloak-base-resources/realm/keycloak-realm.yaml:3:kind: KeycloakRealm
2876:  keycloak-legacy-operator:
2877-    info:
2878-      description: installs Keycloak "legacy" operator
2879-      maturity: stable

Flux keeps that objects in the unit's inventory. On main the realm is managed via Crossplane and the manifest is gone and the keycloak-legacy-operator unit was removed in d2f162111:

[git:main]root@tibi-vbmh:sylva-core# git grep -n 'KeycloakRealm' origin/main -- kustomize-units/keycloak-base-resources; git show --stat d2f162111 | head -20
commit d2f16211183c50f7fdc4961f838bce964255d09e
Author: Cristina Isaroiu <cristina.isaroiu@orange.com>
Date:   Mon Jul 27 16:36:02 2026 +0000

    Remove `keycloak-realm-operator` unit

 .../test-framework/tests/test_crd_recreation.py    |  7 +-
 .../scripts/keycloak-legacy-operator-cleanup.sh    |  5 ++
 charts/sylva-units/units-description.md            |  2 +-
 charts/sylva-units/values.yaml                     | 82 ++++++++++------------
 .../keycloak-base-resources/kustomization.yaml     |  1 -
 .../realm/keycloak-realm.yaml                      | 15 ----
 6 files changed, 49 insertions(+), 63 deletions(-)
[git:main]root@tibi-vbmh:sylva-core#

During the upgrade:

The sylva-units Helm upgrade deletes the keycloak-legacy-operator Kustomization and Flux prunes everything that was owned by that (CRDs included). In the snapshot, non of them exists anymore

crustgather-job-16414822060 ~> kubectl --context management get ks -n sylva-system keycloak-legacy-operator; kubectl --context management get crd -o name | grep -c legacy.k8s.keycloak.org
Error from server (NotFound): the server could not find the requested resource (get kustomizations.kustomize.toolkit.fluxcd.io keycloak-legacy-operator)
0
crustgather-job-16414822060 ~>

keycloak-legacy-operator-cleanup strips the finalizer left on KyecloakRealm/sylva by removed ooperator, so the CRD deletion can be fully completed. Additionaly, the CR goes along with the CRD. The scripts is only doing that and the job is running long before keycloak-base-resource applied anything:

crustgather-job-16414822060 ~> cat ~/sylva-core/charts/sylva-units/scripts/keycloak-legacy-operator-cleanup.sh; kubectl --context management -n keycloak get job keycloak-legacy-operator-cleanup -o jsonpath='succeeded={.status.succeeded} completed={.status.completionTime}{"\n"}'
#!/bin/bash
set -e

kubectl -n keycloak patch keycloakrealm.legacy.k8s.keycloak.org sylva \
    --type=merge -p '{"metadata":{"finalizers":[]}}' 2>/dev/null || true
succeeded=1 completed=2026-09-10T10:44:32Z
crustgather-job-16414822060 ~>

keycloak-base-resource applies the new revision only after crossplane, crossplane-provider-keycloak, fix-keycloak-ownerrefs and keycloak . By then, the CRD is completely gone and the stale inventory entry can not be pruned anymore. It is still on the 1.7 revision and the entry is the only legacy.k8s.keycloak.org entry in any inventory in the cluster:

crustgather-job-16414822060 ~> kubectl --context management get ks -n sylva-system keycloak-base-resources -o jsonpath='dependsOn: {range .spec.dependsOn[*]}{.name}{" "}{end}{"\n"}lastApplied={.status.lastAppliedRevision}{"\n"}{range .status.conditions[*]}{.lastTransitionTime}{"\t"}{.type}{"\t"}{.status}{"\t"}{.reason}{"\n"}{end}{range .status.inventory.entries[*]}{.id}{"\n"}{end}' | grep -E 'dependsOn|lastApplied|Ready|legacy'
dependsOn: cluster-machines-ready crossplane crossplane-provider-keycloak fix-keycloak-ownerrefs flux-system keycloak kyverno kyverno-policies root-dependency-3
lastApplied=0.0.0-git-4ba82e06@sha256:ec3a108a3498b938083a047e15527a4c3bd5cd706e2c31df07a5536f093c2182
2026-09-10T11:04:45Z    Ready   False   PruneFailed
keycloak_sylva_legacy.k8s.keycloak.org_KeycloakRealm
crustgather-job-16414822060 ~>

crustgather-job-16414822060 ~> kubectl --context management get ks -A -o jsonpath='{range .items[*]}{range .status.inventory.entries[*]}{.id}{"\n"}{end}{end}' | grep -i legacy
keycloak_sylva_legacy.k8s.keycloak.org_KeycloakRealm
_kube-job-keycloak-legacy-operator-cleanup-sylva-system_rbac.authorization.k8s.io_ClusterRole
_kube-job-keycloak-legacy-operator-cleanup-sylva-system_rbac.authorization.k8s.io_ClusterRoleBinding
keycloak_keycloak-legacy-operator-cleanup-sa__ServiceAccount
keycloak_keycloak-legacy-operator-cleanup_rbac.authorization.k8s.io_Role
keycloak_keycloak-legacy-operator-cleanup_rbac.authorization.k8s.io_RoleBinding
keycloak_kube-job-keycloak-legacy-operator-cleanup__ConfigMap
keycloak_keycloak-legacy-operator-cleanup_batch_Job
crustgather-job-16414822060 ~>

The cleanup unit waits for none of the units keycloak-base-resources watis for and so it always runs first:

crustgather-job-16414822060 ~> kubectl --context management get ks -n sylva-system keycloak-legacy-operator-cleanup -o jsonpath='dependsOn: {range .spec.dependsOn[*]}{.name}{" "}{end}{"\n"}'
dependsOn: cluster-machines-ready flux-system kyverno kyverno-policies root-dependency-3
crustgather-job-16414822060 ~>

Flux is not tolerating the stale entry: ssa.ResourceManager.Delete ignores only NotFound and a missing kind is returned as query failed and fails the whole prunning process. I have also checked the current main of fluxcd/pkg so a Flux upgrade would not change this:

crustgather-job-16414822060 ~> curl -sSL https://raw.githubusercontent.com/fluxcd/pkg/main/ssa/manager_delete.go | sed -n 63,75p
// Delete deletes the given object (not found errors are ignored).


func (m *ResourceManager) Delete(ctx context.Context, object *unstructured.Unstructured, opts DeleteOptions) (*ChangeSetEntry, error) {

        existingObject := &unstructured.Unstructured{}
        existingObject.SetGroupVersionKind(object.GroupVersionKind())
        err := m.client.Get(ctx, client.ObjectKeyFromObject(object), existingObject)
        if err != nil {
                if !apierrors.IsNotFound(err) {
                        return m.changeSetEntry(object, UnknownAction),
                                fmt.Errorf("%s query failed: %w", utils.FmtUnstructured(object), err)
                }
                return m.changeSetEntry(object, DeletedAction), nil
        }
crustgather-job-16414822060 ~>

This can not be fixed by modifying the depends_on as the CRD removal is a side effect of a Ks being deleted, not a unit anyone can wait for.

The proposed fix:

keycloak-legacy-operator-cleanup already runs early on every upgrade from 1.7 and already owns this migration. It now also removes the stale keycloak_sylva_legacy.k8s.keycloak.org_KeycloakRealm entry from status.inventory of sylva-system/keycloak-base-resources - the same thing a successful prune would have done, on a CR that the CRD cascade deletes anyway.

charts/sylva-units/scripts/keycloak-legacy-operator-cleanup.sh: after the finalizer patch, it is locating the entry by id and remove it with a JSON patch within 5 attempts on the status subresource
charts/sylva-units/values.yaml: the ClusterRole of the unit gains get and patch on kustomizations and kustomizations/status in kustomize.toolkit.fluxcd.io

Important mention: coauthored by Claude (Opus 5, via Claude Code).

Responds to issue: #4577 (closed)

Related commit: d2f16211

Test coverage

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	`capd`, `capo`, `capm3`
🚀	Bootstrap Provider	`kubeadm` (alias `kadm`), `rke2`, `okd`, `ck8s`
🐧	Node OS	`ubuntu`, `suse`, `na`, `leapmicro`
🛠️	Deployment Options	Deployment option list and description
🎬	Pipeline Scenarios	Available scenario list and description
🟢	Enabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🔴	Disabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🏗️	Target platform	Can be used to select specific deployment environment Available platform list and description
⚡	Pipeline control	`autorun`, `manual` or `blocking`. Can be used to override global config and start a deployment pipeline the required way

Global config for deployment pipelines

autorun pipelines
allow failure on pipelines
record sylvactl events

Notes:

Enabling autorun will make deployment pipelines to be run automatically without human interaction
Disabling allow failure will make deployment pipelines mandatory for pipeline success.
if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited Sep 18, 2026 by Tiberiu Mihai

Keycloak legacy cleanup inventory

What does this MR do and why?

Related reference(s)

Test coverage

CI configuration

Global config for deployment pipelines

Merge request reports