[Release-1.6] Keycloak unit intermittently fails during management-cluster control-plane rolling updates

On release-1.6, CI jobs with a management-cluster CP rolling update intermittently fails on the keycloak unit.

main/release-1.7 are not affected (they use the keycloak Helm chart instead of the operator).

Failed pipelines:

update-management-cluster (#15924118144) · Jobs · Sylva-projects / sylva-core · GitLab

update-management-cluster (#15995002906) · Jobs · Sylva-projects / sylva-core · GitLab

When it might happen

  1. During cp node rolling update, when one of the node goes down, the apiServer on that node is removed from the available endpoints. Already established TCP connections to that apiServer might not be reset/closed, so requests using those connections can stay stuck until they timeout.

  2. The Keycloak controller gets the resource from the informer cache for reconciliation. If the watch connection to API server is also affected, the informer cache can remain stale and keep the older resourceVersion.

    2026-08-09 20:50:49,853 WARN  [io.jav.ope.pro.eve.EventProcessor] (ReconcilerExecutor-keycloakcontroller-135) Uncaught error during event processing ExecutionScope{ resource id: ResourceID{name='keycloak', namespace='keycloak'}, version: 187290} - but another reconciliation will be attempted because a superseding event has been received or another retry attempt is pending.: io.javaoperatorsdk.operator.AggregatedOperatorException: Exception(s) during workflow execution.        


>>> Note the resourceVersion here is 187290 and controller keeps retrying with the same

  3. During the same time, the keycloak controller itself writes the error into the Keycloak CR status by making new write request to API server on another node, and that request goes through. This moves the resourceVersion of the object forward, but the controller keeps retrying with the older one.

    manager: kustomize-controller
time: "2026-08-09T19:46:44Z"

manager: keycloakcontroller
subresource: status
time: "2026-08-09T20:50:49Z"       >>>> Same time when operator is during mid-reconcile
    resourceVersion: "188311"        >>>> The resorceVersion moved forward on the object
      - lastTransitionTime: "2026-08-09T20:50:49.709826406Z"
    message: "...The timeout period of 10000ms has been exceeded while executing
      GET /api/v1/namespaces/keycloak/secrets/keycloak-bootstrap-admin ..."
    type: HasErrors

  4. When the reconciliation is retried, it continues with the older resourceVersion, which leads to a conflict on the status update, resulting in exhausted retries and the same issue as https://github.com/keycloak/keycloak/issues/43198, with the 409 error:

Operation cannot be fulfilled on keycloaks.k8s.keycloak.org "keycloak": the object has been modified; please apply your changes to the latest version and try again.

The upstream issue https://github.com/keycloak/keycloak/issues/43198 is fixed in JOSDK 5.1.5 / keycloak 26.6.0, but we're pinned to 26.4.0 in release-1.6. Bumping to 26.6.0 is not an option because of the blocker regression keycloak/keycloak#48030 (all operator-managed resources need deleting). The fix landed in 26.6.2/26.7.0.

co-authored and co-analysed by claude

Edited by Ishita Mittal