Engine upgrade deadlocks on Grafana with a ReadWriteOnce volume

Problem

Upgrading the engine chart deadlocks on Grafana whenever its persistence is backed by a ReadWriteOnce volume, which is the default on most cloud storage classes.

ryax-grafana is a Deployment with strategy: RollingUpdate (maxSurge: 25%, maxUnavailable: 25%, the kube-prometheus-stack/grafana subchart default) and a ReadWriteOnce PVC. On upgrade the two ReplicaSets deadlock:

  • the new pod cannot start — FailedAttachVolume: Waiting for detach for volume "pvc-…" Volume is already used by pod(s) ryax-grafana-<old>;
  • the old pod is not torn down, because the rollout will not scale the old ReplicaSet down until the new pod is Ready.

Neither side can move. helm upgrade --wait then blocks until it times out and marks the release failed, even though every other component upgraded fine.

Deleting the old pod by hand does not fix it: the old ReplicaSet still wants one replica, so the pod is recreated and re-attaches the volume, and the new pod keeps waiting.

Reproduce

On a cluster with an RWO default storage class (Scaleway sbs-default, EBS, Azure Disk, …), with Grafana persistence enabled:

helm upgrade ryax oci://registry.ryax.org/release-charts/ryax-engine \
  --version 26.9.0 -n ryaxns -f values.yaml --wait --timeout 20m

Observed on dev1 upgrading 26.7.0 → 26.9.0 on 2026-09-15:

Warning  FailedAttachVolume  attachdetach-controller
  Waiting for detach for volume "pvc-59368a51-472d-42e7-a423-0bfa6c15c433"
  Volume is already used by pod(s) ryax-grafana-7f94476dc7-dccsn

with ryax-grafana-7cb79d7687-fq4kf 0/3 Init:0/1 sitting there for 20 minutes while ryax-grafana-7f94476dc7-dccsn 3/3 Running held the volume.

Fix

Set Recreate on the Grafana Deployment — the correct strategy for a single-replica workload on an RWO volume, since there is no availability to preserve anyway. In the engine chart's values:

kube-prometheus-stack:
  grafana:
    deploymentStrategy:
      type: Recreate

(the grafana subchart exposes deploymentStrategy; worth confirming the key against the pinned chart version).

Current state of dev1

I patched the live Deployment by hand to unblock the upgrade:

kubectl patch deploy ryax-grafana -n ryaxns --type=json \
  -p '[{"op":"replace","path":"/spec/strategy","value":{"type":"Recreate"}}]'

This is unmanaged drift. The next helm upgrade of the engine chart will reconcile it back to RollingUpdate, and the next upgrade after that will deadlock again. The fix belongs in the chart values.

Why it matters

Any admin following the documented upgrade procedure on an RWO cluster hits a 20-minute hang and a failed release, with nothing in the release notes to explain it, and the obvious remedy (delete the stuck pod) does not work.