# SGShardedCluster reconciliation issues: recurring ClusterUpdated churn (regression of #2114), stale PendingUpgrade aggregation, unsafe replicas=0 after "Completed" restart, and unretried 409 on pod patch ## Environment - StackGres operator: reproduced on both `1.19.0-rc5` and `1.19.0` (GA) after upgrading - Cluster type: `SGShardedCluster` (Citus), 1 coordinator + 4 workers, single instance each (no HA replicas configured) - Kubernetes: internal cluster, cloud-managed ## Summary While investigating a stuck `postgresVersion` upgrade on one of our `SGShardedCluster`s (separate root cause, our own chart's `SGPostgresConfig` had a static name and hit the documented immutability rejection — not a StackGres bug, mentioning only for context), we found four distinct issues in the `SGShardedCluster` reconciler. Filing together since they were all found in the same investigation, but happy to split into separate issues if preferred. ## Issue 1: `ClusterUpdated` churn on `SGShardedCluster` — likely a regression of #2114 This looks like the same class of problem reported in #2114 (StackGres 1.4.0, Dec 2022), which was closed via !1335 — but that MR's actual scope was narrower than what #2114 reported: it fixed `ClusterBootstrapCompleted` repeating during backup-restore bootstrap (closing #1935/#2226), not the `ClusterUpdated` spam that #2114's report *also* described. That second symptom appears to have never actually been fixed, and we're seeing it again now: ``` Normal ClusterUpdated 2s (x641337 over 14d) stackgres SGShardedCluster citus-qa.example-cluster updated: +Service:example-cluster-routers, +Service:example-cluster-workers, +Endpoints:example-cluster-workers, +Endpoints:example-cluster-routers and other 7 resources where created, 3 resources where deleted, 168 resources where patched ``` Firing roughly every 1-2 seconds, continuously, for 14+ days straight (event's own age-tracking, not us estimating). The per-`SGCluster`-level reconciler (`ClusterUpdated` on the individual coordinator/worker objects) shows the likely mechanism directly in its diff output: ``` SGCluster citus-qa.example-coord updated: StatefulSet:example-coord (+/metadata/creationTimestamp -> 2026-07-20T06:42:31Z), StatefulSet:example-coord (+/metadata/generation -> 2) and other 42 resources where patched ``` `metadata.creationTimestamp` and `metadata.generation` are server-managed fields that should never be part of a desired-vs-live diff — a freshly-built desired-state object will always differ from the live object on these, every single reconciliation cycle, regardless of whether anything meaningful actually changed. That would produce exactly this symptom: an unresolvable "diff" that gets "fixed" every cycle and immediately reappears. Notably, after upgrading the operator from `1.19.0-rc5` to `1.19.0` GA, the **per-instance `SGCluster`-level** churn actually stopped (consistent with the changelog's "wire in the skipUpdate conciliator hook" and the pod-label-propagation fix) — but the **`SGShardedCluster`-level** churn (the `ClusterUpdated` event shown above, which recreates/patches Services and Endpoints) continued unchanged after the upgrade, same rate, same pattern. So whatever fix landed for the per-instance path in 1.19.0 doesn't cover the sharded-cluster-level reconciler. **Impact observed:** none directly on data or pod stability that we could find — we ran two full cluster restarts and a minor version upgrade while this was actively firing, with no disruption traceable to it. But it's continuous, wasteful API server load, and the resulting event/log noise makes genuine problems much harder to spot (which is how we initially became worried something was actually wrong). ## Issue 2: stale `PendingUpgrade` on `SGShardedCluster`, doesn't refresh even after confirmed changes The `SGShardedCluster`'s own `PendingUpgrade` condition kept a `lastTransitionTime` from **five days before we touched anything**, unchanged through: - a full cluster restart (`SGShardedDbOps` `op: restart`, both `ReducedImpact` and later `InPlace` methods, both reaching `Completed`) - a real, confirmed minor version upgrade (`SGShardedDbOps` `op: minorVersionUpgrade`, `17.9` -> `17.10`, verified via `SHOW server_version` on every instance) ``` Last Transition Time: 2026-08-17T06:28:48.556178Z Reason: ShardedClusterRequiresUpgrade Status: True Type: PendingUpgrade ``` Meanwhile the **per-instance `SGCluster`**-level `PendingUpgrade` condition (on `example-coord`) *did* correctly refresh, with an accurate, current reason: ``` Last Transition Time: 2026-09-01T04:20:56.326380Z Reason: ClusterRequiresUpgrade Status: True Type: PendingUpgrade ``` (status message: `"PostgreSQL 17.10 is the latest minor version for major 17. A newer major version 18.4 is available."` — correctly reflecting the real, remaining major-version gap.) So the per-instance status computation works correctly and updates live; the `SGShardedCluster`-level aggregation of that same condition does not, even though the underlying data it should be summarizing is fresh and accurate. (`PendingRestart` at the `SGShardedCluster` level *did* get a fresh timestamp after the restart operations, for comparison — so the aggregation mechanism isn't universally broken, just `PendingUpgrade` specifically seems stuck.) ## Issue 3: `SGShardedDbOps` restart (`method: ReducedImpact`) can leave StatefulSets at `replicas: 0` with orphaned running pods, despite reporting `Completed` This is the one we consider most serious from a safety standpoint. Sequence: 1. Submitted `SGShardedDbOps`, `op: restart`, `restart.method: ReducedImpact`, `onlyPendingRestart: true` 2. Operation proceeded as expected: scaled each StatefulSet 1->2, synced the new replica (confirmed via `patronictl list`, streaming, 0 lag), performed the switchover, old primary pod (`-0`) removed 3. All 5 per-shard `SGDbOps` and the parent `SGShardedDbOps` reported `Completed` / `OperationCompleted` 4. **After completion**, every StatefulSet (`coord`, `shard0`-`shard3`) was left with `spec.replicas: 0`, while a healthy, fully-functioning `Leader` pod (ordinal `-1`) was still `Running` and serving traffic — effectively orphaned, with no StatefulSet backing to recreate it if it were ever evicted 5. The condition did not self-resolve — we observed it stuck for several minutes with no operator activity on it, and a manual `kubectl patch statefulset ... replicas=1` was immediately reverted by the operator on its next reconcile pass (so it wasn't simply "still working," something was actively re-asserting `0`) 6. Recovery required: pausing reconciliation (`stackgres.io/reconciliation-pause=true` on the affected `SGCluster`s) to stop the fight, manually restoring `replicas`, then a manual, controlled `patronictl switchover` back to the properly-StatefulSet-owned pod before it was safe to remove the orphan We could not determine from the operator logs *why* it settled on `replicas: 0` as the "completed" end state instead of `1` (the actual configured `SGShardedCluster.spec.coordinator.instances` / `workers.instancesPerCluster`). Retrying the identical restart with `method: InPlace` instead completed cleanly with no `replicas: 0` gap, for what it's worth as a data point — the bug appears specific to the `ReducedImpact` finalization path. ## Issue 4: `SGShardedDbOps-ReconciliationLoop` errors on already-`Completed` operations After the `SGShardedDbOps` from Issue 3 completed, the reconciler kept erroring on it every reconcile cycle instead of leaving the finished object alone: ``` ERROR [io.st.op.conciliation] (SGShardedDbOps-ReconciliationLoop) Reconciliation of SGShardedDbOps citus-qa.qa-pending-restart-20260901 failed: java.lang.IllegalArgumentException: SGShardedDbOps citus-qa.qa-pending-restart-20260901 target non existent SGShardedCluster example-cluster at io.stackgres.operator.conciliation.shardeddbops.StackGresShardedDbOpsContext.lambda$getShardedCluster$0(...) at java.base/java.util.Optional.orElseThrow(Optional.java:403) ... ``` (The `SGShardedCluster` referenced was never actually missing — we confirmed it existed and was healthy throughout. This looks like a lookup/caching issue specific to reconciling a terminal/completed `SGShardedDbOps`.) Deleting the completed `SGShardedDbOps` object stopped the errors immediately. Not harmful (self-contained to that one object, no downstream effect we could find), but noisy, and surprising that a finished operation keeps getting reconciled at all. ## Bonus, lower priority: unretried 409 on pod patch during restarts Separately, during active restarts we consistently saw this exact error, always from the same code path: ``` ERROR [io.st.op.conciliation] (SGCluster-ReconciliationLoop) Reconciliation of SGCluster citus-qa.example-coord failed: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: .../pods/example-coord-0?fieldManager=StackGres&force=true. Message: Operation cannot be fulfilled on pods "example-coord-0": the object has been modified; please apply your changes to the latest version and try again. ... reason=Conflict ... at io.stackgres.operator.conciliation.cluster.ClusterStatefulSetWithPrimaryReconciliationHandler.lambda$fixPods$47(ClusterStatefulSetWithPrimaryReconciliationHandler.java:716) at io.stackgres.operator.conciliation.cluster.ClusterStatefulSetWithPrimaryReconciliationHandler.fixPods(...) ``` Other patch call sites elsewhere in the operator appear to wrap conflicts in a retry (we saw `WARN [io.st.co.RetryUtil] Will retry after 2 milliseconds due to error: ... 409` elsewhere in the same logs), but `ClusterStatefulSetWithPrimaryReconciliationHandler.fixPods` doesn't seem to use that same retry wrapper — it just lets the 409 bubble up as an ERROR and fails that reconciliation cycle, relying on the next scheduled cycle to eventually succeed. Self-healing, just noisy, and inconsistent with how conflicts are handled elsewhere in the codebase. ## Happy to provide - Full operator logs around any of the above - The exact `SGShardedDbOps`/`SGShardedCluster`/`SGCluster` manifests used - Further reproduction if a maintainer wants a specific test built Thanks for StackGres — filing this after a long night of investigation, but the tooling (`patronictl`, the `SGDbOps` restart/upgrade operations, the operator's own condition/status reporting once we found the right fields) made it very tractable to diagnose and safely recover from all of the above without any data loss.