SGCluster is silently pinned to its init CPU architecture: changing pods.scheduling to another arch causes cryptic crashloops with no validation/condition, and there's no documented migration path

An existing SGCluster that was initialized on one CPU architecture (e.g. arm64) cannot be moved to nodes of another architecture (e.g. amd64). This is effectively a hard constraint (Postgres can't stream-replicate across architectures, and StackGres persists arch-specific content on the data volume), but:

  1. It is not validated - you can set spec.pods.scheduling to target a different arch and StackGres will happily try, then fail.
  2. The failure surfaces as cryptic container errors (exec /bin/sh: no such file or directory, exec /bin/sh: accessing a corrupted shared library) before the one clear message appears, and that clear message only lives in the cluster-controller log - not as an SGCluster status condition or Kubernetes event.
  3. There is no documentation stating a cluster is pinned to its init architecture, nor a supported procedure to migrate architectures.
  4. A data PVC that even briefly hosted a wrong-arch instance gets contaminated and then fails on the correct architecture too, until the PVC is recreated.

Environment

  • StackGres operator & cluster-controller: 1.18.8
  • Patroni/Postgres image: quay.io/ongres/patroni:v4.0.5-pg17.4-build-6.41 (PostgreSQL 17.4) - verified correctly multi-arch (amd64 + arm64 manifests both valid)
  • Kubernetes: 1.34.9 (Talos Linux 1.13.4)
  • Cloud/CSI: Hetzner Cloud, csi.hetzner.cloud (hcloud-volumes, RWO, WaitForFirstConsumer)
  • SGCluster: 3 instances, initialized on arm64 nodes; attempted move to amd64 nodes in the same cluster

Steps to reproduce

  1. Run a healthy multi-instance SGCluster whose pods are on arm64 nodes.
  2. Add amd64 worker nodes to the cluster.
  3. Set the SGCluster to schedule onto the amd64 nodes: spec: pods: scheduling: nodeSelector: kubernetes.io/arch: amd64 tolerations: - { key: , operator: Equal, value: , effect: NoSchedule }
  4. Recreate one replica pod so it reschedules onto an amd64 node: - (a) reusing its existing PVC, and/or - (b) with a fresh/empty PVC (delete the pod + its PVC).

Expected behavior One of:

  • The operator/validating webhook rejects a scheduling change that would place instances on an architecture incompatible with the cluster's initialized architecture (with a clear message), or
  • StackGres surfaces a clear, actionable SGCluster status condition + Event (e.g. ArchitectureMismatch) instead of a generic container crashloop, and
  • Docs describe that a cluster is pinned to its init architecture and link a supported migration procedure.

Actual behavior Three distinct failure modes observed:

(a) Reusing the existing (arm64-bootstrapped) PVC on an amd64 node - patroni container: exec /bin/sh: no such file or directory (wrong-arch ELF: the data volume carries arch-specific content)

(b) Fresh/empty PVC on an amd64 node, same (arm64-initialized) cluster - cluster-controller crashes, which stalls the patroni container (it waits on the controller): java.lang.IllegalStateException: The cluster was initialized with aarch64/linux but this instance is x86_64/linux at io.stackgres.common.postgres.PostgresBootstrapReconciliator.reconcile(PostgresBootstrapReconciliator.java:63) at io.stackgres.cluster.controller.ClusterControllerReconciliator.reconcile(ClusterControllerReconciliator.java:133) ... io.quarkus.runtime.Quarkus ... stackgres-cluster-controller stopped → pod stays CrashLoopBackOff; this message is only in the controller log (no SGCluster condition/event).

(c) PVC contamination - a fresh PVC that briefly hosted the failed amd64 attempt, then rescheduled back to an arm64 node, fails with: exec /bin/sh: accessing a corrupted shared library i.e. amd64 content written during the brief attempt corrupts the volume; only fixed by deleting that PVC and rebuilding the replica clean.

The production database itself was never lost (leader + one replica stayed healthy on arm64); the affected replica was reverted to arm64 by removing scheduling and recreating it with a clean PVC.