SGCluster reconciliation stuck during major version upgrade because the operator reverts status.postgresVersion

Summary

During an SGDbOps major version upgrade the SGCluster reconciliation can get permanently stuck: the operator reverts status.postgresVersion to the source version right after the major version upgrade Job set it to the target version, and from then on every reconciliation of the SGCluster fails with:

java.lang.IllegalArgumentException: Invalid postgres version 12.16 for SGPostgresConfig postgresconf-for-major-version-upgrade that uses version 13
	at io.stackgres.operator.conciliation.cluster.context.ClusterPostgresConfigContextAppender.appendContext(ClusterPostgresConfigContextAppender.java:57)
	at io.stackgres.operator.conciliation.cluster.context.ClusterPostgresVersionContextAppender.appendContext(ClusterPostgresVersionContextAppender.java:231)
	...
	at io.stackgres.operator.conciliation.AbstractReconciliator.reconciliationCycle(AbstractReconciliator.java:320)
	at io.stackgres.operator.conciliation.cluster.ClusterReconciliator.reconciliationCycle(ClusterReconciliator.java:142)

The StatefulSet is never updated with the major-version-upgrade init container, so the major version upgrade Job loops forever on Waiting StatefulSet to be updated... until the SGDbOps times out.

Steps to reproduce

It is a race, observed in the e2e test dbops-major-version-upgrade (sub-test "Checking that major version upgrade check is working"), see job https://gitlab.com/ongresinc/stackgres/-/jobs/16930792818

  1. Create an SGCluster with Postgres 12 and an SGDbOps majorVersionUpgrade to Postgres 13 with an sgPostgresConfig for Postgres 13.
  2. The Job run-major-version-upgrade.sh replaces the SGCluster setting spec.postgres.version to the target version, spec.configurations.sgPostgresConfig to the target SGPostgresConfig and status.postgresVersion to the target version.
  3. If an SGCluster reconciliation cycle of the operator started before that replace and ends after it, the SGCluster status is written back with the source status.postgresVersion.

Root cause

ClusterReconciliator.onPostReconciliation re-reads the current SGCluster but then replaces the whole status with the one computed from the snapshot used during the reconciliation (currentCluster.setStatus(config.getStatus())), preserving only os, arch, podStatuses, dbOps and managedSql from the current SGCluster. So postgresVersion (and buildVersion) set by the major version upgrade Job in the meantime are overwritten with stale values.

After that, since majorVersionUpgrade is not in DbOpsUtil.ROLLOUT_OPS and the default update strategy is OnlyDbOps, ClusterRolloutUtil.isRolloutAllowed returns false and ClusterPostgresVersionContextAppender uses the (now stale) status.postgresVersion 12.16, which does not match the Postgres 13 SGPostgresConfig already set in the spec. The reconciliation fails on every cycle and nothing ever fixes the status, so the cluster is stuck.

Timeline from the operator log of the job above (namespace dbops-major-version-upgrade-6ac353c1):

  • The Job replace returned the SGCluster (resourceVersion 7130) with status.postgresVersion: 13.12.
  • A concurrent reconciliation (triggered by the pods count mismatch after the Job set instances to 1) patched the StatefulSet and ran onPostReconciliation.
  • From 07:45:14 until the test timeout at 07:55:10 the reconciliation failed ~1 time per second with the exception above.

Expected behavior

The operator must not overwrite the fields of the SGCluster status it did not compute from the current state (in particular postgresVersion and buildVersion while a major version upgrade SGDbOps is in progress) so that the major version upgrade can proceed.

Possible fix

In ClusterReconciliator.onPostReconciliation preserve postgresVersion and buildVersion from the current SGCluster (or merge only the fields computed by the reconciliation) at least when status.dbOps.majorVersionUpgrade is set.