SGDbOps rollout operations may complete before all the Pods have been restarted

Summary

The restart, securityUpgrade and minorVersionUpgrade SGDbOps may complete before all the required Pods have actually been restarted.

DbOpsStatusManager.updateRolloutBasedDbOps decides that the rollout completed by looking at the Pods and at the patroni members:

if ((primaryIsReadyAndUpdated || primaryIsExternal)
    && securityUpgradeWasApplied
    && minorVersionUpgradeWasApplied
    && pods.size() == podsReadyAndUpdated.size()) {
  updateCondition(getRolloutCompleted(), source);
  ...
}

Patroni and the StatefulSet controller update their status with some delay, so there is a window where all the Pods still look ready and updated while a restart is actually pending. When the reconciliation cycle observes the resources in such window the operation is marked as completed while some Pod has still to be restarted.

Proposed changes

  • Add the fields SGDbOps.spec.restart.statusUpdateDelay, SGDbOps.spec.securityUpgrade.statusUpdateDelay and SGDbOps.spec.minorVersionUpgrade.statusUpdateDelay, an ISO-8601 duration that default to PT1M.
  • Add the fields SGDbOps.status.restart.lastUpdate, SGDbOps.status.securityUpgrade.lastUpdate and SGDbOps.status.minorVersionUpgrade.lastUpdate, that are set to the current instant whenever a condition or any other status field of the SGDbOps is updated in DbOpsStatusManager.updateRolloutBasedDbOps.
  • The conditions DbOpsStatusCondition.DBOPS_ROLLOUT_COMPLETED and DbOpsStatusCondition.DBOPS_COMPLETED can not be set until the current time is after lastUpdate plus statusUpdateDelay. This gives the time to the controllers to update the status and to trigger any other rollout operation before the SGDbOps is considered completed.
  • Document this behavior in the SGDbOps.spec.(restart|securityUpgrade|minorVersionUpgrade).statusUpdateDelay fields of the CRD.