Prevent failure when patching the cluster StatefulSet
Summary
Some changes in SGCluster
make patch of StatedulSet to fail.
Current Behaviour
The operator throws an KubernetesClientException
that states that changes other than .spec.replicas
, .spec.template
and .spec.updateStrategy
are forbidden.
Steps to reproduce
- Create a cluster with
.spec.pods.persistentVolume.size
of1Gi
- Edit the cluster with
.spec.pods.persistentVolume.size
of2Gi
Expected Behaviour
The StatefulSet is reconcilied without errors.
Possible Solution
The io.stackgres.operatorframework.resource.AbstractResourceHandler
should be specialized for the resource of type StatefulSet
that deletes leaving orphans (so that Pods are not removed) and re-creates the StatefulSet in case of a change ther than .spec.replicas
, .spec.template
and .spec.updateStrategy
is issued.
Environment
- StackGres version: 1.0.0-alpha4
- Kubernetes version: 1.16.9 (but should affect any version)
- Cloud provider or hardware configuration: ?
Relevant logs and/or screenshots
2021-06-14 13:49:28,666 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) <-- END HTTP (45600-byte body)
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) --> PATCH https://10.96.0.1/apis/apps/v1/namespaces/default/statefulsets/stackgres h2
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) Content-Type: application/json-patch+json; charset=utf-8
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) Content-Length: 2989
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Im85YklxbVJDSndsWTBtNTMtUTJydUdIMlZ1QWhpVm9lVUJWWUNmNHg1Sk0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3V
udC9uYW1lc3BhY2UiOiJzdGFja2dyZXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3RhY2tncmVzLW9wZXJhdG9yLXRva2VuLTVkc2p2Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InN0YWNrZ3Jlcy1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc
2VydmljZS1hY2NvdW50LnVpZCI6IjFhY2FjOGFkLTY0MTktNGYxOS04YWY5LWRkYTkxZDFhMDcwMCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpzdGFja2dyZXM6c3RhY2tncmVzLW9wZXJhdG9yIn0.KAXOA3GvF_WmgAPYn7eo30W5GYEwn97iUi41NOVFg-lGpVN1_m6T6q82B0DYNU7jcptuXOryRG3-AX_6ngqYkwnmbHEwZeDkmwjpPj0Z5dqgYmQfigs
z9CspkxdHp13HyWioGEN8hed91Ko3XBkl0SluULkyvffh-YGtoA7STmiqFqdf9I7-sczH3Sx-EsVpyU2txdKEWEobVGaziwUdV4Bhu8KoevUAWf6NVyk2HQrr7jsIXP2Q95fpus9TxsnGZTrYz6sSNuDxA2BwT7HlbHVpFec9qr6VbZe2BvPfdzRdRGH4jLTiF3EylysrrZhkfRMvBnzV8TmUEVJTwAhJgQ
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) Host: 10.96.0.1
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) Connection: Keep-Alive
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) Accept-Encoding: gzip
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) User-Agent: okhttp/3.14.9
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle)
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) [{"op":"replace","path":"/spec/replicas","value":3},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/0/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/
0/volumeMounts/1/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/2/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/7/readOnly","value":false},{"op":"add","path":"/spec/template/spec/contai
ners/0/volumeMounts/8/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/9/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/10/readOnly","value":false},{"op":"add","path":"/spec/template/spec/
containers/0/volumeMounts/11/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/12/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/13/readOnly","value":false},{"op":"add","path":"/spec/templa
te/spec/containers/0/volumeMounts/14/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/15/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/16/readOnly","value":false},{"op":"add","path":"/spe
c/template/spec/containers/0/volumeMounts/17/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/18/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/19/readOnly","value":false},{"op":"add","pat
h":"/spec/template/spec/containers/2/volumeMounts/0/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/3/volumeMounts/0/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/4/volumeMounts/0/readOnly","value":false},{"op":"add"
,"path":"/spec/template/spec/containers/5/volumeMounts/0/readOnly","value":false},{"op":"add","path":"/spec/template/spec/containers/5/volumeMounts/1/readOnly","value":false},{"op":"add","path":"/spec/template/spec/initContainers/0/volumeMounts/0/readOnly","value":false},{"
op":"add","path":"/spec/template/spec/initContainers/0/volumeMounts/1/readOnly","value":false},{"op":"add","path":"/spec/template/spec/initContainers/1/volumeMounts/0/readOnly","value":false},{"op":"add","path":"/spec/template/spec/initContainers/1/volumeMounts/1/readOnly",
"value":false},{"op":"add","path":"/spec/template/spec/initContainers/2/volumeMounts/0/readOnly","value":false},{"op":"add","path":"/spec/template/spec/initContainers/2/volumeMounts/1/readOnly","value":false},{"op":"add","path":"/spec/template/spec/initContainers/3/volumeMo
unts/0/readOnly","value":false},{"op":"add","path":"/spec/template/spec/initContainers/3/volumeMounts/1/readOnly","value":false},{"op":"add","path":"/spec/template/spec/initContainers/4/volumeMounts/0/readOnly","value":false},{"op":"replace","path":"/spec/volumeClaimTemplat
es/0/spec/resources/requests/storage","value":"1Gi"}]
2021-06-14 13:49:28,668 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) --> END PATCH (2989-byte body)
2021-06-14 13:49:28,694 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) <-- 422 https://10.96.0.1/apis/apps/v1/namespaces/default/statefulsets/stackgres (25ms)
2021-06-14 13:49:28,694 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) cache-control: no-cache, private
2021-06-14 13:49:28,694 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) content-type: application/json
2021-06-14 13:49:28,694 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) content-length: 538
2021-06-14 13:49:28,694 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) date: Mon, 14 Jun 2021 13:49:28 GMT
2021-06-14 13:49:28,694 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle)
2021-06-14 13:49:28,694 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"StatefulSet.apps \"stackgres\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replic
as', 'template', and 'updateStrategy' are forbidden","reason":"Invalid","details":{"name":"stackgres","group":"apps","kind":"StatefulSet","causes":[{"reason":"FieldValueForbidden","message":"Forbidden: updates to statefulset spec for fields other than 'replicas', 'template'
, and 'updateStrategy' are forbidden","field":"spec"}]},"code":422}
2021-06-14 13:49:28,694 INFO [ok.OkHttpClient] (Cluster-ReconciliationCycle) <-- END HTTP (538-byte body)
021-06-14 13:37:38,693 ERROR [io.st.op.co.ClusterReconciliationCycle] (Cluster-ReconciliationCycle) StackGres Cluster default.stackgres reconciliation failed: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.96.0.1/apis/apps/v
1/namespaces/default/statefulsets/stackgres. Message: StatefulSet.apps "stackgres" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden. Received status: Status(apiVersion=v1, code=422, det
ails=StatusDetails(causes=[StatusCause(field=spec, message=Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden, reason=FieldValueForbidden, additionalProperties={})], group=apps, kind=StatefulSet, name=stac
kgres, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=StatefulSet.apps "stackgres" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden, metadata=ListMeta(
_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:583)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:522)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:487)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:448)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handlePatch(OperationSupport.java:325)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handlePatch(BaseOperation.java:880)
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$patch$1(HasMetadataOperation.java:120)
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:125)
at io.fabric8.kubernetes.client.dsl.internal.apps.v1.RollableScalableResourceOperation.patch(RollableScalableResourceOperation.java:169)
at io.fabric8.kubernetes.client.dsl.internal.apps.v1.RollableScalableResourceOperation.patch(RollableScalableResourceOperation.java:42)
at io.stackgres.operatorframework.resource.AbstractResourceHandler.patch(AbstractResourceHandler.java:62)
at io.stackgres.operator.cluster.handler.ClusterStatefulSetHandler_ClientProxy.patch(ClusterStatefulSetHandler_ClientProxy.zig:294)
at io.stackgres.operatorframework.resource.AbstractResourceHandlerSelector.patch(AbstractResourceHandlerSelector.java:70)
at io.stackgres.operator.resource.ClusterResourceHandlerSelector_ClientProxy.patch(ClusterResourceHandlerSelector_ClientProxy.zig:200)
at io.stackgres.operatorframework.reconciliation.ResourceGeneratorReconciliator.createOrUpdateRequiredResources(ResourceGeneratorReconciliator.java:153)
at io.stackgres.operatorframework.reconciliation.ResourceGeneratorReconciliator.reconcile(ResourceGeneratorReconciliator.java:55)
at io.stackgres.operatorframework.reconciliation.ResourceGeneratorReconciliator.reconcile(ResourceGeneratorReconciliator.java:22)
at io.stackgres.operator.controller.ClusterReconciliator_ClientProxy.reconcile(ClusterReconciliator_ClientProxy.zig:467)
at io.stackgres.operatorframework.reconciliation.ReconciliationCycle.reconciliationCycle(ReconciliationCycle.java:158)
at io.stackgres.operatorframework.reconciliation.ReconciliationCycle.reconciliationCycleLoop(ReconciliationCycle.java:88)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)