### Summary There are a lot of 409 errors there, seems like the operator is fighting with some other component to update the resources Currently, I'm on operator version 1.13.0. ### Current Behaviour It seems like the operator is not able to Patch/Put some SGShardedCluster resources. In the operator log I can find this messages ``` 2024-11-24 04:04:31,941 ERROR [io.st.op.conciliation] (SGCluster-ReconciliationLoop) Reconciliation of SGCluster prod.postgres-stackgres-shard0 failed: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.0.0.1:443/api/v1/namespaces/prod/pods/postgres-stackgres-shard0-1?fieldManager=StackGres&force=true. Message: Operation cannot be fulfilled on pods "postgres-stackgres-shard0-1": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=pods, name=postgres-stackgres-shard0-1, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on pods "postgres-stackgres-shard0-1": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:764) at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:231) at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:236) at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:261) at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:44) at io.fabric8.kubernetes.client.extension.ResourceAdapter.patch(ResourceAdapter.java:195) at io.stackgres.operator.conciliation.AbstractReconciliationHandler.patch(AbstractReconciliationHandler.java:49) at io.stackgres.operator.conciliation.cluster.ClusterDefaultReconciliationHandler_ClientProxy.patch(Unknown Source) at io.stackgres.operator.conciliation.AbstractStatefulSetReconciliationHandler.lambda$fixPods$28(AbstractStatefulSetReconciliationHandler.java:446) at java.base@21.0.4/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base@21.0.4/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base@21.0.4/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base@21.0.4/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base@21.0.4/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base@21.0.4/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939) at java.base@21.0.4/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base@21.0.4/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base@21.0.4/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base@21.0.4/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base@21.0.4/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base@21.0.4/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at org.jooq.lambda.SeqImpl.forEach(SeqImpl.java:151) at io.stackgres.operator.conciliation.AbstractStatefulSetReconciliationHandler.fixPods(AbstractStatefulSetReconciliationHandler.java:446) at io.stackgres.operator.conciliation.AbstractStatefulSetReconciliationHandler.concileSts(AbstractStatefulSetReconciliationHandler.java:238) at io.stackgres.operator.conciliation.AbstractStatefulSetReconciliationHandler.patch(AbstractStatefulSetReconciliationHandler.java:122) at io.stackgres.operator.conciliation.cluster.ClusterStatefulSetReconciliationHandler_ClientProxy.patch(Unknown Source) at io.stackgres.operator.conciliation.cluster.ClusterHandlerDelegator.patch(ClusterHandlerDelegator.java:43) at io.stackgres.operator.conciliation.cluster.ClusterHandlerDelegator.patch(ClusterHandlerDelegator.java:19) at io.stackgres.operator.conciliation.cluster.ClusterHandlerDelegator_ClientProxy.patch(Unknown Source) at io.stackgres.operator.conciliation.AbstractReconciliator.lambda$reconciliationCycle$10(AbstractReconciliator.java:249) at java.base@21.0.4/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base@21.0.4/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) at java.base@21.0.4/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510) at java.base@21.0.4/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base@21.0.4/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base@21.0.4/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base@21.0.4/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base@21.0.4/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at io.stackgres.operator.conciliation.AbstractReconciliator.reconciliationCycle(AbstractReconciliator.java:243) at io.stackgres.operator.conciliation.cluster.ClusterReconciliator.reconciliationCycle(ClusterReconciliator.java:85) at io.stackgres.operator.conciliation.cluster.ClusterReconciliator.reconciliationCycle(ClusterReconciliator.java:38) at io.stackgres.operator.conciliation.AbstractReconciliator.lambda$reconciliationsCycle$5(AbstractReconciliator.java:161) at java.base@21.0.4/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base@21.0.4/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base@21.0.4/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) at java.base@21.0.4/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base@21.0.4/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base@21.0.4/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base@21.0.4/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base@21.0.4/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base@21.0.4/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at io.stackgres.operator.conciliation.AbstractReconciliator.reconciliationsCycle(AbstractReconciliator.java:161) at io.stackgres.operator.conciliation.AbstractReconciliator.reconciliationLoop(AbstractReconciliator.java:145) at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base@21.0.4/java.lang.Thread.runWith(Thread.java:1596) at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583) at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:896) at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:872) Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.0.0.1:443/api/v1/namespaces/prod/pods/postgres-stackgres-shard0-1?fieldManager=StackGres&force=true. Message: Operation cannot be fulfilled on pods "postgres-stackgres-shard0-1": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=pods, name=postgres-stackgres-shard0-1, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on pods "postgres-stackgres-shard0-1": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:660) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:640) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:589) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549) at java.base@21.0.4/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) at java.base@21.0.4/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base@21.0.4/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142) at java.base@21.0.4/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base@21.0.4/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base@21.0.4/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base@21.0.4/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51) at java.base@21.0.4/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base@21.0.4/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base@21.0.4/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base@21.0.4/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) at io.fabric8.kubernetes.client.vertx.VertxHttpRequest.lambda$null$1(VertxHttpRequest.java:121) at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279) at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261) at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76) at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:421) at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279) at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:157) at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:709) at io.vertx.core.impl.ContextImpl.lambda$execute$4(ContextImpl.java:315) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 4 more ``` ### Expected Behaviour Patching the version of SGShardedCluster should be done without errors. Added all logs file and yaml of SGCluster and SGShardedCluster. Also in attachment logs from the operator. [sgclusters.postgres-stackgres.watch.log](/uploads/08417a523d79cc4556965f853f11292d/sgclusters.postgres-stackgres.watch.log) [sgshardedcluster.postgres-stackgres.watch.log](/uploads/f4fdc1c32dbd89794f9d57490f4f9cdc/sgshardedcluster.postgres-stackgres.watch.log)[postgres-stackgres-operator__1_.log](/uploads/0ce72b6acfa38e7e2d355ffbd2e7b976/postgres-stackgres-operator__1_.log)