Operator admissionwebhook validation fail
I have this weird issue where I cannot apply any changes or even kubectl diff a SGCluster.yaml definition to any clusters. For example, if I run kubectl diff -f myCluster.yaml for a cluster that exists, I get the following error from Stackgres Operator:
Error from server (https://stackgres.io/doc/1.16/api/responses/error#forbidden-configuration-update): error when patching "myCluster.yaml": admission webhook "sgcluster.validating-webhook.stackgres.io" denied the request: to upgrade a minor Postgres version, please create an SGDbOps operation with "op: minorversionUpgrade" and the target postgres version.
The odd thing is that when I run kubectl diff, it is on a SGCluster definition that is exactly the same as the currently installed/deployed one. I am not at all attempting to do a minor version upgrade. Moreover, this is the full stacktrace from the operator pod:
ERROR [io.st.op.va.ClusterValidationResource_Subclass] (executor-thread-1268) cannot proceed with request ac109ca1-7865-4bf8-88da-ec0a351cb3c6 cause: to upgrade a minor Postgres version, please create an SGDbOps operation with "op: minorVersionUpgrade" and the target version.: io.stackgres.operatorframework.admissionwebhook.validating.ValidationFailed: to upgrade a minor Postgres version, please create an SGDbOps operation with "op: minorVersionUpgrade" and the target postgres version.
at io.stackgres.operatorframework.admissionwebhook.validating.Validator.fail(Validator.java:106)
at io.stackgres.operator.validation.cluster.ClusterValidator.fail(ClusterValidator.java:19)
at io.stackgres.operator.validation.cluster.PostgresConfigValidator.validate(PostgresConfigValidator.java:157)
at io.stackgres.operator.validation.cluster.PostgresConfigValidator.validate(PostgresConfigValidator.java:35)
at io.stackgres.operatorframework.admissionwebhook.validating.ValidationPipeline.validate(ValidationPipeline.java:23)
at io.stackgres.operator.validation.AbstractValidationPipeline.validate(AbstractValidationPipeline.java:60)
at io.stackgres.operator.validation.cluster.ClusterValidationPipeline_ClientProxy.validate(Unknown_Source)
at io.stackgres.operatorframework.admissionwebhook.validating.ValidationResource.validate(ValidationResource.java:60)
at io.stackgres.operatorframework.admissionwebhook.validating.ValidationResource.validate(ValidationResource.java:37)
at io.stackgres.operator.validation.ClusterValidationResource.validate(ClusterValidationResource.java:49)
at io.stackgres.operator.validation.ClusterValidationResource_Subclass.validate$$superforward(Unknown_Source)
at io.stackgres.operator.validation.ClusterValidationResource_Subclass$$function$$1.apply(Unknown_Source)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
at io.quarkus.resteasy.runtime.QuarkusRestPathTemplateInterceptor.restMethodInvoke(QuarkusRestPathTemplateInterceptor.java:39)
at io.quarkus.resteasy.runtime.QuarkusRestPathTemplateInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
at io.stackgres.operator.validation.ClusterValidationResource_Subclass.validate(Unknown Source)
at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:154)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:118)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:560)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:452)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:413)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:415)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:415)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:378)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:356)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:70)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:429)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:240)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:154)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:157)
at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:84)
at io.quarkus.resteasy.runtime.standalone.VertextRequestHandler.dispatch(VertextRequestHandler.java:151)
at io.quarkus.resteasy.runtime.standalone.VertextRequestHandler$1.run(VertextRequestHandler.java:97)
at io.quarkus.vertx.core.runtime.VertexCoreRecorder$14.runWith(VertexCoreRecorder.java:635)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base@21.0.4/java.lang.Thread.runWith(Thread.java:1596)
at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:896)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:872)
For the record, this probably happened after I upgraded my Stackgres operator from 1.14.1 -> 1.16.3. I have already applied security upgrade to all my Postgres cluster, so that kubectl get sgclusters --all-namespaces -o jq | jq '.items[].metadata.annotations."stackgres.io/operatorVersion"' returns "1.16.3" for all clusters.
I have not attempted to uninstall and re-install the Stackgres operator. Is this safe to do, when I have three Postgres clusters running? I cannot risk losing any data. If so, is it as simple as doing a helm uninstall [...] and then helm upgrade --install [...]?
Stackgres operator version: 1.16.3
Postgres cluster version: 16.4
K8s version: v1.30.9+k3s1
Any help here is appreciated!