Skip to content

Operator admissionwebhook validation fail

I have this weird issue where I cannot apply any changes or even kubectl diff a SGCluster.yaml definition to any clusters. For example, if I run kubectl diff -f myCluster.yaml for a cluster that exists, I get the following error from Stackgres Operator:

Error from server (https://stackgres.io/doc/1.16/api/responses/error#forbidden-configuration-update): error when patching "myCluster.yaml": admission webhook "sgcluster.validating-webhook.stackgres.io" denied the request: to upgrade a minor Postgres version, please create an SGDbOps operation with "op: minorversionUpgrade" and the target postgres version.

The odd thing is that when I run kubectl diff, it is on a SGCluster definition that is exactly the same as the currently installed/deployed one. I am not at all attempting to do a minor version upgrade. Moreover, this is the full stacktrace from the operator pod:

ERROR [io.st.op.va.ClusterValidationResource_Subclass] (executor-thread-1268) cannot proceed with request ac109ca1-7865-4bf8-88da-ec0a351cb3c6 cause: to upgrade a minor Postgres version, please create an SGDbOps operation with "op: minorVersionUpgrade" and the target version.: io.stackgres.operatorframework.admissionwebhook.validating.ValidationFailed: to upgrade a minor Postgres version, please create an SGDbOps operation with "op: minorVersionUpgrade" and the target postgres version.
    at io.stackgres.operatorframework.admissionwebhook.validating.Validator.fail(Validator.java:106)
    at io.stackgres.operator.validation.cluster.ClusterValidator.fail(ClusterValidator.java:19)
    at io.stackgres.operator.validation.cluster.PostgresConfigValidator.validate(PostgresConfigValidator.java:157)
    at io.stackgres.operator.validation.cluster.PostgresConfigValidator.validate(PostgresConfigValidator.java:35)
    at io.stackgres.operatorframework.admissionwebhook.validating.ValidationPipeline.validate(ValidationPipeline.java:23)
    at io.stackgres.operator.validation.AbstractValidationPipeline.validate(AbstractValidationPipeline.java:60)
    at io.stackgres.operator.validation.cluster.ClusterValidationPipeline_ClientProxy.validate(Unknown_Source)
    at io.stackgres.operatorframework.admissionwebhook.validating.ValidationResource.validate(ValidationResource.java:60)
    at io.stackgres.operatorframework.admissionwebhook.validating.ValidationResource.validate(ValidationResource.java:37)
    at io.stackgres.operator.validation.ClusterValidationResource.validate(ClusterValidationResource.java:49)
    at io.stackgres.operator.validation.ClusterValidationResource_Subclass.validate$$superforward(Unknown_Source)
    at io.stackgres.operator.validation.ClusterValidationResource_Subclass$$function$$1.apply(Unknown_Source)
    at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
    at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
    at io.quarkus.resteasy.runtime.QuarkusRestPathTemplateInterceptor.restMethodInvoke(QuarkusRestPathTemplateInterceptor.java:39)
    at io.quarkus.resteasy.runtime.QuarkusRestPathTemplateInterceptor_Bean.intercept(Unknown Source)
    at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
    at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
    at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
    at io.stackgres.operator.validation.ClusterValidationResource_Subclass.validate(Unknown Source)
    at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:154)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:118)
    at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:560)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:452)
    at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:413)
    at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:415)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:415)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:378)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:356)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:70)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:429)
    at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:240)
    at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:154)
    at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)
    at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:157)
    at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:84)
    at io.quarkus.resteasy.runtime.standalone.VertextRequestHandler.dispatch(VertextRequestHandler.java:151)
    at io.quarkus.resteasy.runtime.standalone.VertextRequestHandler$1.run(VertextRequestHandler.java:97)
    at io.quarkus.vertx.core.runtime.VertexCoreRecorder$14.runWith(VertexCoreRecorder.java:635)
    at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
    at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
    at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base@21.0.4/java.lang.Thread.runWith(Thread.java:1596)
    at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:896)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:872)

For the record, this probably happened after I upgraded my Stackgres operator from 1.14.1 -> 1.16.3. I have already applied security upgrade to all my Postgres cluster, so that kubectl get sgclusters --all-namespaces -o jq | jq '.items[].metadata.annotations."stackgres.io/operatorVersion"' returns "1.16.3" for all clusters.

I have not attempted to uninstall and re-install the Stackgres operator. Is this safe to do, when I have three Postgres clusters running? I cannot risk losing any data. If so, is it as simple as doing a helm uninstall [...] and then helm upgrade --install [...]?

Stackgres operator version: 1.16.3

Postgres cluster version: 16.4

K8s version: v1.30.9+k3s1

Any help here is appreciated!

Edited by Monochrome Muncher