Cannot upgrade cluster to 16.2 since version 1.9.0
Hi,
I'm unable to upgrade an existing 16.1 cluster to version 16.2.
Although version 16.2 is available as an option and is listed as a new feature in the release notes of 1.9.0, trying to upgrade a cluster results in rejection by the operator.
Stack trace:
2024-05-06 03:47:07,614 ERROR [io.st.op.mu.ClusterMutationResource] (executor-thread-28) cannot proceed with request 3cfa60dd-0351-42d7-8f2b-c1a8196f7ea7 cause: postgresql version 16.2 not available: java.lang.IllegalArgumentException: postgresql version 16.2 not available
at io.stackgres.common.component.Component.lambda$getVersion$19(Component.java:424)
at java.base@21.0.2/java.util.Optional.orElseThrow(Optional.java:403)
at io.stackgres.common.component.Component.getVersion(Component.java:424)
at io.stackgres.operator.mutation.cluster.DefaultPostgresVersionMutator.mutate(DefaultPostgresVersionMutator.java:33)
at io.stackgres.operator.mutation.cluster.DefaultPostgresVersionMutator.mutate(DefaultPostgresVersionMutator.java:19)
at io.stackgres.operator.mutation.cluster.DefaultPostgresVersionMutator_ClientProxy.mutate(Unknown Source)
at io.stackgres.operatorframework.admissionwebhook.mutating.MutationPipeline.mutate(MutationPipeline.java:25)
at io.stackgres.operator.mutation.cluster.ClusterPipeline_ClientProxy.mutate(Unknown Source)
at io.stackgres.operatorframework.admissionwebhook.mutating.AbstractMutationResource.mutate(AbstractMutationResource.java:64)
at io.stackgres.operatorframework.admissionwebhook.mutating.AbstractMutationResource.mutate(AbstractMutationResource.java:43)
at io.stackgres.operator.mutation.ClusterMutationResource.mutate(ClusterMutationResource.java:49)
at java.base@21.0.2/java.lang.reflect.Method.invoke(Method.java:580)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:154)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:118)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:560)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:452)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:413)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:415)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:378)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:356)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:70)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:429)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:240)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:154)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:157)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:229)
at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:82)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:147)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
at java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:833)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:211)
And operator first log lines:
_____ _ _ _____
/ ____| | | | / ____|
| (___ | |_ __ _ ___| | _| | __ _ __ ___ ___
\___ \| __/ _` |/ __| |/ / | |_ | '__/ _ \/ __|
____) | || (_| | (__| <| |__| | | | __/\__ \
|_____/ \__\__,_|\___|_|\_\\_____|_| \___||___/
by OnGres, Inc.
Powered by Quarkus 3.6.9
2024-05-06 03:14:07,263 WARN [io.fa.ku.cl.ds.in.VersionUsageUtils] (main) The client is using resource type 'sgobjectstorages' with unstable version 'v1beta1'
2024-05-06 03:14:07,422 WARN [io.fa.ku.cl.ds.in.VersionUsageUtils] (main) The client is using resource type 'sgshardedclusters' with unstable version 'v1alpha1'
2024-05-06 03:14:07,464 INFO [io.st.op.ap.CertificateInstaller] (main) Installing Certificate
2024-05-06 03:14:07,560 INFO [io.st.op.ap.CertificateInstaller] (main) Certificate already installed
2024-05-06 03:14:07,560 INFO [io.st.op.ap.CertificateInstaller] (main) Waiting for certificate
2024-05-06 03:14:07,605 INFO [io.st.op.ap.CertificateInstaller] (main) Certificate available
2024-05-06 03:14:07,605 INFO [io.st.op.ap.StackGresOperatorApp] (OperatorStartup) The operator is starting...
2024-05-06 03:14:07,606 INFO [io.st.op.mu.ShardedBackupMutationResource] (main) Sharded Backup mutation resource started
2024-05-06 03:14:07,606 INFO [io.st.op.mu.DistributedLogsMutationResource] (main) Distributed logs mutation resource started
2024-05-06 03:14:07,607 INFO [io.st.op.co.SgShardedClusterConversionResource] (main) SgShardedCluster configuration conversion resource started
2024-05-06 03:14:07,608 INFO [io.st.op.co.SgPostgresConfigConversionResource] (main) SGPostgresConfig configuration conversion resource started
2024-05-06 03:14:07,608 INFO [io.st.op.co.SgPoolingConfigConversionResource] (main) SGPoolingConfig configuration conversion resource started
2024-05-06 03:14:07,607 INFO [io.st.op.conciliation] (SGBackup-ReconciliationLoop) SGBackup reconciliation loop started
2024-05-06 03:14:07,608 INFO [io.st.op.conciliation] (SGConfig-ReconciliationLoop) SGConfig reconciliation loop started
2024-05-06 03:14:07,609 INFO [io.st.op.va.PgBouncerValidationResource] (main) Pgbouncer validation resource started
2024-05-06 03:14:07,609 INFO [io.st.op.conciliation] (SGDbOps-ReconciliationLoop) SGDbOps reconciliation loop started
2024-05-06 03:14:07,610 INFO [io.st.op.mu.ClusterMutationResource] (main) Cluster configuration mutator resource started
2024-05-06 03:14:07,610 INFO [io.st.op.co.SgInstanceProfileConversionResource] (main) SGInstanceProfile configuration conversion resource started
2024-05-06 03:14:07,610 INFO [io.st.op.co.SgClusterConversionResource] (main) SgCluster configuration conversion resource started
2024-05-06 03:14:07,610 INFO [io.st.op.co.SgBackupConversionResource] (main) SGBackup configuration conversion resource started
2024-05-06 03:14:07,611 INFO [io.st.op.va.ClusterValidationResource] (main) Cluster validation resource started
2024-05-06 03:14:07,611 INFO [io.st.op.ap.DefaultOperatorBootstrap] (OperatorStartup) Kubernetes version: v1.29.3-eks-adc7111
2024-05-06 03:14:07,611 INFO [io.st.op.ap.DefaultOperatorBootstrap] (OperatorStartup) URL of this Kubernetes cluster: https://10.100.0.1:443/
2024-05-06 03:14:07,611 INFO [io.st.op.app] (OperatorStartup) Starting Operator Lock Reconciliation
2024-05-06 03:14:07,612 INFO [io.st.op.va.SgProfileValidationResource] (main) SgProfile validation resource started
2024-05-06 03:14:07,612 INFO [io.st.op.conciliation] (SGShardedDbOps-ReconciliationLoop) SGShardedDbOps reconciliation loop started
2024-05-06 03:14:07,613 INFO [io.st.op.co.SgConfigConversionResource] (main) SgConfig configuration conversion resource started
2024-05-06 03:14:07,613 INFO [io.st.op.conciliation] (SGScript-ReconciliationLoop) SGScript reconciliation loop started
2024-05-06 03:14:07,613 INFO [io.st.op.va.ShardedDbOpsValidationResource] (main) SGShardedDbOps validation resource started
2024-05-06 03:14:07,613 INFO [io.st.op.va.BackupValidationResource] (main) Backup validation resource started
2024-05-06 03:14:07,614 INFO [io.st.op.mu.ObjectStorageMutationResource] (main) Object Storage mutation resource started
2024-05-06 03:14:07,614 INFO [io.st.op.mu.SgPgBouncerMutationResource] (main) PgBouncer configuration mutation resource started
2024-05-06 03:14:07,614 INFO [io.st.op.conciliation] (SGShardedBackup-ReconciliationLoop) SGShardedBackup reconciliation loop started
2024-05-06 03:14:07,615 INFO [io.st.op.va.DbOpsValidationResource] (main) DbOps validation resource started
2024-05-06 03:14:07,615 INFO [io.st.op.va.ShardedClusterValidationResource] (main) Sharded Cluster validation resource started
2024-05-06 03:14:07,615 INFO [io.st.op.conciliation] (SGShardedCluster-ReconciliationLoop) SGShardedCluster reconciliation loop started
2024-05-06 03:14:07,616 INFO [io.st.op.mu.ShardedClusterMutationResource] (main) ShardedCluster mutation resource started
2024-05-06 03:14:07,616 INFO [io.st.op.va.PostgresConfigValidationResource] (main) Postgres configuration validation resource started
2024-05-06 03:14:07,616 INFO [io.st.op.co.SgObjectStorageConversionResource] (main) SgObjectStorage configuration conversion resource started
2024-05-06 03:14:07,616 INFO [io.st.op.va.ScriptValidationResource] (main) Script validation resource started
2024-05-06 03:14:07,617 INFO [io.st.op.mu.PgConfigMutationResource] (main) Postgres configuration mutation resource started
2024-05-06 03:14:07,617 INFO [io.st.op.mu.ConfigMutationResource] (main) Config mutation resource started
2024-05-06 03:14:07,617 INFO [io.st.op.va.ShardedBackupValidationResource] (main) SGShardedBackup validation resource started
2024-05-06 03:14:07,617 INFO [io.st.op.va.ConfigValidationResource] (main) Config validation resource started
2024-05-06 03:14:07,617 INFO [io.st.op.mu.SgProfileMutationResource] (main) SgProfile configuration mutation resource started
2024-05-06 03:14:07,618 INFO [io.st.op.mu.ShardedDbOpsMutationResource] (main) Sharded DbOps mutation resource started
2024-05-06 03:14:07,618 INFO [io.st.op.va.DistributedLogsValidationResource] (main) SgProfile validation resource started
2024-05-06 03:14:07,618 INFO [io.st.op.va.ObjectStorageValidationResource] (main) Object storage validation resource started
2024-05-06 03:14:07,618 INFO [io.st.op.mu.ScriptMutationResource] (main) Script mutation resource started
2024-05-06 03:14:07,619 INFO [io.st.op.mu.BackupMutationResource] (main) Backup mutation resource started
2024-05-06 03:14:07,619 INFO [io.st.op.conciliation] (SGDistributedLogs-ReconciliationLoop) SGDistributedLogs reconciliation loop started
2024-05-06 03:14:07,620 INFO [io.st.op.co.SgDistributedLogsConversionResource] (main) SGDistributedLogs configuration conversion resource started
2024-05-06 03:14:07,620 INFO [io.st.op.conciliation] (SGCluster-ReconciliationLoop) SGCluster reconciliation loop started
2024-05-06 03:14:07,620 INFO [io.st.op.mu.DbOpsMutationResource] (main) DbOps mutation resource started
2024-05-06 03:14:07,621 INFO [io.st.op.co.SgDbOpsConversionResource] (main) SGDbOps configuration conversion resource started
2024-05-06 03:14:07,631 WARN [io.st.op.app] (OperatorLockHolder) Waiting for the lock on SGConfig to be released
2024-05-06 03:14:07,820 INFO [io.quarkus] (main) stackgres-operator 1.10.0 native (powered by Quarkus 3.6.9) started in 1.453s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443
2024-05-06 03:14:07,820 INFO [io.quarkus] (main) Profile prod activated.
2024-05-06 03:14:07,820 INFO [io.quarkus] (main) Installed features: [cdi, hibernate-validator, kubernetes-client, rest-client, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, vertx]
2024-05-06 03:14:12,679 INFO [io.st.op.app] (OperatorLockHolder) Lock on SGConfig was set. I am the leader!
2024-05-06 03:14:12,730 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRDs
2024-05-06 03:14:13,093 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgconfigs.stackgres.io
2024-05-06 03:14:13,192 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgclusters.stackgres.io
2024-05-06 03:14:13,548 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sginstanceprofiles.stackgres.io
2024-05-06 03:14:13,604 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgpgconfigs.stackgres.io
2024-05-06 03:14:13,632 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgpoolconfigs.stackgres.io
2024-05-06 03:14:13,671 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgbackups.stackgres.io
2024-05-06 03:14:13,760 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgdistributedlogs.stackgres.io
2024-05-06 03:14:13,852 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgdbops.stackgres.io
2024-05-06 03:14:13,984 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgobjectstorages.stackgres.io
2024-05-06 03:14:14,047 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgscripts.stackgres.io
2024-05-06 03:14:14,089 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgshardedclusters.stackgres.io
2024-05-06 03:14:15,427 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgshardedbackups.stackgres.io
2024-05-06 03:14:15,596 INFO [io.st.op.ap.CrdInstaller] (OperatorStartup) Installing CRD sgshardeddbops.stackgres.io
2024-05-06 03:14:16,724 INFO [io.st.op.ap.CrdWebhookInstaller] (OperatorStartup) Installing Conversion Webhooks
2024-05-06 03:14:19,337 INFO [io.st.op.ap.CrdWebhookInstaller] (OperatorStartup) Installing Mutating Webhooks
2024-05-06 03:14:19,400 INFO [io.st.op.ap.CrdWebhookInstaller] (OperatorStartup) Installing Validating Webhooks