Scheduled reconciliation stop working
Summary
At some point scheduled reconciliation that is performed each 10 seconds is no more performed.
Current Behavior
Reconciliation is only performed on CR changes
Steps to reproduce
This is not deterministically reproducible but has seen appear after some time the operator is running
Expected Behavior
The reconciliation cycle is repeated al least each 10 seconds
Possible Solution
Method io.stackgres.operator.app.ReconciliationClockImpl.reconcile
is called from java.util.concurrent.ScheduledExecutorService.scheduleAtFixedRate
and, as stated by Javadoc:
The sequence of task executions continues indefinitely until one of the following exceptional completions occur:
The task is explicitly cancelled via the returned future. The executor terminates, also resulting in task cancellation. An execution of the task throws an exception. In this case calling get on the returned future will throw ExecutionException, holding the exception as its cause. Subsequent executions are suppressed. Subsequent calls to isDone() on the returned future will return true.
Probably a try catch in method io.stackgres.operator.app.ReconciliationClockImpl.reconcile
with proper logging of error will solve the problem.
Environment
- StackGres version: 1.0.0-beta2
- Kubernetes version: ?
- Cloud provider or hardware configuration: ?
Relevant logs and/or screenshots
2021-07-23 10:23:21,573 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Cluster dbops-security-upgrade-60fa88bc/dbops-security-upgrade-1 it's not up to date. Reconciling
2021-07-23 10:23:21,573 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Patching resource dbops-security-upgrade-1-patroni of kind: ServiceAccount
2021-07-23 10:23:21,587 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Patching resource dbops-security-upgrade-1-dbops of kind: ServiceAccount
2021-07-23 10:23:21,599 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Patching resource dbops-security-upgrade-1-security-upgrade-13609655f35d40be-0 of kind: Job
2021-07-23 10:23:21,622 ERROR [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Reconciliation of cluster dbops-security-upgrade-60fa88bc/dbops-security-upgrade-1 failed: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.96.0.1/apis/batch/v1/namespaces/dbops-security-upgrade-60fa88bc/jobs/dbops-security-upgrade-1-security-upgrade-13609655f35d40be-0. Message: Job.batch "dbops-security-upgrade-1-security-upgrade-13609655f35d40be-0" is invalid: [spec.selector: Required value, spec.template.metadata.labels: Invalid value: map[string]string{"app":"StackGresCluster", "cluster-name":"dbops-security-upgrade-1", "cluster-uid":"bd94659e-277f-4a68-9595-599bb2275452", "db-ops":"true"}: `selector` does not match template `labels`, spec.selector: Invalid value: "null": field is immutable, spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"dbops-security-upgrade-1-securityUpgrade-13609655f35d40be-0", GenerateName:"", Namespace:"dbops-security-upgrade-60fa88bc", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"StackGresCluster", "cluster-name":"dbops-security-upgrade-1", "cluster-uid":"bd94659e-277f-4a68-9595-599bb2275452", "db-ops":"true"}, Annotations:map[string]string{"stackgres.io/operatorVersion":"1.0.0-beta3-SNAPSHOT"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"security-upgrade", Image:"docker.io/stackgres/jobs:2298c7f4-jvm", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"OPERATOR_NAME", Value:"stackgres-operator", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"OPERATOR_NAMESPACE", Value:"stackgres-60fa885e", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"JOB_NAMESPACE", Value:"dbops-security-upgrade-60fa88bc", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"OPERATOR_VERSION", Value:"1.0.0-beta3-SNAPSHOT", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"CRD_UPGRADE", Value:"false", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"CONVERSION_WEBHOOKS", Value:"false", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DATABASE_OPERATION_JOB", Value:"true", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DATABASE_OPERATION_CR_NAME", Value:"dbops-security-upgrade-1", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"POD_NAME", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00e01ca00)}, core.EnvVar{Name:"APP_OPTS", Value:"-Dquarkus.log.category.\"io.stackgres\".level=DEBUG -Dquarkus.log.category.\"io.quarkus\".level=INFO -Dquarkus.log.category.\"io.stackgres.dbops\".level=TRACE -Dquarkus.log.category.\"io.stackgres.backup\".level=TRACE -Dquarkus.log.category.\"io.stackgres.wal-g\".level=TRACE -Dquarkus.log.category.\"io.stackgres.patroni\".level=TRACE -Dquarkus.log.category.\"io.stackgres.fluent-bit\".level=TRACE -Dquarkus.log.category.\"io.stackgres.fluentd\".level=TRACE -Dquarkus.log.category.\"io.stackgres.prometheus-postgres-exporter\".level=TRACE -Dquarkus.log.category.\"okhttp3.logging.HttpLoggingInterceptor\".level=TRACE ", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"JAVA_OPTS", Value:"", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DEBUG_CLUSTER_CONTROLLER", Value:"", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DEBUG_CLUSTER_CONTROLLER_SUSPEND", Value:"", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc01d1210b8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"dbops-security-upgrade-1-dbops", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc01b4b4a80), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable]. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.selector, message=Required value, reason=FieldValueRequired, additionalProperties={}), StatusCause(field=spec.template.metadata.labels, message=Invalid value: map[string]string{"app":"StackGresCluster", "cluster-name":"dbops-security-upgrade-1", "cluster-uid":"bd94659e-277f-4a68-9595-599bb2275452", "db-ops":"true"}: `selector` does not match template `labels`, reason=FieldValueInvalid, additionalProperties={}), StatusCause(field=spec.selector, message=Required value, reason=FieldValueRequired, additionalProperties={}), StatusCause(field=spec.template.metadata.labels, message=Invalid value: map[string]string{"app":"StackGresCluster", "cluster-name":"dbops-security-upgrade-1", "cluster-uid":"bd94659e-277f-4a68-9595-599bb2275452", "db-ops":"true"}: `selector` does not match template `labels`, reason=FieldValueInvalid, additionalProperties={}), StatusCause(field=spec.selector, message=Invalid value: "null": field is immutable, reason=FieldValueInvalid, additionalProperties={}), StatusCause(field=spec.template, message=Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"dbops-security-upgrade-1-securityUpgrade-13609655f35d40be-0", GenerateName:"", Namespace:"dbops-security-upgrade-60fa88bc", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"StackGresCluster", "cluster-name":"dbops-security-upgrade-1", "cluster-uid":"bd94659e-277f-4a68-9595-599bb2275452", "db-ops":"true"}, Annotations:map[string]string{"stackgres.io/operatorVersion":"1.0.0-beta3-SNAPSHOT"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"security-upgrade", Image:"docker.io/stackgres/jobs:2298c7f4-jvm", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"OPERATOR_NAME", Value:"stackgres-operator", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"OPERATOR_NAMESPACE", Value:"stackgres-60fa885e", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"JOB_NAMESPACE", Value:"dbops-security-upgrade-60fa88bc", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"OPERATOR_VERSION", Value:"1.0.0-beta3-SNAPSHOT", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"CRD_UPGRADE", Value:"false", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"CONVERSION_WEBHOOKS", Value:"false", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DATABASE_OPERATION_JOB", Value:"true", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DATABASE_OPERATION_CR_NAME", Value:"dbops-security-upgrade-1", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"POD_NAME", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00e01ca00)}, core.EnvVar{Name:"APP_OPTS", Value:"-Dquarkus.log.category.\"io.stackgres\".level=DEBUG -Dquarkus.log.category.\"io.quarkus\".level=INFO -Dquarkus.log.category.\"io.stackgres.dbops\".level=TRACE -Dquarkus.log.category.\"io.stackgres.backup\".level=TRACE -Dquarkus.log.category.\"io.stackgres.wal-g\".level=TRACE -Dquarkus.log.category.\"io.stackgres.patroni\".level=TRACE -Dquarkus.log.category.\"io.stackgres.fluent-bit\".level=TRACE -Dquarkus.log.category.\"io.stackgres.fluentd\".level=TRACE -Dquarkus.log.category.\"io.stackgres.prometheus-postgres-exporter\".level=TRACE -Dquarkus.log.category.\"okhttp3.logging.HttpLoggingInterceptor\".level=TRACE ", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"JAVA_OPTS", Value:"", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DEBUG_CLUSTER_CONTROLLER", Value:"", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DEBUG_CLUSTER_CONTROLLER_SUSPEND", Value:"", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc01d1210b8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"dbops-security-upgrade-1-dbops", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc01b4b4a80), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable, reason=FieldValueInvalid, additionalProperties={})], group=batch, kind=Job, name=dbops-security-upgrade-1-security-upgrade-13609655f35d40be-0, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Job.batch "dbops-security-upgrade-1-security-upgrade-13609655f35d40be-0" is invalid: [spec.selector: Required value, spec.template.metadata.labels: Invalid value: map[string]string{"app":"StackGresCluster", "cluster-name":"dbops-security-upgrade-1", "cluster-uid":"bd94659e-277f-4a68-9595-599bb2275452", "db-ops":"true"}: `selector` does not match template `labels`, spec.selector: Invalid value: "null": field is immutable, spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"dbops-security-upgrade-1-securityUpgrade-13609655f35d40be-0", GenerateName:"", Namespace:"dbops-security-upgrade-60fa88bc", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"StackGresCluster", "cluster-name":"dbops-security-upgrade-1", "cluster-uid":"bd94659e-277f-4a68-9595-599bb2275452", "db-ops":"true"}, Annotations:map[string]string{"stackgres.io/operatorVersion":"1.0.0-beta3-SNAPSHOT"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"security-upgrade", Image:"docker.io/stackgres/jobs:2298c7f4-jvm", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"OPERATOR_NAME", Value:"stackgres-operator", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"OPERATOR_NAMESPACE", Value:"stackgres-60fa885e", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"JOB_NAMESPACE", Value:"dbops-security-upgrade-60fa88bc", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"OPERATOR_VERSION", Value:"1.0.0-beta3-SNAPSHOT", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"CRD_UPGRADE", Value:"false", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"CONVERSION_WEBHOOKS", Value:"false", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DATABASE_OPERATION_JOB", Value:"true", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DATABASE_OPERATION_CR_NAME", Value:"dbops-security-upgrade-1", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"POD_NAME", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00e01ca00)}, core.EnvVar{Name:"APP_OPTS", Value:"-Dquarkus.log.category.\"io.stackgres\".level=DEBUG -Dquarkus.log.category.\"io.quarkus\".level=INFO -Dquarkus.log.category.\"io.stackgres.dbops\".level=TRACE -Dquarkus.log.category.\"io.stackgres.backup\".level=TRACE -Dquarkus.log.category.\"io.stackgres.wal-g\".level=TRACE -Dquarkus.log.category.\"io.stackgres.patroni\".level=TRACE -Dquarkus.log.category.\"io.stackgres.fluent-bit\".level=TRACE -Dquarkus.log.category.\"io.stackgres.fluentd\".level=TRACE -Dquarkus.log.category.\"io.stackgres.prometheus-postgres-exporter\".level=TRACE -Dquarkus.log.category.\"okhttp3.logging.HttpLoggingInterceptor\".level=TRACE ", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"JAVA_OPTS", Value:"", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DEBUG_CLUSTER_CONTROLLER", Value:"", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"DEBUG_CLUSTER_CONTROLLER_SUSPEND", Value:"", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc01d1210b8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"dbops-security-upgrade-1-dbops", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc01b4b4a80), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable], metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:639)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:578)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:543)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:504)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handlePatch(OperationSupport.java:400)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handlePatch(OperationSupport.java:361)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handlePatch(BaseOperation.java:903)
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$patch$1(HasMetadataOperation.java:176)
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:181)
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:191)
at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:38)
at io.fabric8.kubernetes.client.dsl.Patchable.patch(Patchable.java:35)
at io.stackgres.operator.conciliation.AbstractReconciliationHandler.lambda$patch$1(AbstractReconciliationHandler.java:38)
at io.stackgres.common.KubernetesClientFactory.withNewClient(KubernetesClientFactory.java:18)
at io.stackgres.operator.app.KubernetesClientProvider_ClientProxy.withNewClient(KubernetesClientProvider_ClientProxy.zig:157)
at io.stackgres.operator.conciliation.AbstractReconciliationHandler.patch(AbstractReconciliationHandler.java:35)
at io.stackgres.operator.conciliation.cluster.ClusterDefaultReconciliationHandler_ClientProxy.patch(ClusterDefaultReconciliationHandler_ClientProxy.zig:162)
at io.stackgres.operator.conciliation.cluster.ClusterHandlerDelegator.patch(ClusterHandlerDelegator.java:43)
at io.stackgres.operator.conciliation.cluster.ClusterHandlerDelegator_ClientProxy.patch(ClusterHandlerDelegator_ClientProxy.zig:162)
at io.stackgres.operator.conciliation.StackGresReconciliator.lambda$reconcile$1(StackGresReconciliator.java:65)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at io.stackgres.operator.conciliation.StackGresReconciliator.lambda$reconcile$3(StackGresReconciliator.java:62)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at io.stackgres.operator.conciliation.StackGresReconciliator.reconcile(StackGresReconciliator.java:35)
at io.stackgres.operator.conciliation.cluster.ClusterReconciliator_ClientProxy.reconcile(ClusterReconciliator_ClientProxy.zig:126)
at io.stackgres.operator.app.OperatorWatchersHandlerImpl.lambda$reconcileCluster$3(OperatorWatchersHandlerImpl.java:120)
at io.stackgres.operatorframework.resource.AbstractResourceWatcherFactory$WatcherInstance.eventReceived(AbstractResourceWatcherFactory.java:51)
at io.stackgres.operatorframework.resource.AbstractResourceWatcherFactory$WatcherInstance.eventReceived(AbstractResourceWatcherFactory.java:35)
at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.eventReceived(AbstractWatchManager.java:145)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$TypedWatcherWebSocketListener.onMessage(WatchConnectionManager.java:83)
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:322)
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:273)
at okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:209)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2021-07-23 10:23:21,668 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 55dbd245-5624-4a8b-bfc7-21b6e29874c2
2021-07-23 10:23:21,681 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 2a8180d6-6036-48e7-a8be-38e1f116bcb8
2021-07-23 10:23:21,682 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Checking reconciliation status of cluster dbops-security-upgrade-60fa88bc/dbops-security-upgrade-2
2021-07-23 10:23:21,692 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 6b988adf-d175-4b4a-bc78-839c6c3faff5
2021-07-23 10:23:21,704 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 6514b573-d091-446e-87e9-543ffc8517d9
2021-07-23 10:23:21,717 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 0c439f96-d187-4d8e-b23f-4b9a4bc4aef3
2021-07-23 10:23:21,742 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review be2272ea-7476-4abb-a76c-11a34b362ed3
2021-07-23 10:23:21,753 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 9196cb4c-5476-482e-bd58-f5deadcbeb74
2021-07-23 10:23:21,897 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Cluster dbops-security-upgrade-60fa88bc/dbops-security-upgrade-2 it's not up to date. Reconciling
2021-07-23 10:23:21,897 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Patching resource dbops-security-upgrade-2-patroni of kind: ServiceAccount
2021-07-23 10:23:22,000 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review abe3e578-d8a8-48d7-96a0-dbfd02712e44
2021-07-23 10:23:22,012 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 971e295a-824c-433d-924d-97ca984ca7aa
2021-07-23 10:23:22,014 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Checking reconciliation status of cluster dbops-security-upgrade-60fa88bc/dbops-security-upgrade-3
2021-07-23 10:23:22,025 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 62628167-d351-4715-8833-28d3a64d59cd
2021-07-23 10:23:22,037 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review f8e8f18f-02a5-4660-ad4b-18e4c0c1c14e
2021-07-23 10:23:22,053 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 10d4505c-87e2-48d2-8fef-07756c811ef8
2021-07-23 10:23:22,067 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 81bf7928-0ff2-4725-b14f-9c31613f3a71
2021-07-23 10:23:22,080 INFO [io.st.op.co.ConversionResource] (executor-thread-9) Converting conversion review 71b4aa56-4673-4eca-ba15-348a7d3218ed
2021-07-23 10:23:22,308 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Cluster dbops-security-upgrade-60fa88bc/dbops-security-upgrade-3 it's not up to date. Reconciling
2021-07-23 10:23:22,308 INFO [io.st.reconciliator] (OkHttp https://10.96.0.1/...) Patching resource dbops-security-upgrade-3-patroni of kind: ServiceAccount
2021-07-23 10:23:23,288 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review d3ab877d-8810-4f98-8d2e-0b09bd2b6ad3 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:23:23,291 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 26fa26cc-77b2-47b8-b668-8d33793ba6a9 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:23:33,325 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review e5c0f49f-a680-4f8b-baa3-8167ffa03bc8 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:23:33,328 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 0a9cf8f8-b987-4253-a4df-7f52d8cd461e of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:23:43,667 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review a58a13a1-5967-4c15-9d09-39598096327a of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:23:43,669 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 1b8d05ab-377f-4786-beaa-062fa2a922a2 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:23:53,432 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review e134c950-6d8a-4575-8e65-0f34b8e2fc60 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:23:53,434 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 02dfc67e-f75e-4439-b4c3-879ef93e9ec7 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:24:04,110 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review 2ae621e4-2f0d-4f2a-8530-eff2f3b679ef of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:24:04,114 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 5bd30ad0-5750-4660-82bf-bbfc56474fcd of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:24:14,308 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review 53e1773f-0181-4239-a5dc-88fb7adb5fc5 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:24:14,339 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-10) Validating admission review 9c9a8481-be45-4564-9135-19f67be2120d of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:24:23,492 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-10) Mutating admission review 82ad6882-79da-449f-a5dc-fc01af36c5bb of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:24:23,504 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-10) Validating admission review 11d744ed-1bfb-4e6f-9652-ce3f2be29b0e of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:24:33,623 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-10) Mutating admission review 69a4eca2-9541-4350-af44-cbbe12f75969 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:24:33,627 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-10) Validating admission review 5f1d1fdb-2f60-47ce-92fc-b67f78891f48 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:12,332 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review 26ddaac3-7429-48b0-9470-424f22024e51 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:12,343 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review a67e188a-03e5-45ac-8eef-752ed821d7ae of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:12,345 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review b7e5243d-0973-454e-9be6-a5450b0f60df of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:13,644 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review a6987c83-0743-4098-b539-c0ac0aec2a2f of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:13,648 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review bfb1a7e7-af52-40a0-8c96-5725057b5519 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:23,326 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review 6f326068-c580-417b-a252-f2906c9962d8 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:23,335 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review dce8e99e-86e0-442f-9ce9-572b433141d5 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:33,440 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review 727ec3c6-d058-48fe-931b-97d12bbc8c3e of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:33,444 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review c8f060dc-7e6d-4681-a00b-68ce2b0ff2f1 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:43,928 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review e06f4a81-e070-4811-90df-e6e25b99592b of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:43,936 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 09056764-a688-47eb-8b39-d4f5351bb4d5 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:53,363 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review 66110def-73dd-4549-98a5-ef63177f05ff of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:25:53,367 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 01988fb9-c766-4ecf-a702-794e26de6ea2 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:26:03,440 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review 32078115-1b7f-4796-aac9-0a23b340b989 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:26:03,444 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 0d9e0ccb-4486-46cd-b572-4aedb75225e6 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:26:13,432 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review 7c472666-fc5b-43a6-b162-b04952e1bc7e of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:26:13,435 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review fbcb70a9-7721-4b0a-896b-695fabe926ff of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:26:23,344 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review e8c13849-7837-40e6-a153-1150814cf112 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:26:23,347 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 9bedb746-c439-407b-ae8c-064052914369 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:26:33,583 INFO [io.st.op.ad.mu.MutationResource] (executor-thread-9) Mutating admission review 52f9ac12-a052-49f5-bbbd-beaed5490e04 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})
2021-07-23 10:26:33,586 INFO [io.st.op.ad.va.ValidationResource] (executor-thread-9) Validating admission review 4ba69d0f-250c-42cf-8c30-c47c47d07f22 of kind GroupVersionKind(group=stackgres.io, kind=SGDistributedLogs, version=v1, additionalProperties={})