Cluster reconciliation cycle failed reconciling java.lang.NullPointerException
Checking the operator logs an error appears continuously:
2021-03-08 09:15:09,100 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-1 of type Pod
2021-03-08 09:15:09,100 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-2 of type Pod
2021-03-08 09:15:09,102 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory of type StatefulSet
2021-03-08 09:15:09,102 DEBUG [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Updating resource sg-prod.mminventory-backup of type CronJob to meet Cluster requirements
2021-03-08 09:15:09,103 ERROR [io.st.op.re.AbstractReconciliationCycle] (Cluster-ReconciliationCycle) 208896| Cluster reconciliation cycle failed reconciling sg-prod.mminventory: java.lang.NullPointerException
2021-03-08 09:15:09,173 TRACE [io.st.op.re.AbstractReconciliationCycle] (Cluster-ReconciliationCycle) 208896| Cluster reconciliation cycle ended successfully
2021-03-08 09:15:18,384 TRACE [io.st.op.re.AbstractReconciliationCycle] (DistributeLogs-ReconciliationCycle) 206419| DistributeLogs reconciliation cycle starting
2021-03-08 09:15:18,384 TRACE [io.st.op.re.AbstractReconciliationCycle] (DistributeLogs-ReconciliationCycle) 206419| DistributeLogs reconciliation cycle getting existing distributelogs list
2021-03-08 09:15:18,388 TRACE [io.st.op.re.AbstractReconciliationCycle] (Cluster-ReconciliationCycle) 208897| Cluster reconciliation cycle starting
2021-03-08 09:15:18,388 TRACE [io.st.op.re.AbstractReconciliationCycle] (Cluster-ReconciliationCycle) 208897| Cluster reconciliation cycle getting existing cluster list
2021-03-08 09:15:18,524 TRACE [io.st.op.re.AbstractReconciliationCycle] (DistributeLogs-ReconciliationCycle) 206419| DistributeLogs reconciliation cycle working on monitoring.stackgres-distributedlogs
2021-03-08 09:15:18,751 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-templates of type ConfigMap
2021-03-08 09:15:18,751 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs of type ConfigMap
2021-03-08 09:15:18,751 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-00000-distributed-logs-template-templ of type ConfigMap
2021-03-08 09:15:18,751 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs of type Secret
2021-03-08 09:15:18,752 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-patroni of type ServiceAccount
2021-03-08 09:15:18,752 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-patroni of type Role
2021-03-08 09:15:18,752 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-patroni of type RoleBinding
2021-03-08 09:15:18,752 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-config of type Service
2021-03-08 09:15:18,752 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs of type Service
2021-03-08 09:15:18,752 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-primary of type Service
2021-03-08 09:15:18,753 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-replicas of type Service
2021-03-08 09:15:18,753 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-config of type Endpoints
2021-03-08 09:15:18,753 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-fluentd of type ConfigMap
2021-03-08 09:15:18,753 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-fluentd of type Service
2021-03-08 09:15:18,754 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-0 of type Pod
2021-03-08 09:15:18,754 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-0 of type Pod
2021-03-08 09:15:18,755 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs of type StatefulSet
2021-03-08 09:15:18,755 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-backup of type ConfigMap
2021-03-08 09:15:18,755 TRACE [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Found resource monitoring.stackgres-distributedlogs-backup of type Secret
2021-03-08 09:15:18,759 DEBUG [io.st.op.re.AbstractReconciliator] (DistributeLogs-ReconciliationCycle) Centralized Logging synced: 'monitoring.stackgres-distributedlogs'
2021-03-08 09:15:18,759 TRACE [io.st.op.re.AbstractReconciliationCycle] (DistributeLogs-ReconciliationCycle) 206419| DistributeLogs reconciliation cycle ended successfully
2021-03-08 09:15:18,770 TRACE [io.st.op.re.AbstractReconciliationCycle] (Cluster-ReconciliationCycle) Prometheus auto bind enabled, looking for prometheus installations
2021-03-08 09:15:18,877 TRACE [io.st.op.re.AbstractReconciliationCycle] (Cluster-ReconciliationCycle) 208897| Cluster reconciliation cycle working on sg-prod.mminventory
2021-03-08 09:15:19,007 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-templates of type ConfigMap
2021-03-08 09:15:19,007 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory of type ConfigMap
2021-03-08 09:15:19,007 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-internal-00000-prometheus-postgres-exporter-init-po of type ConfigMap
2021-03-08 09:15:19,007 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-00001-create-mm-database of type ConfigMap
2021-03-08 09:15:19,007 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory of type Secret
2021-03-08 09:15:19,007 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-patroni of type ServiceAccount
2021-03-08 09:15:19,008 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-patroni of type Role
2021-03-08 09:15:19,008 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-patroni of type RoleBinding
2021-03-08 09:15:19,008 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-config of type Service
2021-03-08 09:15:19,008 DEBUG [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Updating resource sg-prod.mminventory of type Service to meet Cluster requirements
2021-03-08 09:15:19,059 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-primary of type Service
2021-03-08 09:15:19,059 DEBUG [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Updating resource sg-prod.mminventory-replicas of type Service to meet Cluster requirements
2021-03-08 09:15:19,111 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-config of type Endpoints
2021-03-08 09:15:19,111 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-prometheus-postgres-exporter-config of type ConfigMap
2021-03-08 09:15:19,111 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-prometheus-postgres-exporter of type Service
2021-03-08 09:15:19,111 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource monitoring.sg-prod-mminventory-stackgres-postgres-exporter of type ServiceMonitor
2021-03-08 09:15:19,111 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-envoy-config of type ConfigMap
2021-03-08 09:15:19,111 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-prometheus-envoy of type Service
2021-03-08 09:15:19,111 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource monitoring.sg-prod-mminventory-stackgres-envoy of type ServiceMonitor
2021-03-08 09:15:19,112 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-fluent-bit of type ConfigMap
2021-03-08 09:15:19,112 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-connection-pooling-config of type ConfigMap
2021-03-08 09:15:19,112 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-0 of type Pod
2021-03-08 09:15:19,112 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-1 of type Pod
2021-03-08 09:15:19,112 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory-2 of type Pod
2021-03-08 09:15:19,112 TRACE [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Found resource sg-prod.mminventory of type StatefulSet
2021-03-08 09:15:19,113 DEBUG [io.st.op.re.AbstractReconciliator] (Cluster-ReconciliationCycle) Updating resource sg-prod.mminventory-backup of type CronJob to meet Cluster requirements
2021-03-08 09:15:19,113 ERROR [io.st.op.re.AbstractReconciliationCycle] (Cluster-ReconciliationCycle) 208897| Cluster reconciliation cycle failed reconciling sg-prod.mminventory: java.lang.NullPointerException
2021-03-08 09:15:19,189 TRACE [io.st.op.re.AbstractReconciliationCycle] (Cluster-ReconciliationCycle) 208897| Cluster reconciliation cycle ended successfully
Probably something related to backups
Jobs:
$ kubectl get jobs -n sg-prod
NAME COMPLETIONS DURATION AGE
mminventory-backup-1612837800 0/1 27d 27d
mminventory-backup-1613874600 1/1 3h15m 15d
mminventory-backup-1614220200 1/1 10h 11d
mminventory-backup-1614565800 1/1 8h 7d6h
A weird thing is the first one on the list DURATION==AGE
Pods:
$ kubectl get pods -n sg-prod
NAME READY STATUS RESTARTS AGE
mminventory-0 6/6 Running 0 24d
mminventory-1 6/6 Running 0 2d18h
mminventory-2 6/6 Running 0 2d18h
mminventory-backup-1613874600-z6pkx 0/1 Completed 0 15d
mminventory-backup-1614220200-bgd6w 0/1 Completed 0 11d
mminventory-backup-1614565800-hld2g 0/1 Completed 0 7d6h
backups:
$ kubectl get sgbackups -n sg-prod
NAME AGE
mminventory-2021-02-17-02-30-13 19d
mminventory-2021-02-21-02-30-12 15d
mminventory-2021-02-25-02-30-19 11d
mminventory-2021-03-01-02-30-15 7d6h
The uncompleted job describe:
$ kubectl describe job -n sg-prod mminventory-backup-1612837800
Name: mminventory-backup-1612837800
Namespace: sg-prod
Selector: controller-uid=9de9da1c-4bb7-4bad-a230-1031f5936ad0
Labels: app=StackGresCluster
cluster-name=mminventory
cluster-uid=263319a1-6eea-424b-9b6f-554f7d127e06
scheduled-backup=true
Annotations: <none>
Controlled By: CronJob/mminventory-backup
Parallelism: 1
Completions: 1
Start Time: Tue, 09 Feb 2021 03:30:04 +0100
Pods Statuses: 0 Running / 0 Succeeded / 1 Failed
Pod Template:
Labels: app=StackGresCluster
cluster-name=mminventory
cluster-uid=263319a1-6eea-424b-9b6f-554f7d127e06
controller-uid=9de9da1c-4bb7-4bad-a230-1031f5936ad0
job-name=mminventory-backup-1612837800
scheduled-backup=true
Service Account: mminventory-patroni
Containers:
create-backup:
Image: bitnami/kubectl:1.18.3
Port: <none>
Host Port: <none>
Command:
/bin/bash
-ex
/usr/local/bin/create-backup.sh
Environment:
ETC_PASSWD_PATH: /etc/passwd
ETC_GROUP_PATH: /etc/group
ETC_SHADOW_PATH: /etc/shadow
ETC_GSHADOW_PATH: /etc/gshadow
LOCAL_BIN_PATH: /usr/local/bin
LOCAL_BIN_SHELL_UTILS_PATH: /usr/local/bin/shell-utils
LOCAL_BIN_SETUP_DATA_PATHS_SH_PATH: /usr/local/bin/setup-data-paths.sh
LOCAL_BIN_SETUP_ARBITRARY_USER_SH_PATH: /usr/local/bin/setup-arbitrary-user.sh
LOCAL_BIN_SETUP_SCRIPTS_SH_PATH: /usr/local/bin/setup-scripts.sh
LOCAL_BIN_START_PATRONI_SH_PATH: /usr/local/bin/start-patroni.sh
LOCAL_BIN_START_PATRONI_WITH_RESTORE_SH_PATH: /usr/local/bin/start-patroni-with-restore.sh
LOCAL_BIN_POST_INIT_SH_PATH: /usr/local/bin/post-init.sh
LOCAL_BIN_EXEC_WITH_ENV_PATH: /usr/local/bin/exec-with-env
LOCAL_BIN_CREATE_BACKUP_SH_PATH: /usr/local/bin/create-backup.sh
PG_BASE_PATH: /var/lib/postgresql
PG_DATA_PATH: /var/lib/postgresql/data
PG_RUN_PATH: /var/run/postgresql
PG_LOG_PATH: /var/log/postgresql
BASE_ENV_PATH: /etc/env
SHARED_MEMORY_PATH: /dev/shm
BASE_SECRET_PATH: /etc/env/.secret
PATRONI_ENV_PATH: /etc/env/patroni
PATRONI_CONFIG_PATH: /etc/patroni
BACKUP_ENV_PATH: /etc/env/backup
BACKUP_SECRET_PATH: /etc/env/.secret/backup
RESTORE_ENV_PATH: /etc/env/restore
RESTORE_SECRET_PATH: /etc/env/.secret/restore
TEMPLATES_PATH: /templates
PATRONI_ENV: patroni
BACKUP_ENV: backup
RESTORE_ENV: restore
POSTGRES_ENTRY_PORT: 7432
POSTGRES_REPL_ENTRY_PORT: 7433
POSTGRES_POOL_PORT: 6432
POSTGRES_PORT: 5432
CLUSTER_NAMESPACE: sg-prod
CLUSTER_NAME: mminventory
CRONJOB_NAME: mminventory-backup
BACKUP_CONFIG_CRD_NAME: sgbackupconfigs.stackgres.io
BACKUP_CONFIG: backupconfig-prod
BACKUP_CRD_KIND: SGBackup
BACKUP_CRD_NAME: sgbackups.stackgres.io
BACKUP_CRD_APIVERSION: stackgres.io/v1beta1
BACKUP_PHASE_RUNNING: Running
BACKUP_PHASE_COMPLETED: Completed
BACKUP_PHASE_FAILED: Failed
PATRONI_ROLE_KEY: role
PATRONI_PRIMARY_ROLE: master
PATRONI_REPLICA_ROLE: replica
SCHEDULED_BACKUP_KEY: scheduled-backup
RIGHT_VALUE: true
PATRONI_CLUSTER_LABELS: app=StackGresCluster,cluster-uid=263319a1-6eea-424b-9b6f-554f7d127e06,cluster-name=mminventory,cluster=true
POD_NAME: (v1:metadata.name)
RETAIN: 4
WINDOW: 3600
Mounts:
/usr/local/bin/create-backup.sh from templates (ro,path="create-backup.sh")
/usr/local/bin/shell-utils from templates (ro,path="shell-utils")
Volumes:
templates:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mminventory-templates
Optional: false
Events: <n