keycloak unavailable due to low-disk space on cnpg-keycloak PVC's
Problem
SSO login with keycloak is not working, Keycloak dashboard is unavailable ("503 Service Temporarily Unavailable").
cnpg-keycloak and keycloak pods are in CrashLoopBackOff:
ubuntu@bootstrap25-vm:~/my-deployment/sylva-core$ k get pods -n keycloak
NAME READY STATUS RESTARTS AGE
cnpg-keycloak-2 0/1 CrashLoopBackOff 17 (44s ago) 63m
keycloak-0 0/1 CrashLoopBackOff 16 (2m55s ago) 29d
keycloak-1 0/1 CrashLoopBackOff 16 (3m16s ago) 29d
keycloak-2 0/1 CrashLoopBackOff 16 (4m5s ago) 66m
keycloak-add-client-scope-w8wj5 0/1 Completed 0 11d
keycloak-add-realm-role-lp4j9 0/1 Completed 0 11d
keycloak-operator-5564c9cdd7-tlwzg 1/1 Running 3 (65m ago) 4d
keycloak-realm-operator-74b574f8b4-7xxlg 1/1 Running 2 (65m ago) 29d
postgres-primary-0 1/1 Running 1 (65m ago) 29d
postgres-read-0 1/1 Running 1 (65m ago) 29d
postgres-read-1 1/1 Running 1 (69m ago) 29d
postgres-read-2 1/1 Running 0 61m
sylva-dz7r5 0/1 Completed 0 29d
keycloak pods are failing because they cannot connect to the cnpg-keycloak-rw service. Example in kycloak-0 logs:
2025-04-29 14:30:21,029 WARN [io.agroal.pool] (agroal-11) Datasource '<default>': Connection to cnpg-keycloak-rw:5432 refused. Check that the hostname and port are correct and that the post master is accepting TCP/IP connections.
cnpg-keycloak-2 is not starting because there is not enough disk space on the postgresql PVC (pgdata volume):
ubuntu@bootstrap25-vm:~/my-deployment/sylva-core$ k logs cnpg-keycloak-2 -n keycloak
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
{"level":"info","ts":"2025-04-29T14:32:32.017212644Z","msg":"Starting CloudNativePG Instance Manager","logger":"instance-manager","logging_pod":"cnpg-keycloak-2","version":"1.24.1","build":{"Version":"1.24.1","Commit":"3f96930d","Date":"2024-10-16"}}
...
{"level":"info","ts":"2025-04-29T14:32:32.259454023Z","msg":"Checking for free disk space for WALs before starting PostgreSQL","logger":"instance-manager","logging_pod":"cnpg-keycloak-2"}
{"level":"info","ts":"2025-04-29T14:32:32.30827256Z","msg":"Detected low-disk space condition, avoid starting the instance","logger":"instance-manager","logging_pod":"cnpg-keycloak-2"}