keycloak unavailable due to low-disk space on cnpg-keycloak PVC's

Problem

SSO login with keycloak is not working, Keycloak dashboard is unavailable ("503 Service Temporarily Unavailable").

cnpg-keycloak and keycloak pods are in CrashLoopBackOff:

ubuntu@bootstrap25-vm:~/my-deployment/sylva-core$ k get pods -n keycloak
NAME                                       READY   STATUS             RESTARTS         AGE
cnpg-keycloak-2                            0/1     CrashLoopBackOff   17 (44s ago)     63m
keycloak-0                                 0/1     CrashLoopBackOff   16 (2m55s ago)   29d
keycloak-1                                 0/1     CrashLoopBackOff   16 (3m16s ago)   29d
keycloak-2                                 0/1     CrashLoopBackOff   16 (4m5s ago)    66m
keycloak-add-client-scope-w8wj5            0/1     Completed          0                11d
keycloak-add-realm-role-lp4j9              0/1     Completed          0                11d
keycloak-operator-5564c9cdd7-tlwzg         1/1     Running            3 (65m ago)      4d
keycloak-realm-operator-74b574f8b4-7xxlg   1/1     Running            2 (65m ago)      29d
postgres-primary-0                         1/1     Running            1 (65m ago)      29d
postgres-read-0                            1/1     Running            1 (65m ago)      29d
postgres-read-1                            1/1     Running            1 (69m ago)      29d
postgres-read-2                            1/1     Running            0                61m
sylva-dz7r5                                0/1     Completed          0                29d

keycloak pods are failing because they cannot connect to the cnpg-keycloak-rw service. Example in kycloak-0 logs:

2025-04-29 14:30:21,029 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': Connection to cnpg-keycloak-rw:5432 refused. Check that the hostname and port are correct and that the post master is accepting TCP/IP connections.

cnpg-keycloak-2 is not starting because there is not enough disk space on the postgresql PVC (pgdata volume):

ubuntu@bootstrap25-vm:~/my-deployment/sylva-core$ k logs cnpg-keycloak-2 -n keycloak
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
{"level":"info","ts":"2025-04-29T14:32:32.017212644Z","msg":"Starting CloudNativePG Instance Manager","logger":"instance-manager","logging_pod":"cnpg-keycloak-2","version":"1.24.1","build":{"Version":"1.24.1","Commit":"3f96930d","Date":"2024-10-16"}}
...
{"level":"info","ts":"2025-04-29T14:32:32.259454023Z","msg":"Checking for free disk space for WALs before starting PostgreSQL","logger":"instance-manager","logging_pod":"cnpg-keycloak-2"}
{"level":"info","ts":"2025-04-29T14:32:32.30827256Z","msg":"Detected low-disk space condition, avoid starting the instance","logger":"instance-manager","logging_pod":"cnpg-keycloak-2"}

cc @marc.bailly1 @satyawanj @matrohon

Assignee Loading
Time tracking Loading