issues around Keycloak Postgres DB dedicated WAL partition
Context: we merged !4520 (merged) a few weeks ago and backported it to Sylva 1.3 (!4538 (merged)) and 1.4 (!4974 (merged)).
After a discussion today with @feleouet, we realize that this change raises a few questions:
- first, it did not resolve the "getting out of space" issues (see #2648, #2744 (closed))
- in fact, it seems that the amount of CI failures due to this problem has rather increased
- one argument in favor of merging !4520 (merged) was a simplification of ops during an event where the WAL eats the disk
-
Suppose if we had dedicated WAL disk apart from the pg_data disk and all the WAL disks are filled with WAL files. Then, by only deleting the WAL pvcs and the pods will bring the cluster up and running.
- this felt like a valid argument, however after some testing today, @feleouet observed that, in fact, deleting the WAL PVC does not help much, because the corresponding postgres instance can't be revived without also deleting its non-WAL PVC
-
- the size of 2GiB is possibly too small to hold the WAL in some situations; this seems plausibly explain by the fact that
max_wal_sizedefaults to 2GiB (not 1GiB as initially believed) and that it's documented that data on disk can sometimes exceedsmax_wal_size
The above leads us to conclude that moving the WAL to a dedicated partition is possibly not a good idea.
It is important to have in mind that it will be significant work to transition deployments done with "WAL on a dedicated partition" to "WAL on same partition" (once set, the corresponding field is immutable in CNPG CRD).
With all of the above in mind, we feel that we should revert !4520 (merged) and its backports.
We'll still have the possibility of introducing it later, if more discussion and a better understanding of #2648 lead us to conclude that it really is the right thing to do.