Consider offloading WAL archiving outside of PostgreSQL master node.
According to https://gitlab.com/gitlab-com/infrastructure/issues/4650, WAL-E is driving a significant amount of I/O on the PostgreSQL master node:
-
gpg
(driven by WAL-e) is eating 66% of all the write I/O! - 2/3s of write I/O are driven by WAL-e, encrypting the files. This could be, potentially, offloaded to an external server. WAL-e is also generating a lot of read I/O, required to read the files that need to be encrypted. In total, WAL-e accounts for 29% of the total read + write I/O.
The proposal under this issue is to study how to architect a solution were WAL archiving could be performed to an external location, and encryption and sending to S3 could be done from an external server, to save very significant I/O on the master node.