Set PostgreSQL's max_connections to a sane value
max_connections is set to 2000, but this used to be 8000. During yesterday's replication problems we at some point restarted the master after increasing
max_wal_senders from 3 to 16. This resulted in the PostgreSQL master complaining about there being too many semaphores, caused by a high
max_connections value. The only way to get things working again was to drop this value to 2000.
I suspect that some other kernel parameter changed, causing this to fail. Either way, 8000 connections is way too much. Per Grafana we peak at 800 active connections, so even 2000 should be more than enough (if not too many).
We should find historical data to see what our peaks are (besides the 800 I found in the last week), and figure out what a sane default is. We also need to find out why this was working for almost a year, only to fail now.