db1.cluster was shut down unexpectedly
Today around 17:38 UTC the primary database was shut down and restarted. This caused a brief moment of 500 errors. Logs from around the time:
2017-02-14_17:37:57.03686 db1 postgresql: 2017-02-14 17:37:57 GMT [9324]: [38-1] DETAIL: parameters: $1 = 'f', $2 = 'f'
2017-02-14_17:37:57.33979 db1 postgresql: received TERM from runit, sending INT instead to force quit connections
2017-02-14_17:37:57.34224 db1 postgresql: 2017-02-14 17:37:57 GMT [63340]: [2-1] LOG: received fast shutdown request
2017-02-14_17:37:57.34237 db1 postgresql: 2017-02-14 17:37:57 GMT [63340]: [3-1] LOG: aborting any active transactions
2017-02-14_17:37:57.34274 db1 postgresql: 2017-02-14 17:37:57 GMT [14514]: [1-1] FATAL: terminating connection due to administrator command
2017-02-14_17:37:57.34459 db1 postgresql: 2017-02-14 17:37:57 GMT [14513]: [1-1] FATAL: terminating connection due to administrator command
Followed by:
2017-02-14_17:38:12.57976 db1 postgresql: 2017-02-14 17:38:12 GMT [34552]: [1-1] FATAL: the database system is shutting down
2017-02-14_17:38:12.58083 db1 postgresql: 2017-02-14 17:38:12 GMT [34554]: [1-1] FATAL: the database system is shutting down
2017-02-14_17:38:12.58090 db1 postgresql: 2017-02-14 17:38:12 GMT [34553]: [1-1] FATAL: the database system is shutting down
2017-02-14_17:38:12.58127 db1 postgresql: 2017-02-14 17:38:12 GMT [34556]: [1-1] FATAL: the database system is shutting down
2017-02-14_17:38:12.58143 db1 postgresql: 2017-02-14 17:38:12 GMT [34557]: [1-1] FATAL: the database system is shutting down
2017-02-14_17:38:12.58218 db1 postgresql: 2017-02-14 17:38:12 GMT [34558]: [1-1] FATAL: the database system is shutting down
2017-02-14_17:38:12.58238 db1 postgresql: 2017-02-14 17:38:12 GMT [34560]: [1-1] FATAL: the database system is shutting down
2017-02-14_17:38:12.58260 db1 postgresql: 2017-02-14 17:38:12 GMT [34559]: [1-1] FATAL: the database system is shutting down
This line stands out:
2017-02-14_17:37:57.33979 db1 postgresql: received TERM from runit, sending INT instead to force quit connections
This suggests either somebody manually restarted PostgreSQL (unlikely), or e.g. chef (or some other process) did.