Too many unicorn workers on dev, leading to PostgreSQL downtime
root@dev:/# ps -ef | grep unicorn | wc -l
67
root@dev:/var/log/gitlab/postgresql# tail -f current
2017-04-14_00:37:42.82726 LOG: database system is shut down
2017-04-14_00:37:43.86440 FATAL: could not map anonymous shared memory: Cannot allocate memory
2017-04-14_00:37:43.86442 HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 7571652608 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2017-04-14_00:37:43.86449 LOG: database system is shut down
2017-04-14_00:37:44.90002 FATAL: could not map anonymous shared memory: Cannot allocate memory
2017-04-14_00:37:44.90003 HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 7571652608 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2017-04-14_00:37:44.90004 LOG: database system is shut down
2017-04-14_00:37:45.93546 FATAL: could not map anonymous shared memory: Cannot allocate memory
2017-04-14_00:37:45.93546 HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 7571652608 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2017-04-14_00:37:45.93549 LOG: database system is shut down
2017-04-14_00:37:46.97468 FATAL: could not map anonymous shared memory: Cannot allocate memory
2017-04-14_00:37:46.97470 HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 7571652608 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2017-04-14_00:37:46.97472 LOG: database system is shut down
2017-04-14_00:37:48.00987 FATAL: could not map anonymous shared memory: Cannot allocate memory
2017-04-14_00:37:48.00993 HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory, swap space, or huge pages. To reduce the request size (currently 7571652608 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2017-04-14_00:37:48.00996 LOG: database system is shut down
Temporary workaround:
gitlab-ctl stop unicorn
ps aux | grep -ie unicorn | awk '{print $2}' | xargs kill -9
gitlab-ctl start unicorn