Sidekiq ES OOM killer terminating some jobs
We have some errors showing up in Sentry:
OOM killer is having some fun on the sidekiq-elasticsearch nodes:
vlopez@sidekiq-elasticsearch-03:~$ dmesg -T | grep "Killed process"
[Wed Aug 9 13:39:05 2017] Killed process 53627 (gitlab-elastics) total-vm:1652776kB, anon-rss:1510508kB, file-rss:0kB
[Wed Aug 9 13:40:24 2017] Killed process 20281 (gitlab-elastics) total-vm:2167344kB, anon-rss:1987088kB, file-rss:0kB
[Wed Aug 9 13:40:25 2017] Killed process 20283 (gitlab-elastics) total-vm:2167344kB, anon-rss:1987552kB, file-rss:0kB
[Wed Aug 9 13:51:04 2017] Killed process 39556 (gitlab-elastics) total-vm:1806304kB, anon-rss:1676820kB, file-rss:0kB
[Wed Aug 9 13:53:02 2017] Killed process 45231 (gitlab-elastics) total-vm:1621516kB, anon-rss:1490632kB, file-rss:0kB
[Wed Aug 9 14:00:22 2017] Killed process 62479 (gitlab-elastics) total-vm:1795616kB, anon-rss:1663924kB, file-rss:0kB
[Wed Aug 9 14:05:46 2017] Killed process 11175 (gitlab-elastics) total-vm:2216772kB, anon-rss:1792356kB, file-rss:0kB
[Wed Aug 9 14:05:46 2017] Killed process 11177 (gitlab-elastics) total-vm:2216772kB, anon-rss:1792892kB, file-rss:0kB
[Wed Aug 9 15:05:56 2017] Killed process 32672 (gitlab-elastics) total-vm:2268832kB, anon-rss:2120112kB, file-rss:304kB
[Wed Aug 9 15:05:57 2017] Killed process 32674 (gitlab-elastics) total-vm:2268832kB, anon-rss:2120588kB, file-rss:816kB
[Wed Aug 9 15:05:57 2017] Killed process 32675 (gitlab-elastics) total-vm:2268832kB, anon-rss:2120596kB, file-rss:1104kB
[Wed Aug 9 15:05:57 2017] Killed process 32676 (gitlab-elastics) total-vm:2268832kB, anon-rss:2120704kB, file-rss:1300kB
[Wed Aug 9 15:35:46 2017] Killed process 5785 (gitlab-elastics) total-vm:2056364kB, anon-rss:1901796kB, file-rss:0kB
[Wed Aug 9 15:35:46 2017] Killed process 5787 (gitlab-elastics) total-vm:2056364kB, anon-rss:1902756kB, file-rss:944kB
[Wed Aug 9 17:27:04 2017] Killed process 23665 (gitlab-elastics) total-vm:1926388kB, anon-rss:1815276kB, file-rss:0kB
[Wed Aug 9 17:51:21 2017] Killed process 928 (gitlab-elastics) total-vm:1483196kB, anon-rss:1371104kB, file-rss:744kB
[Wed Aug 9 22:00:23 2017] Killed process 60188 (gitlab-elastics) total-vm:2150708kB, anon-rss:1877116kB, file-rss:0kB
[Wed Aug 9 22:05:53 2017] Killed process 60518 (gitlab-elastics) total-vm:2331752kB, anon-rss:2187460kB, file-rss:0kB
[Wed Aug 9 22:10:29 2017] Killed process 4555 (gitlab-elastics) total-vm:1969476kB, anon-rss:1837264kB, file-rss:0kB
[Thu Aug 10 01:39:19 2017] Killed process 19492 (gitlab-elastics) total-vm:2340532kB, anon-rss:2201044kB, file-rss:0kB
[Thu Aug 10 01:53:44 2017] Killed process 26113 (gitlab-elastics) total-vm:2419824kB, anon-rss:2281924kB, file-rss:0kB
[Thu Aug 10 01:58:55 2017] Killed process 39938 (gitlab-elastics) total-vm:2307052kB, anon-rss:2148324kB, file-rss:0kB
[Thu Aug 10 01:58:55 2017] Killed process 39940 (gitlab-elastics) total-vm:2307052kB, anon-rss:2149020kB, file-rss:0kB
[Thu Aug 10 02:16:47 2017] Killed process 3404 (gitlab-elastics) total-vm:3291704kB, anon-rss:3164724kB, file-rss:700kB
[Thu Aug 10 03:10:42 2017] Killed process 39279 (gitlab-elastics) total-vm:2557144kB, anon-rss:1997776kB, file-rss:0kB
[Thu Aug 10 03:54:07 2017] Killed process 38794 (gitlab-elastics) total-vm:2522552kB, anon-rss:2377452kB, file-rss:0kB
[Thu Aug 10 05:49:49 2017] Killed process 7460 (gitlab-elastics) total-vm:1910992kB, anon-rss:1798776kB, file-rss:0kB
[Thu Aug 10 06:01:19 2017] Killed process 20031 (gitlab-elastics) total-vm:2201620kB, anon-rss:2079644kB, file-rss:0kB
[Thu Aug 10 06:08:44 2017] Killed process 26325 (gitlab-elastics) total-vm:2323036kB, anon-rss:2205560kB, file-rss:0kB
[Thu Aug 10 06:28:46 2017] Killed process 53030 (gitlab-elastics) total-vm:2250480kB, anon-rss:2089316kB, file-rss:0kB
[Thu Aug 10 06:28:47 2017] Killed process 53034 (gitlab-elastics) total-vm:2250480kB, anon-rss:2090088kB, file-rss:0kB
[Thu Aug 10 07:08:58 2017] Killed process 62350 (gitlab-elastics) total-vm:2004024kB, anon-rss:1889452kB, file-rss:0kB
[Thu Aug 10 07:18:48 2017] Killed process 10391 (gitlab-elastics) total-vm:1734916kB, anon-rss:1619564kB, file-rss:0kB
[Thu Aug 10 07:18:48 2017] Killed process 10393 (gitlab-elastics) total-vm:1734916kB, anon-rss:1619928kB, file-rss:0kB
[Thu Aug 10 09:30:29 2017] Killed process 1794 (gitlab-elastics) total-vm:2558352kB, anon-rss:2419896kB, file-rss:0kB
[Thu Aug 10 10:18:40 2017] Killed process 40346 (gitlab-elastics) total-vm:2485308kB, anon-rss:1855592kB, file-rss:0kB
[Thu Aug 10 10:39:11 2017] Killed process 43346 (gitlab-elastics) total-vm:2380384kB, anon-rss:2240080kB, file-rss:0kB
Current config for the nodes is:
"queue_groups": [
"elastic_indexer",
"elastic_commit_indexer",
"elastic_commit_indexer",
"elastic_commit_indexer"
]