Check for evidence of oomkiller activity on the system
If oomkiller comes into play on a customer's system, it can cause real damage if the wrong processes are killed. This should do a spot check (in /var/log/messages, /var/log/syslog and dmesg for evidence of oomkiller activity.
OOMkiller is a bit frustrating because the message content may be logged by dmesg or in /var/log/messages or I swear I've also seen it in /var/log/syslog, but no matter where it shows up it seems that there's no standard content that you will find in every case. I've tried searching across old support tickets, and found examples from dmesg and from /var/log/messages as a starting point. We may find that this check needs tweaking as we find more examples in the future.
I was able to trigger an oom-killer event on a test system in less than a minute with tail /dev/zero.
dmesg examples
[Wed Sep 3 14:02:41 2025] Out of memory: Killed process 2626304 (ruby) total-vm:3934332kB, anon-rss:1921108kB, file-rss:0kB, shmem-rss:4300kB, UID:152993154 pgtables:6956kB oom_score_adj:0
[Wed Sep 3 14:09:31 2025] puma srv tp 003 invoked oom-killer: gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
--
[Wed Sep 3 14:09:34 2025] Out of memory: Killed process 2642564 (puma srv tp 002) total-vm:4158904kB, anon-rss:1913792kB, file-rss:0kB, shmem-rss:8384kB, UID:152993154 pgtables:7084kB oom_score_adj:0
[Wed Sep 3 14:15:51 2025] gitlab-workhors invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
--
[Wed Sep 3 14:15:53 2025] Out of memory: Killed process 3146093 (puma srv tp 001) total-vm:3986336kB, anon-rss:1870512kB, file-rss:0kB, shmem-rss:11200kB, UID:152993154 pgtables:6896kB oom_score_adj:0
[Wed Sep 3 15:05:10 2025] gitlab-workhors invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0[172105.518484] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-71.scope,task=tail,pid=160436,uid=0
[172105.518512] Out of memory: Killed process 160436 (tail) total-vm:5237468kB, anon-rss:5230260kB, file-rss:1460kB, shmem-rss:0kB, UID:0 pgtables:10288kB oom_score_adj:0[Tue Sep 5 21:10:43 2023] Out of memory: Kill process 20763 (ruby) score 341 or sacrifice child
[Tue Sep 5 21:10:43 2023] Killed process 20763 (ruby) total-vm:9564764kB, anon-rss:5426616kB, file-rss:0kB, shmem-rss:0kB
[Tue Sep 5 21:10:43 2023] oom_reaper: reaped process 20763 (ruby), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB/var/log/messages examples
2025-09-03T14:02:48.440808+01:00 xpff7640033rl12 kernel: main-worker invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
2025-09-03T14:09:40.815655+01:00 xpff7640033rl12 kernel: puma srv tp 003 invoked oom-killer: gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
2025-09-03T14:15:57.922570+01:00 xpff7640033rl12 kernel: gitlab-workhors invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
2025-09-03T15:05:20.437550+01:00 xpff7640033rl12 kernel: gitlab-workhors invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
2025-09-03T15:05:47.904422+01:00 xpff7640033rl12 kernel: nginx invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), order=0, oom_score_adj=0
/var/log/syslog examples
Nov 12 19:34:38 interview-instance kernel: [172105.518484] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-71.scope,task=tail,pid=160436,uid=0
Nov 12 19:34:38 interview-instance kernel: [172105.518512] Out of memory: Killed process 160436 (tail) total-vm:5237468kB, anon-rss:5230260kB, file-rss:1460kB, shmem-rss:0kB, UID:0 pgtables:10288kB oom_score_adj:0