Thanos: memory consumption issue leading to pod being OOM killed

On a capo RKE2 deployment with 3 controller nodes on the management cluster with logging, loki neuvector and gitea units enabled we have random failed of the control nodes

the failed node is in out of memory and OOM killer is called on thanos proces

2024-07-31T07:32:19.536261+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [143248.633938][ T73] oom_reaper: reaped process 12006 (thanos), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB 2024-07-31T07:32:19.632251+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [143248.729117][ T3221] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T07:32:21.680251+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [143250.779652][T20757] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T07:32:29.424271+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [143258.521533][T14498] EXT4-fs (vdj): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. 2024-07-31T07:32:29.424291+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [143258.522084][T14500] EXT4-fs (vdv): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. 2024-07-31T07:32:29.728253+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [143258.825557][ T5132] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T07:40:00.680250+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [143709.758899][ T6068] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T07:50:00.696294+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [144309.750291][ T7999] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T08:00:00.652256+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [144909.683298][T18149] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T08:10:00.656266+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [145509.664960][T28014] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T08:20:00.680252+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [146109.665828][T19039] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T08:30:00.636275+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [146709.594563][T10331] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T08:40:00.656250+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [147309.590540][T30779] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T08:50:00.688316+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [147909.599551][T29296] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:00:00.620257+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [148509.507769][T24443] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:10:00.693123+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [149109.552434][T27335] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:20:00.656252+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [149709.493331][T18885] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:30:00.688258+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [150309.503360][T26184] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:37:00.692252+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [150729.490834][T18529] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:40:00.640247+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [150909.429114][T18291] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:45:56.128267+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151264.902971][T20245] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:50:00.727637+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151509.488676][T21245] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:24.960386+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151653.719624][T21245] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:26.028264+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151654.787417][ T2125] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:27.176263+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151655.938106][T20245] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:28.383567+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151657.142271][T12941] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:29.465883+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151658.225126][T12228] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:30.860538+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151659.618609][T20245] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:31.864250+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151660.623182][T12941] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:33.184380+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151661.938341][T25414] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:34.227526+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151662.980392][T21245] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:35.283832+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151664.034587][T21245] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:36.360260+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151665.122309][T12985] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:37.500286+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151666.256616][T28648] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:38.564258+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151667.323248][ T7967] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:52:45.542512+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151674.298503][T22260] pgpool[22260]: segfault at 10 ip 00005645aaeb9084 sp 00007ffc584ca070 error 4 in pgpool[5645aae54000+160000] 2024-07-31T09:52:45.542552+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151674.306961][T22260] Code: 89 e5 53 48 83 ec 48 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 e8 86 12 01 00 89 c2 48 8b 45 b0 48 63 d2 48 8b 44 d0 08 <48> 8b 40 10 0f b6 80 8c 00 00 00 3c 45 74 0a b8 01 00 00 00 e9 15 2024-07-31T09:52:50.235475+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151678.991437][T22242] pgpool[22242]: segfault at 10 ip 00005645aaeb9084 sp 00007ffc584ca070 error 4 in pgpool[5645aae54000+160000] 2024-07-31T09:52:50.235506+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151678.999746][T22242] Code: 89 e5 53 48 83 ec 48 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 e8 86 12 01 00 89 c2 48 8b 45 b0 48 63 d2 48 8b 44 d0 08 <48> 8b 40 10 0f b6 80 8c 00 00 00 3c 45 74 0a b8 01 00 00 00 e9 15 2024-07-31T09:52:55.484282+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151684.237249][T28006] pgpool[28006]: segfault at 10 ip 00005645aaeb9084 sp 00007ffc584c9f60 error 4 in pgpool[5645aae54000+160000] 2024-07-31T09:52:55.484307+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151684.244479][T28006] Code: 89 e5 53 48 83 ec 48 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 e8 86 12 01 00 89 c2 48 8b 45 b0 48 63 d2 48 8b 44 d0 08 <48> 8b 40 10 0f b6 80 8c 00 00 00 3c 45 74 0a b8 01 00 00 00 e9 15 2024-07-31T09:53:00.316287+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151689.068346][T22240] pgpool[22240]: segfault at 10 ip 00005645aaeb9084 sp 00007ffc584ca070 error 4 in pgpool[5645aae54000+160000] 2024-07-31T09:53:00.316311+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151689.074903][T22240] Code: 89 e5 53 48 83 ec 48 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 e8 86 12 01 00 89 c2 48 8b 45 b0 48 63 d2 48 8b 44 d0 08 <48> 8b 40 10 0f b6 80 8c 00 00 00 3c 45 74 0a b8 01 00 00 00 e9 15 2024-07-31T09:53:05.222710+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151693.978885][T22247] pgpool[22247]: segfault at 10 ip 00005645aaeb9084 sp 00007ffc584ca070 error 4 in pgpool[5645aae54000+160000] 2024-07-31T09:53:05.222733+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151693.983916][T22247] Code: 89 e5 53 48 83 ec 48 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 e8 86 12 01 00 89 c2 48 8b 45 b0 48 63 d2 48 8b 44 d0 08 <48> 8b 40 10 0f b6 80 8c 00 00 00 3c 45 74 0a b8 01 00 00 00 e9 15 2024-07-31T09:53:05.241133+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151693.997676][T22252] pgpool[22252]: segfault at 10 ip 00005645aaeb9084 sp 00007ffc584ca070 error 4 in pgpool[5645aae54000+160000] 2024-07-31T09:53:05.241159+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [151694.004782][T22252] Code: 89 e5 53 48 83 ec 48 48 89 7d c8 48 89 75 c0 48 89 55 b8 48 89 4d b0 e8 86 12 01 00 89 c2 48 8b 45 b0 48 63 d2 48 8b 44 d0 08 <48> 8b 40 10 0f b6 80 8c 00 00 00 3c 45 74 0a b8 01 00 00 00 e9 15 2024-07-31T09:59:15.295957+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.034635][ T4150] pci 0000:00:12.0: [1af4:1001] type 00 class 0x010000 2024-07-31T09:59:15.297099+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.037328][ T4150] pci 0000:00:12.0: reg 0x10: [io 0x0000-0x003f] 2024-07-31T09:59:15.297103+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.039754][ T4150] pci 0000:00:12.0: reg 0x14: [mem 0x00000000-0x00000fff] 2024-07-31T09:59:15.297213+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.043662][ T4150] pci 0000:00:12.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] 2024-07-31T09:59:15.302847+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.048170][ T4150] pci 0000:00:12.0: BAR 4: assigned [mem 0x84002c000-0x84002ffff 64bit pref] 2024-07-31T09:59:15.306097+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.051709][ T4150] pci 0000:00:12.0: BAR 1: assigned [mem 0xc000b000-0xc000bfff] 2024-07-31T09:59:15.312534+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.054887][ T4150] pci 0000:00:12.0: BAR 0: assigned [io 0x18c0-0x18ff] 2024-07-31T09:59:15.312544+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.059215][ T4150] virtio-pci 0000:00:12.0: enabling device (0000 -> 0003) 2024-07-31T09:59:15.354376+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.095607][ T4150] virtio_blk virtio15: 1/0/0 default/read/poll queues 2024-07-31T09:59:15.354391+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152064.099154][ T4150] virtio_blk virtio15: [vdm] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) 2024-07-31T09:59:16.594383+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.338395][T13186] pci 0000:00:16.0: [1af4:1001] type 00 class 0x010000 2024-07-31T09:59:16.594402+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.340825][T13186] pci 0000:00:16.0: reg 0x10: [io 0x0000-0x003f] 2024-07-31T09:59:16.597977+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.343158][T13186] pci 0000:00:16.0: reg 0x14: [mem 0x00000000-0x00000fff] 2024-07-31T09:59:16.600697+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.346887][T13186] pci 0000:00:16.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] 2024-07-31T09:59:16.604373+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.350424][T13186] pci 0000:00:16.0: BAR 4: assigned [mem 0x84003c000-0x84003ffff 64bit pref] 2024-07-31T09:59:16.609534+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.353175][T13186] pci 0000:00:16.0: BAR 1: assigned [mem 0xc000f000-0xc000ffff] 2024-07-31T09:59:16.609548+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.355505][T13186] pci 0000:00:16.0: BAR 0: assigned [io 0x1cc0-0x1cff] 2024-07-31T09:59:16.612360+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.358322][T13186] virtio-pci 0000:00:16.0: enabling device (0000 -> 0003) 2024-07-31T09:59:16.658893+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.398143][T13186] virtio_blk virtio19: 1/0/0 default/read/poll queues 2024-07-31T09:59:16.658923+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152065.402764][T13186] virtio_blk virtio19: [vdq] 16777216 512-byte logical blocks (8.59 GB/8.00 GiB) 2024-07-31T09:59:17.460249+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.198643][T20959] EXT4-fs (vdm): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. 2024-07-31T09:59:18.193097+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.937459][T13186] pci 0000:00:17.0: [1af4:1001] type 00 class 0x010000 2024-07-31T09:59:18.193118+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.939712][T13186] pci 0000:00:17.0: reg 0x10: [io 0x0000-0x003f] 2024-07-31T09:59:18.196281+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.942336][T13186] pci 0000:00:17.0: reg 0x14: [mem 0x00000000-0x00000fff] 2024-07-31T09:59:18.200250+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.945136][T13186] pci 0000:00:17.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] 2024-07-31T09:59:18.206179+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.950085][T13186] pci 0000:00:17.0: BAR 4: assigned [mem 0x840040000-0x840043fff 64bit pref] 2024-07-31T09:59:18.206191+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.952617][T13186] pci 0000:00:17.0: BAR 1: assigned [mem 0xc0010000-0xc0010fff] 2024-07-31T09:59:18.209127+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.954842][T13186] pci 0000:00:17.0: BAR 0: assigned [io 0x2000-0x203f] 2024-07-31T09:59:18.213937+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.957890][T13186] virtio-pci 0000:00:17.0: enabling device (0000 -> 0003) 2024-07-31T09:59:18.252260+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.994354][T13186] virtio_blk virtio20: 1/0/0 default/read/poll queues 2024-07-31T09:59:18.252280+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152066.998244][T13186] virtio_blk virtio20: [vdr] 10485760 512-byte logical blocks (5.37 GB/5.00 GiB) 2024-07-31T09:59:18.252282+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152067.000102][T18529] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:59:19.120257+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152067.863054][T21301] EXT4-fs (vdq): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. 2024-07-31T09:59:19.817640+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152068.555337][T32068] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:59:20.360327+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152069.101240][T21934] EXT4-fs (vdr): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. 2024-07-31T09:59:20.884261+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152069.627976][T28648] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:59:22.974446+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.717891][T31337] pci 0000:00:18.0: [1af4:1001] type 00 class 0x010000 2024-07-31T09:59:22.974463+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.720472][T31337] pci 0000:00:18.0: reg 0x10: [io 0x0000-0x003f] 2024-07-31T09:59:22.978060+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.722970][T31337] pci 0000:00:18.0: reg 0x14: [mem 0x00000000-0x00000fff] 2024-07-31T09:59:22.982511+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.726800][T31337] pci 0000:00:18.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] 2024-07-31T09:59:22.988972+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.732498][T31337] pci 0000:00:18.0: BAR 4: assigned [mem 0x840044000-0x840047fff 64bit pref] 2024-07-31T09:59:22.992973+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.737537][T31337] pci 0000:00:18.0: BAR 1: assigned [mem 0xc0011000-0xc0011fff] 2024-07-31T09:59:22.999343+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.741452][T31337] pci 0000:00:18.0: BAR 0: assigned [io 0x2040-0x207f] 2024-07-31T09:59:22.999351+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.744085][T31337] virtio-pci 0000:00:18.0: enabling device (0000 -> 0003) 2024-07-31T09:59:23.047715+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.785354][T31337] virtio_blk virtio21: 1/0/0 default/read/poll queues 2024-07-31T09:59:23.047735+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152071.789956][T31337] virtio_blk virtio21: [vds] 16777216 512-byte logical blocks (8.59 GB/8.00 GiB) 2024-07-31T09:59:25.163554+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152073.900643][T22620] EXT4-fs (vds): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. 2024-07-31T09:59:25.772252+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152074.516353][T26537] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T09:59:42.907290+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.648158][T18248] pci 0000:00:19.0: [1af4:1001] type 00 class 0x010000 2024-07-31T09:59:42.907326+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.651593][T18248] pci 0000:00:19.0: reg 0x10: [io 0x0000-0x003f] 2024-07-31T09:59:42.911606+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.655042][T18248] pci 0000:00:19.0: reg 0x14: [mem 0x00000000-0x00000fff] 2024-07-31T09:59:42.916406+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.659524][T18248] pci 0000:00:19.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] 2024-07-31T09:59:42.922048+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.665412][T18248] pci 0000:00:19.0: BAR 4: assigned [mem 0x840048000-0x84004bfff 64bit pref] 2024-07-31T09:59:42.924650+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.669785][T18248] pci 0000:00:19.0: BAR 1: assigned [mem 0xc0012000-0xc0012fff] 2024-07-31T09:59:42.929553+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.672324][T18248] pci 0000:00:19.0: BAR 0: assigned [io 0x2080-0x20bf] 2024-07-31T09:59:42.929566+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.674804][T18248] virtio-pci 0000:00:19.0: enabling device (0000 -> 0003) 2024-07-31T09:59:42.969217+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.709738][T18248] virtio_blk virtio22: 1/0/0 default/read/poll queues 2024-07-31T09:59:42.969238+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152091.713087][T18248] virtio_blk virtio22: [vdt] 4194304 512-byte logical blocks (2.15 GB/2.00 GiB) 2024-07-31T09:59:44.999648+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152093.736777][T25040] EXT4-fs (vdt): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. 2024-07-31T09:59:45.472273+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152094.216002][T18171] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 2024-07-31T10:11:25.967001+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.578273][T13374] thanos invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=937 2024-07-31T10:11:25.968146+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.585414][T13374] CPU: 0 PID: 13374 Comm: thanos Not tainted 5.14.21-150500.55.65-default #1 (closed) SLE15-SP5 16a69e3ef5df74ca3142d0eb99487f215875bfbb 2024-07-31T10:11:25.968149+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.593993][T13374] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.14.0-1.module+el8.4.0+8855+a9e237a9 04/01/2014 2024-07-31T10:11:25.968154+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.602372][T13374] Call Trace: 2024-07-31T10:11:25.968156+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.606214][T13374] 2024-07-31T10:11:25.968158+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.609057][T13374] dump_stack_lvl+0x58/0x7b 2024-07-31T10:11:25.968159+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.611492][T13374] dump_header+0x4a/0x220 2024-07-31T10:11:25.968160+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.615580][T13374] oom_kill_process+0xe8/0x140 2024-07-31T10:11:25.968161+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.619692][T13374] out_of_memory+0x113/0x580 2024-07-31T10:11:25.968161+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.622673][T13374] mem_cgroup_out_of_memory+0xe3/0x100 2024-07-31T10:11:25.968162+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.625379][T13374] try_charge_memcg+0x6bb/0x700 2024-07-31T10:11:25.968162+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.629293][T13374] ? __alloc_pages+0x180/0x320 2024-07-31T10:11:25.968164+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.632920][T13374] charge_memcg+0x40/0xa0 2024-07-31T10:11:25.968165+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.636115][T13374] __mem_cgroup_charge+0x2c/0xa0 2024-07-31T10:11:25.968175+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.639928][T13374] __handle_mm_fault+0xa37/0x1230 2024-07-31T10:11:25.968175+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.643051][T13374] handle_mm_fault+0xd5/0x290 2024-07-31T10:11:25.968176+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.646317][T13374] do_user_addr_fault+0x1eb/0x730 2024-07-31T10:11:25.968177+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.649699][T13374] exc_page_fault+0x67/0x150 2024-07-31T10:11:25.968177+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.652786][T13374] asm_exc_page_fault+0x59/0x60 2024-07-31T10:11:25.968178+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.655778][T13374] RIP: 0033:0x1355e29 2024-07-31T10:11:25.968179+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.658767][T13374] Code: 49 83 fe 08 7d 11 49 8b 1f 48 89 1f 4c 29 f1 4c 01 f7 4d 01 f6 eb e9 48 89 f8 48 01 cf 48 83 f9 00 0f 8e 8b fd ff ff 49 8b 1f <48> 89 18 49 83 c7 08 48 83 c0 08 48 83 e9 08 eb e2 41 8a 1f 88 1f 2024-07-31T10:11:25.968181+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.668630][T13374] RSP: 002b:000000c087fc0990 EFLAGS: 00010202 2024-07-31T10:11:25.968181+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.670724][T13374] RAX: 000000c0b83c1ffd RBX: 3530312e32372e30 RCX: 0000000000000020 2024-07-31T10:11:25.968182+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.673658][T13374] RDX: 0000000000007699 RSI: 000000c005422ff8 RDI: 000000c0b83c201d 2024-07-31T10:11:25.968183+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.676236][T13374] RBP: 000000c087fc09c0 R08: 000000c0b8308000 R09: 0000000000124d65 2024-07-31T10:11:25.968183+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.679029][T13374] R10: 000000c0b842cd65 R11: 000000c005400003 R12: 000000000002c6f9 2024-07-31T10:11:25.968184+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.681434][T13374] R13: 000000c00542c6fc R14: 0000000000007699 R15: 000000c0b83ba964 2024-07-31T10:11:25.968184+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.683702][T13374] 2024-07-31T10:11:25.968185+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.685222][T13374] memory: usage 3145728kB, limit 3145728kB, failcnt 217156 2024-07-31T10:11:25.974565+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.689961][T13374] memory+swap: usage 3145728kB, limit 9007199254740988kB, failcnt 0 2024-07-31T10:11:25.974590+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.693977][T13374] kmem: usage 8904kB, limit 9007199254740988kB, failcnt 0 2024-07-31T10:11:26.064263+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.698166][T13374] Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6706d9b8_55f2_4bdf_8fa7_2b77c03fdbda.slice: 2024-07-31T10:11:26.064276+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] anon 3212107776 2024-07-31T10:11:26.064277+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] file 0 2024-07-31T10:11:26.064278+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] kernel_stack 212992 2024-07-31T10:11:26.064279+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] pagetables 6627328 2024-07-31T10:11:26.064282+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] percpu 46208 2024-07-31T10:11:26.064285+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] sock 0 2024-07-31T10:11:26.064287+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] shmem 0 2024-07-31T10:11:26.064289+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] file_mapped 0 2024-07-31T10:11:26.064290+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] file_dirty 0 2024-07-31T10:11:26.064291+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] file_writeback 0 2024-07-31T10:11:26.064292+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] swapcached 0 2024-07-31T10:11:26.064293+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] anon_thp 1505755136 2024-07-31T10:11:26.064295+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] file_thp 0 2024-07-31T10:11:26.064298+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] shmem_thp 0 2024-07-31T10:11:26.064300+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] inactive_anon 3212095488 2024-07-31T10:11:26.064301+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] active_anon 12288 2024-07-31T10:11:26.064302+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] inactive_file 0 2024-07-31T10:11:26.064303+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] active_file 0 2024-07-31T10:11:26.064306+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] unevictable 0 2024-07-31T10:11:26.064307+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] slab_reclaimable 1887624 2024-07-31T10:11:26.064309+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] slab_unreclaimable 266896 2024-07-31T10:11:26.064311+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] slab 2154520 2024-07-31T10:11:26.064313+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.727156][T13374] workingset_refault_anon 0 2024-07-31T10:11:26.064314+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.777055][T13374] Tasks state (memory values in pages): 2024-07-31T10:11:26.064315+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.779543][T13374] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name 2024-07-31T10:11:26.064322+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.783661][T13374] [ 13296] 65535 13296 243 1 28672 0 -998 pause 2024-07-31T10:11:26.072924+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.787943][T13374] [ 13344] 1001 13344 1203725 784625 6619136 0 937 thanos 2024-07-31T10:11:26.072942+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.792330][T13374] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-70739437830d4feabefa920f3ac18c70f46cde1c0caa370cca1976b445f3d35d.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6706d9b8_55f2_4bdf_8fa7_2b77c03fdbda.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6706d9b8_55f2_4bdf_8fa7_2b77c03fdbda.slice/cri-containerd-70739437830d4feabefa920f3ac18c70f46cde1c0caa370cca1976b445f3d35d.scope,task=thanos,pid=13344,uid=1001 2024-07-31T10:11:26.100666+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.812223][T13374] Memory cgroup out of memory: Killed process 13344 (thanos) total-vm:4814900kB, anon-rss:3136288kB, file-rss:2212kB, shmem-rss:0kB, UID:1001 pgtables:6464kB oom_score_adj:937 2024-07-31T10:11:26.224246+00:00 mgmt-cluster-cp-2d436cec5b-rmtgw kernel: [152794.938300][ T73] oom_reaper: reaped process 13344 (thanos), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB