kdump - kexec_boot: RIP: 0010:smp_call_function_single+0xc9/0x180
Snippet of test failure
# https://s3.us-east-1.amazonaws.com/arr-cki-prod-datawarehouse-public/datawarehouse-public/3915975323/redhat:803254009/build_x86_64_redhat:803254009-x86_64-kernel/tests/3/results_0001/console.log/console.log
[ 32.658527] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [systemd:1]
[ 32.660749] Modules linked in: scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath i2c_dev fuse loop zram xfs crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 qxl virtio_console virtio_blk drm_ttm_helper ttm ata_generic pata_acpi serio_raw qemu_fw_cfg
[ 32.668235] CPU: 0 PID: 1 Comm: systemd Tainted: G D W ------- --- 6.3.0-0.rc1.ef5f68cc1f82.21.test.fc39.x86_64 #1
[ 32.670526] watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [systemd-journal:561]
[ 32.670723] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 32.672367] Modules linked in:
[ 32.673605] RIP: 0010:smp_call_function_single+0xc9/0x180
[ 32.674775] scsi_dh_rdac
[ 32.675436] Code: 85 c9 00 00 00 85 db 74 77 48 89 e6 44 89 e7 4c 89 74 24 10 4c 89 6c 24 18 e8 83 fe ff ff 89 c3 8b 44 24 08 a8 01 74 0b f3 90 <8b> 54 24 08 83 e2 01 75 f5 bf 01 00 00 00 e8 04 02 f5 ff 65 8b 05
[ 32.676598] scsi_dh_emc
[ 32.677170] RSP: 0018:ffffb82bc0013c00 EFLAGS: 00000202
[ 32.681047] scsi_dh_alua
[ 32.681615]
[ 32.682718] dm_multipath
[ 32.683284] RAX: 0000000000000011 RBX: 0000000000000000 RCX: ffff98b6c0de8038
[ 32.683631] i2c_dev
[ 32.684198] RDX: 0000000000000001 RSI: ffffb82bc0013c00 RDI: ffffb82bc0013c00
[ 32.685695] fuse
[ 32.686177] RBP: ffffb82bc0013c68 R08: ffffb82bc0013d40 R09: ffffb82bc0013c78
[ 32.687674] loop
[ 32.688102] R10: 0000000000000001 R11: 0000000000000110 R12: 0000000000000001
[ 32.689687] zram
[ 32.690116] R13: ffffb82bc0013c80 R14: ffffffff9930d4b0 R15: ffff98b6c5d01e88
[ 32.691608] xfs
[ 32.692036] FS: 00007fbfdeba8940(0000) GS:ffff98bca4c00000(0000) knlGS:0000000000000000
[ 32.693529] crct10dif_pclmul
[ 32.693944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 32.695619] crc32_pclmul
[ 32.696264] CR2: 00007fbfde7f76a0 CR3: 00000001067ae004 CR4: 00000000007706f0
[ 32.697469] crc32c_intel
[ 32.698060] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 32.699551] polyval_clmulni
[ 32.700118] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 32.701604] polyval_generic
[ 32.702235] PKRU: 55555554
[ 32.703719] ghash_clmulni_intel
[ 32.704343] Call Trace:
[ 32.704925] sha512_ssse3
[ 32.705637] <TASK>
[ 32.706160] qxl
[ 32.706741] ? __pfx_remote_function+0x10/0x10
[ 32.707195] virtio_console
[ 32.707612] perf_cgroup_attach+0x76/0xc0
[ 32.708549] virtio_blk
[ 32.709148] ? __pfx___perf_cgroup_move+0x10/0x10
[ 32.709994] drm_ttm_helper
[ 32.710541] cgroup_migrate_execute+0x379/0x480
[ 32.711528] ttm
[ 32.712126] cgroup_attach_task+0x14e/0x200
[ 32.713094] ata_generic
[ 32.713515] ? cgroup_attach_permissions+0x159/0x1d0
[ 32.714389] pata_acpi
[ 32.714949] __cgroup_procs_write+0x12f/0x160
[ 32.715991] serio_raw
[ 32.716512] cgroup_procs_write+0x13/0x20
[ 32.717429] qemu_fw_cfg
[ 32.717962] kernfs_fop_write_iter+0x12f/0x1c0
[ 32.718883]
[ 32.719430] vfs_write+0x241/0x400
[ 32.720369] CPU: 2 PID: 561 Comm: systemd-journal Tainted: G D W ------- --- 6.3.0-0.rc1.ef5f68cc1f82.21.test.fc39.x86_64 #1
[ 32.720723] ksys_write+0x6b/0xf0
[ 32.721445] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 32.724049] do_syscall_64+0x58/0x80
[ 32.724756] RIP: 0010:try_to_grab_pending+0xf7/0x170
[ 32.725973] ? exc_page_fault+0x70/0x170
[ 32.726737] Code: 41 5c 41 5d c3 cc cc cc cc 48 8d 7f 20 e8 b1 a6 0a 00 85 c0 0f 84 46 ff ff ff b8 01 00 00 00 e9 46 ff ff ff fb 0f 1f 44 00 00 <eb> b6 40 30 f6 4c 8b 26 e9 6e ff ff ff f6 03 02 75 59 48 8d 6b 08
[ 32.727801] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 32.728620] RSP: 0018:ffffb82bc04039b8 EFLAGS: 00000202
[ 32.732504] RIP: 0033:0x7fbfde721f44
[ 32.733562]
[ 32.734675] Code: 89 02 48 c7 c0 ff ff ff ff eb bd 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d bd c6 0d 00 00 74 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec 20 48 89
[ 32.735426] RAX: 0000000000000246 RBX: ffff98b6c094cc40 RCX: 0000000000000000
[ 32.735771] RSP: 002b:00007ffdcd04e148 EFLAGS: 00000202
[ 32.739612] RDX: 0000000000000001 RSI: ffff98b6c0402da0 RDI: ffff98b6c6db4000
[ 32.741110] ORIG_RAX: 0000000000000001
[ 32.742218] RBP: ffffb82bc04039e0 R08: ffff98b6c0402dd0 R09: ffffffff9b05b740
[ 32.743723] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fbfde721f44
[ 32.744535] R10: 0000000000000000 R11: 0000000000000000 R12: ffff98bca4c32730
[ 32.746033] RDX: 0000000000000004 RSI: 00007ffdcd04e34a RDI: 000000000000001d
[ 32.747521] R13: ffff98b6c09c2800 R14: 0000000000000000 R15: 0000000000000001
[ 32.749105] RBP: 00007ffdcd04e170 R08: 00007fbfde7f32a0 R09: 0000000000000000
[ 32.750592] FS: 00007f0219a31940(0000) GS:ffff98bca4d00000(0000) knlGS:0000000000000000
[ 32.752085] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000004
[ 32.753566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 32.755186] R13: 00007ffdcd04e34a R14: 000055f5432432a0 R15: 0000000000000004
[ 32.756472] CR2: 00007f0218400028 CR3: 0000000108806006 CR4: 00000000007706e0
[ 32.757554] </TASK>
[ 32.758844] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 32.797108] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 32.799023] PKRU: 55555554
[ 32.800028] Call Trace:
[ 32.800980] <TASK>
[ 32.801861] mod_delayed_work_on+0x45/0xa0
[ 32.803148] kblockd_mod_delayed_work_on+0x17/0x20
[ 32.804595] blk_mq_submit_bio+0x4f7/0x600
[ 32.805883] __submit_bio+0xf5/0x180
[ 32.807059] submit_bio_noacct_nocheck+0x163/0x370
[ 32.808485] iomap_submit_ioend+0x4b/0x80
[ 32.809839] iomap_do_writepage+0x4a2/0x7e0
[ 32.811152] write_cache_pages+0x13b/0x390
[ 32.812442] ? __pfx_iomap_do_writepage+0x10/0x10
[ 32.813867] iomap_writepages+0x1c/0x40
[ 32.815098] xfs_vm_writepages+0x79/0xb0 [xfs]
[ 32.817113] do_writepages+0xcb/0x1e0
[ 32.818310] filemap_fdatawrite_wbc+0x5f/0x80
[ 32.819655] __filemap_fdatawrite_range+0x58/0x80
[ 32.821055] file_write_and_wait_range+0x46/0xb0
[ 32.822422] xfs_file_fsync+0x5a/0x2a0 [xfs]
[ 32.823905] __x64_sys_fsync+0x37/0x60
[ 32.825106] do_syscall_64+0x58/0x80
[ 32.826264] ? exc_page_fault+0x70/0x170
[ 32.827480] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 32.828913] RIP: 0033:0x7f0219d28b94
[ 32.830046] Code: 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d 6d 5a 0d 00 00 74 13 b8 4a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 3c c3 0f 1f 00 55 48 89 e5 48 83 ec 10 89 7d
[ 32.834660] RSP: 002b:00007fffbd94ae18 EFLAGS: 00000202 ORIG_RAX: 000000000000004a
[ 32.836638] RAX: ffffffffffffffda RBX: 00005598c41422c0 RCX: 00007f0219d28b94
[ 32.838517] RDX: 00007f0218400000 RSI: 0000004abbc00000 RDI: 0000000000000016
[ 32.840440] RBP: 00007fffbd94ae30 R08: 00000a24751d11e1 R09: 0000000000000000
[ 32.842296] R10: 00007fffbd975080 R11: 0000000000000202 R12: 00007fffbd94b110
[ 32.844151] R13: 0000000100000000 R14: 0000000000000000 R15: 00005598c41422c0
[ 32.846007] </TASK>
Test logs on DataWarehouse
DataWarehouse issue
Additional details
- Similar DW issue: https://datawarehouse.cki-project.org/issue/503
In case opening the links above result in a 404 page on DataWarehouse, please make sure you are correctly logged into DataWarehouse via Red Hat SSO.