• Tetsuo Handa's avatar
    kernel/hung_task.c: show all hung tasks before panic · 401c636a
    Tetsuo Handa authored
    When we get a hung task it can often be valuable to see _all_ the hung
    tasks on the system before calling panic().
    
    Quoting from https://syzkaller.appspot.com/text?tag=CrashReport&id=5316056503549952
    ----------------------------------------
    INFO: task syz-executor0:6540 blocked for more than 120 seconds.
          Not tainted 4.16.0+ #13
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    syz-executor0   D23560  6540   4521 0x80000004
    Call Trace:
     context_switch kernel/sched/core.c:2848 [inline]
     __schedule+0x8fb/0x1ef0 kernel/sched/core.c:3490
     schedule+0xf5/0x430 kernel/sched/core.c:3549
     schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3607
     __mutex_lock_common kernel/locking/mutex.c:833 [inline]
     __mutex_lock+0xb7f/0x1810 kernel/locking/mutex.c:893
     mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
     lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355
     __blkdev_driver_ioctl block/ioctl.c:303 [inline]
     blkdev_ioctl+0x1759/0x1e00 block/ioctl.c:601
     ioctl_by_bdev+0xa5/0x110 fs/block_dev.c:2060
     isofs_get_last_session fs/isofs/inode.c:567 [inline]
     isofs_fill_super+0x2ba9/0x3bc0 fs/isofs/inode.c:660
     mount_bdev+0x2b7/0x370 fs/super.c:1119
     isofs_mount+0x34/0x40 fs/isofs/inode.c:1560
     mount_fs+0x66/0x2d0 fs/super.c:1222
     vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
     vfs_kern_mount fs/namespace.c:2514 [inline]
     do_new_mount fs/namespace.c:2517 [inline]
     do_mount+0xea4/0x2b90 fs/namespace.c:2847
     ksys_mount+0xab/0x120 fs/namespace.c:3063
     SYSC_mount fs/namespace.c:3077 [inline]
     SyS_mount+0x39/0x50 fs/namespace.c:3074
     do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
     entry_SYSCALL_64_after_hwframe+0x42/0xb7
    (...snipped...)
    Showing all locks held in the system:
    (...snipped...)
    2 locks held by syz-executor0/6540:
     #0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: alloc_super fs/super.c:211 [inline]
     #0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: sget_userns+0x3b2/0xe60 fs/super.c:502 /* down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING); */
     #1: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
    (...snipped...)
    3 locks held by syz-executor7/6541:
     #0: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
     #1: 000000007bf3d3f9 (&bdev->bd_mutex){+.+.}, at: blkdev_reread_part+0x1e/0x40 block/ioctl.c:192
     #2: 00000000566d4c39 (&type->s_umount_key#50){.+.+}, at: __get_super.part.10+0x1d3/0x280 fs/super.c:663 /* down_read(&sb->s_umount); */
    ----------------------------------------
    
    When reporting an AB-BA deadlock like shown above, it would be nice if
    trace of PID=6541 is printed as well as trace of PID=6540 before calling
    panic().
    
    Showing hung tasks up to /proc/sys/kernel/hung_task_warnings could delay
    calling panic() but normally there should not be so many hung tasks.
    
    Link: http://lkml.kernel.org/r/201804050705.BHE57833.HVFOFtSOMQJFOL@I-love.SAKURA.ne.jpSigned-off-by: default avatarTetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
    Acked-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    Acked-by: default avatarDmitry Vyukov <dvyukov@google.com>
    Cc: Vegard Nossum <vegard.nossum@oracle.com>
    Cc: Mandeep Singh Baines <msb@chromium.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Ingo Molnar <mingo@kernel.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    401c636a
Name
Last commit
Last update
..
bpf Loading commit data...
cgroup Loading commit data...
configs Loading commit data...
debug Loading commit data...
events Loading commit data...
gcov Loading commit data...
irq Loading commit data...
livepatch Loading commit data...
locking Loading commit data...
power Loading commit data...
printk Loading commit data...
rcu Loading commit data...
sched Loading commit data...
time Loading commit data...
trace Loading commit data...
.gitignore Loading commit data...
Kconfig.freezer Loading commit data...
Kconfig.hz Loading commit data...
Kconfig.locks Loading commit data...
Kconfig.preempt Loading commit data...
Makefile Loading commit data...
acct.c Loading commit data...
async.c Loading commit data...
audit.c Loading commit data...
audit.h Loading commit data...
audit_fsnotify.c Loading commit data...
audit_tree.c Loading commit data...
audit_watch.c Loading commit data...
auditfilter.c Loading commit data...
auditsc.c Loading commit data...
backtracetest.c Loading commit data...
bounds.c Loading commit data...
capability.c Loading commit data...
compat.c Loading commit data...
configs.c Loading commit data...
context_tracking.c Loading commit data...
cpu.c Loading commit data...
cpu_pm.c Loading commit data...
crash_core.c Loading commit data...
crash_dump.c Loading commit data...
cred.c Loading commit data...
delayacct.c Loading commit data...
dma.c Loading commit data...
elfcore.c Loading commit data...
exec_domain.c Loading commit data...
exit.c Loading commit data...
extable.c Loading commit data...
fail_function.c Loading commit data...
fork.c Loading commit data...
freezer.c Loading commit data...
futex.c Loading commit data...
futex_compat.c Loading commit data...
groups.c Loading commit data...
hung_task.c Loading commit data...
irq_work.c Loading commit data...
jump_label.c Loading commit data...
kallsyms.c Loading commit data...
kcmp.c Loading commit data...
kcov.c Loading commit data...
kexec.c Loading commit data...
kexec_core.c Loading commit data...
kexec_file.c Loading commit data...
kexec_internal.h Loading commit data...
kmod.c Loading commit data...
kprobes.c Loading commit data...
ksysfs.c Loading commit data...
kthread.c Loading commit data...
latencytop.c Loading commit data...
memremap.c Loading commit data...
module-internal.h Loading commit data...
module.c Loading commit data...
module_signing.c Loading commit data...
notifier.c Loading commit data...
nsproxy.c Loading commit data...
padata.c Loading commit data...
panic.c Loading commit data...
params.c Loading commit data...
pid.c Loading commit data...
pid_namespace.c Loading commit data...
profile.c Loading commit data...
ptrace.c Loading commit data...
range.c Loading commit data...
reboot.c Loading commit data...
relay.c Loading commit data...
resource.c Loading commit data...
seccomp.c Loading commit data...
signal.c Loading commit data...
smp.c Loading commit data...
smpboot.c Loading commit data...
smpboot.h Loading commit data...
softirq.c Loading commit data...
stacktrace.c Loading commit data...
stop_machine.c Loading commit data...
sys.c Loading commit data...
sys_ni.c Loading commit data...
sysctl.c Loading commit data...
sysctl_binary.c Loading commit data...
task_work.c Loading commit data...
taskstats.c Loading commit data...
test_kprobes.c Loading commit data...
torture.c Loading commit data...
tracepoint.c Loading commit data...
tsacct.c Loading commit data...
ucount.c Loading commit data...
uid16.c Loading commit data...
uid16.h Loading commit data...
umh.c Loading commit data...
up.c Loading commit data...
user-return-notifier.c Loading commit data...
user.c Loading commit data...
user_namespace.c Loading commit data...
utsname.c Loading commit data...
utsname_sysctl.c Loading commit data...
watchdog.c Loading commit data...
watchdog_hld.c Loading commit data...
workqueue.c Loading commit data...
workqueue_internal.h Loading commit data...