lockdep detected circular locking between rtnl_mutex and pm_chain_head.rwsem [wireguard and r8152]

Sun Sep 22 01:11:38 UTC 2024

Hi!

While trying out a kernel at commit
88264981f2082248e892a706b2c5004650faac54 (latest mainline) with
lockdep enabled, I hit a lockdep warning - it looks like wireguard
takes the rtnl_lock in a PM callback (meaning pm_chain_head.rwsem is
already held), while r8152 registers a PM callback in a context where
the rtnl_lock is held, and this makes lockdep unhappy. But I don't
know enough about the PM code to know which of those is the problem or
whether this race could even occur. I'm also not sure whether this is
a regression - I don't usually run lockdep kernels on this machine.

[ 1749.181131] PM: suspend entry (s2idle)
[ 1749.209736] Filesystems sync: 0.028 seconds

[ 1749.220240] ======================================================
[ 1749.220242] WARNING: possible circular locking dependency detected
[ 1749.220244] 6.11.0-slowkasan+ #140 Not tainted
[ 1749.220247] ------------------------------------------------------
[ 1749.220249] systemd-sleep/5239 is trying to acquire lock:
[ 1749.220252] ffffffffb1156c88 (rtnl_mutex){+.+.}-{3:3}, at:
wg_pm_notification (drivers/net/wireguard/device.c:81
drivers/net/wireguard/device.c:64)
[ 1749.220265]
but task is already holding lock:
[ 1749.220267] ffffffffb077e170 ((pm_chain_head).rwsem){++++}-{3:3},
at: blocking_notifier_call_chain_robust (kernel/notifier.c:128
kernel/notifier.c:353 kernel/notifier.c:341)
[ 1749.220277]
which lock already depends on the new lock.

[ 1749.220279]
the existing dependency chain (in reverse order) is:
[ 1749.220281]
-> #1 ((pm_chain_head).rwsem){++++}-{3:3}:
[ 1749.220287] down_write (./arch/x86/include/asm/preempt.h:79
kernel/locking/rwsem.c:1304 kernel/locking/rwsem.c:1315
kernel/locking/rwsem.c:1580)
[ 1749.220292] blocking_notifier_chain_register (kernel/notifier.c:272
kernel/notifier.c:290)
[ 1749.220295] rtl8152_open (drivers/net/usb/r8152.c:6994)
[ 1749.220300] __dev_open (net/core/dev.c:1476)
[ 1749.220304] __dev_change_flags (net/core/dev.c:8837)
[ 1749.220308] dev_change_flags (net/core/dev.c:8909)
[ 1749.220311] do_setlink (net/core/rtnetlink.c:2900)
[ 1749.220315] __rtnl_newlink (net/core/rtnetlink.c:3696)
[ 1749.220318] rtnl_newlink (net/core/rtnetlink.c:3744)
[ 1749.220322] rtnetlink_rcv_msg (net/core/rtnetlink.c:6646)
[ 1749.220325] netlink_rcv_skb (net/netlink/af_netlink.c:2550)
[ 1749.220329] netlink_unicast (net/netlink/af_netlink.c:1331
net/netlink/af_netlink.c:1357)
[ 1749.220332] netlink_sendmsg (net/netlink/af_netlink.c:1901)
[ 1749.220335] ____sys_sendmsg (net/socket.c:730 net/socket.c:745
net/socket.c:2603)
[ 1749.220339] ___sys_sendmsg (net/socket.c:2659)
[ 1749.220342] __sys_sendmsg (net/socket.c:2686)
[ 1749.220344] do_syscall_64 (arch/x86/entry/common.c:52
arch/x86/entry/common.c:83)
[ 1749.220348] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[ 1749.220352]
-> #0 (rtnl_mutex){+.+.}-{3:3}:
[ 1749.220357] __lock_acquire (kernel/locking/lockdep.c:3159
kernel/locking/lockdep.c:3277 kernel/locking/lockdep.c:3901
kernel/locking/lockdep.c:5199)
[ 1749.220362] lock_acquire (kernel/locking/lockdep.c:467
kernel/locking/lockdep.c:5824 kernel/locking/lockdep.c:5787)
[ 1749.220365] __mutex_lock (kernel/locking/mutex.c:610
kernel/locking/mutex.c:752)
[ 1749.220369] wg_pm_notification (drivers/net/wireguard/device.c:81
drivers/net/wireguard/device.c:64)
[ 1749.220372] notifier_call_chain (kernel/notifier.c:93)
[ 1749.220375] blocking_notifier_call_chain_robust
(kernel/notifier.c:129 kernel/notifier.c:353 kernel/notifier.c:341)
[ 1749.220378] pm_notifier_call_chain_robust
(./include/linux/notifier.h:207 kernel/power/main.c:104)
[ 1749.220382] pm_suspend (kernel/power/suspend.c:367
kernel/power/suspend.c:588 kernel/power/suspend.c:625)
[ 1749.220386] state_store (kernel/power/main.c:746)
[ 1749.220389] kernfs_fop_write_iter (fs/kernfs/file.c:334)
[ 1749.220393] vfs_write (fs/read_write.c:590 fs/read_write.c:683)
[ 1749.220397] ksys_write (fs/read_write.c:736)
[ 1749.220399] do_syscall_64 (arch/x86/entry/common.c:52
arch/x86/entry/common.c:83)
[ 1749.220402] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[ 1749.220406]
other info that might help us debug this:

[ 1749.220408]  Possible unsafe locking scenario:

[ 1749.220409]        CPU0                    CPU1
[ 1749.220411]        ----                    ----
[ 1749.220413]   rlock((pm_chain_head).rwsem);
[ 1749.220416]                                lock(rtnl_mutex);
[ 1749.220420]                                lock((pm_chain_head).rwsem);
[ 1749.220423]   lock(rtnl_mutex);
[ 1749.220426]
*** DEADLOCK ***

[ 1749.220428] 5 locks held by systemd-sleep/5239:
[ 1749.220430] #0: ffff888125d2e3f8 (sb_writers#6){.+.+}-{0:0}, at:
ksys_write (fs/read_write.c:736)
[ 1749.220439] #1: ffff8881e5cb9888 (&of->mutex){+.+.}-{3:3}, at:
kernfs_fop_write_iter (fs/kernfs/file.c:326)
[ 1749.220447] #2: ffff888460aee2d8 (kn->active#166){.+.+}-{0:0}, at:
kernfs_fop_write_iter (fs/kernfs/file.c:326)
[ 1749.220455] #3: ffffffffb0757008
(system_transition_mutex){+.+.}-{3:3}, at: pm_suspend
(kernel/power/suspend.c:574 kernel/power/suspend.c:625)
[ 1749.220463] #4: ffffffffb077e170
((pm_chain_head).rwsem){++++}-{3:3}, at:
blocking_notifier_call_chain_robust (kernel/notifier.c:128
kernel/notifier.c:353 kernel/notifier.c:341)
[ 1749.220471]
stack backtrace:
[ 1749.220474] CPU: 1 UID: 0 PID: 5239 Comm: systemd-sleep Not tainted
6.11.0-slowkasan+ #140
[ 1749.220478] Hardware name: [...]
[ 1749.220480] Call Trace:
[ 1749.220483]  <TASK>
[ 1749.220485] dump_stack_lvl (lib/dump_stack.c:124)
[ 1749.220491] print_circular_bug (kernel/locking/lockdep.c:2077)
[ 1749.220496] check_noncircular (kernel/locking/lockdep.c:2203)
[...]
[ 1749.220519] __lock_acquire (kernel/locking/lockdep.c:3159
kernel/locking/lockdep.c:3277 kernel/locking/lockdep.c:3901
kernel/locking/lockdep.c:5199)
[...]
[ 1749.220546] lock_acquire (kernel/locking/lockdep.c:467
kernel/locking/lockdep.c:5824 kernel/locking/lockdep.c:5787)
[...]
[ 1749.220577] __mutex_lock (kernel/locking/mutex.c:610
kernel/locking/mutex.c:752)
[...]
[ 1749.220627] wg_pm_notification (drivers/net/wireguard/device.c:81
drivers/net/wireguard/device.c:64)
[ 1749.220631] notifier_call_chain (kernel/notifier.c:93)
[ 1749.220636] blocking_notifier_call_chain_robust
(kernel/notifier.c:129 kernel/notifier.c:353 kernel/notifier.c:341)
[...]
[ 1749.220649] pm_notifier_call_chain_robust
(./include/linux/notifier.h:207 kernel/power/main.c:104)
[ 1749.220652] pm_suspend (kernel/power/suspend.c:367
kernel/power/suspend.c:588 kernel/power/suspend.c:625)
[ 1749.220656] state_store (kernel/power/main.c:746)
[ 1749.220661] kernfs_fop_write_iter (fs/kernfs/file.c:334)
[ 1749.220665] vfs_write (fs/read_write.c:590 fs/read_write.c:683)
[...]
[ 1749.220693] ksys_write (fs/read_write.c:736)
[...]
[ 1749.220701] do_syscall_64 (arch/x86/entry/common.c:52
arch/x86/entry/common.c:83)
[ 1749.220704] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
[ 1749.220708] RIP: 0033:0x7fe2e2917240
[...]
[ 1749.220735]  </TASK>
[ 1749.223599] Freezing user space processes
[ 1749.226307] Freezing user space processes completed (elapsed 0.002 seconds)