locking: Backport upstream PREEMPT_RT locking changes
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2007032
MR: !42 (merged)
This MR backports the v5.15 PREEMPT_RT locking changes for better integration with RT kernel. All the patches are applied cleanly without conflict.
Signed-off-by: Waiman Long longman@redhat.com
Waiman Long (88):
locking/mutex: Use try_cmpxchg()
locking/mutex: Fix HANDOFF condition
locking/mutex: Introduce __mutex_trylock_or_handoff()
locking/mutex: Add MUTEX_WARN_ON
locking/local_lock: Add missing owner initialization
locking/rtmutex: Set proper wait context for lockdep
sched/wakeup: Split out the wakeup ->__state check
sched/wakeup: Introduce the TASK_RTLOCK_WAIT state bit
sched/wakeup: Reorganize the current::__state helpers
sched/wakeup: Prepare for RT sleeping spin/rwlocks
sched/core: Rework the __schedule() preempt argument
sched/core: Provide a scheduling point for RT locks
sched/wake_q: Provide WAKE_Q_HEAD_INITIALIZER()
locking/rtmutex: Remove rt_mutex_is_locked()
locking/rtmutex: Convert macros to inlines
locking/rtmutex: Switch to from cmpxchg_*() to try_cmpxchg_*()
locking/rtmutex: Split API from implementation
locking/rtmutex: Split out the inner parts of 'struct rtmutex'
locking/rtmutex: Provide rt_mutex_slowlock_locked()
locking/rtmutex: Provide rt_mutex_base_is_locked()
locking/rt: Add base code for RT rw_semaphore and rwlock
locking/rwsem: Add rtmutex based R/W semaphore implementation
locking/rtmutex: Add wake_state to rt_mutex_waiter
locking/rtmutex: Provide rt_wake_q_head and helpers
locking/rtmutex: Use rt_mutex_wake_q_head
locking/rtmutex: Prepare RT rt_mutex_wake_q for RT locks
locking/rtmutex: Guard regular sleeping locks specific functions
locking/spinlock: Split the lock types header, and move the raw types
into <linux/spinlock_types_raw.h>
locking/rtmutex: Prevent future include recursion hell
locking/lockdep: Reduce header dependencies in <linux/debug_locks.h>
rbtree: Split out the rbtree type definitions into
<linux/rbtree_types.h>
locking/rtmutex: Reduce <linux/rtmutex.h> header dependencies, only
include <linux/rbtree_types.h>
locking/spinlock: Provide RT specific spinlock_t
locking/spinlock: Provide RT variant header: <linux/spinlock_rt.h>
locking/rtmutex: Provide the spin/rwlock core lock function
locking/spinlock: Provide RT variant
locking/rwlock: Provide RT variant
locking/rtmutex: Squash !RT tasks to DEFAULT_PRIO
locking/mutex: Consolidate core headers, remove
kernel/locking/mutex-debug.h
locking/mutex: Move the 'struct mutex_waiter' definition from
<linux/mutex.h> to the internal header
locking/ww_mutex: Move the ww_mutex definitions from <linux/mutex.h>
into <linux/ww_mutex.h>
locking/mutex: Make mutex::wait_lock raw
locking/ww_mutex: Simplify lockdep annotations
locking/ww_mutex: Gather mutex_waiter initialization
locking/ww_mutex: Split up ww_mutex_unlock()
locking/ww_mutex: Split out the W/W implementation logic into
kernel/locking/ww_mutex.h
locking/ww_mutex: Remove the __sched annotation from ww_mutex APIs
locking/ww_mutex: Abstract out the waiter iteration
locking/ww_mutex: Abstract out waiter enqueueing
locking/ww_mutex: Abstract out mutex accessors
locking/ww_mutex: Abstract out mutex types
locking/ww_mutex: Abstract out internal lock accesses
locking/ww_mutex: Implement rt_mutex accessors
locking/ww_mutex: Add RT priority to W/W order
locking/ww_mutex: Add rt_mutex based lock type and accessors
locking/rtmutex: Extend the rtmutex core to support ww_mutex
locking/ww_mutex: Implement rtmutex based ww_mutex API functions
locking/rtmutex: Add mutex variant for RT
lib/test_lockup: Adapt to changed variables
futex: Validate waiter correctly in futex_proxy_trylock_atomic()
futex: Clean up stale comments
futex: Clarify futex_requeue() PI handling
futex: Remove bogus condition for requeue PI
futex: Correct the number of requeued waiters for PI
futex: Restructure futex_requeue()
futex: Clarify comment in futex_requeue()
futex: Reorder sanity checks in futex_requeue()
futex: Simplify handle_early_requeue_pi_wakeup()
futex: Prevent requeue_pi() lock nesting issue on RT
locking/rtmutex: Prevent lockdep false positive with PI futexes
preempt: Adjust PREEMPT_LOCK_OFFSET for RT
locking/rtmutex: Implement equal priority lock stealing
locking/rtmutex: Add adaptive spinwait mechanism
locking/spinlock/rt: Prepare for RT local_lock
locking/local_lock: Add PREEMPT_RT support
locking/rtmutex: Dont dereference waiter lockless
locking/rtmutex: Dequeue waiter on ww_mutex deadlock
locking/rtmutex: Prevent spurious EDEADLK return caused by ww_mutexes
locking/rtmutex: Return success on deadlock for ww_mutex waiters
locking/rwsem: Add missing __init_rwsem() for PREEMPT_RT
locking/rtmutex: Fix ww_mutex deadlock check
locking/rwbase: Properly match set_and_save_state() to restore_state()
locking/rwbase: Extract __rwbase_write_trylock()
locking/rwbase: Take care of ordering guarantee for fastpath reader
futex: Return error code instead of assigning it without effect
locking/ww_mutex: Initialize waiter.ww_ctx properly
futex: Prevent inconsistent state and exit race
futex: Remove unused variable 'vpid' in futex_proxy_trylock_atomic()
include/linux/debug_locks.h | 3 +-
include/linux/local_lock_internal.h | 86 +-
include/linux/mutex.h | 92 ++-
include/linux/preempt.h | 4 +
include/linux/rbtree.h | 31 +-
include/linux/rbtree_types.h | 34 +
include/linux/rtmutex.h | 63 +-
include/linux/rwbase_rt.h | 39 +
include/linux/rwlock_rt.h | 140 ++++
include/linux/rwlock_types.h | 53 +-
include/linux/rwsem.h | 70 +-
include/linux/sched.h | 119 ++-
include/linux/sched/wake_q.h | 7 +-
include/linux/spinlock.h | 15 +-
include/linux/spinlock_api_smp.h | 3 +
include/linux/spinlock_rt.h | 159 ++++
include/linux/spinlock_types.h | 89 +-
include/linux/spinlock_types_raw.h | 73 ++
include/linux/ww_mutex.h | 50 +-
kernel/Kconfig.locks | 2 +-
kernel/futex.c | 609 +++++++++-----
kernel/locking/Makefile | 3 +-
kernel/locking/mutex-debug.c | 5 +-
kernel/locking/mutex-debug.h | 29 -
kernel/locking/mutex.c | 541 ++----------
kernel/locking/mutex.h | 48 +-
kernel/locking/rtmutex.c | 1192 +++++++++++++--------------
kernel/locking/rtmutex_api.c | 590 +++++++++++++
kernel/locking/rtmutex_common.h | 135 ++-
kernel/locking/rwbase_rt.c | 288 +++++++
kernel/locking/rwsem.c | 111 +++
kernel/locking/spinlock.c | 7 +
kernel/locking/spinlock_debug.c | 5 +
kernel/locking/spinlock_rt.c | 263 ++++++
kernel/locking/ww_mutex.h | 569 +++++++++++++
kernel/locking/ww_rt_mutex.c | 76 ++
kernel/rcu/tree_plugin.h | 6 +-
kernel/sched/core.c | 109 ++-
lib/Kconfig.debug | 11 +-
lib/test_lockup.c | 8 +-
40 files changed, 4100 insertions(+), 1637 deletions(-)
create mode 100644 include/linux/rbtree_types.h
create mode 100644 include/linux/rwbase_rt.h
create mode 100644 include/linux/rwlock_rt.h
create mode 100644 include/linux/spinlock_rt.h
create mode 100644 include/linux/spinlock_types_raw.h
delete mode 100644 kernel/locking/mutex-debug.h
create mode 100644 kernel/locking/rtmutex_api.c
create mode 100644 kernel/locking/rwbase_rt.c
create mode 100644 kernel/locking/spinlock_rt.c
create mode 100644 kernel/locking/ww_mutex.h
create mode 100644 kernel/locking/ww_rt_mutex.cEdited by Waiman Long