Skip to content

locking: Backport upstream PREEMPT_RT locking changes

Waiman Long requested to merge llong1/centos-stream-9:bz2007032_rtlock into main

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2007032
MR: !42 (merged)

This MR backports the v5.15 PREEMPT_RT locking changes for better integration with RT kernel. All the patches are applied cleanly without conflict.

Signed-off-by: Waiman Long longman@redhat.com

Waiman Long (88):
  locking/mutex: Use try_cmpxchg()
  locking/mutex: Fix HANDOFF condition
  locking/mutex: Introduce __mutex_trylock_or_handoff()
  locking/mutex: Add MUTEX_WARN_ON
  locking/local_lock: Add missing owner initialization
  locking/rtmutex: Set proper wait context for lockdep
  sched/wakeup: Split out the wakeup ->__state check
  sched/wakeup: Introduce the TASK_RTLOCK_WAIT state bit
  sched/wakeup: Reorganize the current::__state helpers
  sched/wakeup: Prepare for RT sleeping spin/rwlocks
  sched/core: Rework the __schedule() preempt argument
  sched/core: Provide a scheduling point for RT locks
  sched/wake_q: Provide WAKE_Q_HEAD_INITIALIZER()
  locking/rtmutex: Remove rt_mutex_is_locked()
  locking/rtmutex: Convert macros to inlines
  locking/rtmutex: Switch to from cmpxchg_*() to try_cmpxchg_*()
  locking/rtmutex: Split API from implementation
  locking/rtmutex: Split out the inner parts of 'struct rtmutex'
  locking/rtmutex: Provide rt_mutex_slowlock_locked()
  locking/rtmutex: Provide rt_mutex_base_is_locked()
  locking/rt: Add base code for RT rw_semaphore and rwlock
  locking/rwsem: Add rtmutex based R/W semaphore implementation
  locking/rtmutex: Add wake_state to rt_mutex_waiter
  locking/rtmutex: Provide rt_wake_q_head and helpers
  locking/rtmutex: Use rt_mutex_wake_q_head
  locking/rtmutex: Prepare RT rt_mutex_wake_q for RT locks
  locking/rtmutex: Guard regular sleeping locks specific functions
  locking/spinlock: Split the lock types header, and move the raw types
    into <linux/spinlock_types_raw.h>
  locking/rtmutex: Prevent future include recursion hell
  locking/lockdep: Reduce header dependencies in <linux/debug_locks.h>
  rbtree: Split out the rbtree type definitions into
    <linux/rbtree_types.h>
  locking/rtmutex: Reduce <linux/rtmutex.h> header dependencies, only
    include <linux/rbtree_types.h>
  locking/spinlock: Provide RT specific spinlock_t
  locking/spinlock: Provide RT variant header: <linux/spinlock_rt.h>
  locking/rtmutex: Provide the spin/rwlock core lock function
  locking/spinlock: Provide RT variant
  locking/rwlock: Provide RT variant
  locking/rtmutex: Squash !RT tasks to DEFAULT_PRIO
  locking/mutex: Consolidate core headers, remove
    kernel/locking/mutex-debug.h
  locking/mutex: Move the 'struct mutex_waiter' definition from
    <linux/mutex.h> to the internal header
  locking/ww_mutex: Move the ww_mutex definitions from <linux/mutex.h>
    into <linux/ww_mutex.h>
  locking/mutex: Make mutex::wait_lock raw
  locking/ww_mutex: Simplify lockdep annotations
  locking/ww_mutex: Gather mutex_waiter initialization
  locking/ww_mutex: Split up ww_mutex_unlock()
  locking/ww_mutex: Split out the W/W implementation logic into
    kernel/locking/ww_mutex.h
  locking/ww_mutex: Remove the __sched annotation from ww_mutex APIs
  locking/ww_mutex: Abstract out the waiter iteration
  locking/ww_mutex: Abstract out waiter enqueueing
  locking/ww_mutex: Abstract out mutex accessors
  locking/ww_mutex: Abstract out mutex types
  locking/ww_mutex: Abstract out internal lock accesses
  locking/ww_mutex: Implement rt_mutex accessors
  locking/ww_mutex: Add RT priority to W/W order
  locking/ww_mutex: Add rt_mutex based lock type and accessors
  locking/rtmutex: Extend the rtmutex core to support ww_mutex
  locking/ww_mutex: Implement rtmutex based ww_mutex API functions
  locking/rtmutex: Add mutex variant for RT
  lib/test_lockup: Adapt to changed variables
  futex: Validate waiter correctly in futex_proxy_trylock_atomic()
  futex: Clean up stale comments
  futex: Clarify futex_requeue() PI handling
  futex: Remove bogus condition for requeue PI
  futex: Correct the number of requeued waiters for PI
  futex: Restructure futex_requeue()
  futex: Clarify comment in futex_requeue()
  futex: Reorder sanity checks in futex_requeue()
  futex: Simplify handle_early_requeue_pi_wakeup()
  futex: Prevent requeue_pi() lock nesting issue on RT
  locking/rtmutex: Prevent lockdep false positive with PI futexes
  preempt: Adjust PREEMPT_LOCK_OFFSET for RT
  locking/rtmutex: Implement equal priority lock stealing
  locking/rtmutex: Add adaptive spinwait mechanism
  locking/spinlock/rt: Prepare for RT local_lock
  locking/local_lock: Add PREEMPT_RT support
  locking/rtmutex: Dont dereference waiter lockless
  locking/rtmutex: Dequeue waiter on ww_mutex deadlock
  locking/rtmutex: Prevent spurious EDEADLK return caused by ww_mutexes
  locking/rtmutex: Return success on deadlock for ww_mutex waiters
  locking/rwsem: Add missing __init_rwsem() for PREEMPT_RT
  locking/rtmutex: Fix ww_mutex deadlock check
  locking/rwbase: Properly match set_and_save_state() to restore_state()
  locking/rwbase: Extract __rwbase_write_trylock()
  locking/rwbase: Take care of ordering guarantee for fastpath reader
  futex: Return error code instead of assigning it without effect
  locking/ww_mutex: Initialize waiter.ww_ctx properly
  futex: Prevent inconsistent state and exit race
  futex: Remove unused variable 'vpid' in futex_proxy_trylock_atomic()

 include/linux/debug_locks.h         |    3 +-
 include/linux/local_lock_internal.h |   86 +-
 include/linux/mutex.h               |   92 ++-
 include/linux/preempt.h             |    4 +
 include/linux/rbtree.h              |   31 +-
 include/linux/rbtree_types.h        |   34 +
 include/linux/rtmutex.h             |   63 +-
 include/linux/rwbase_rt.h           |   39 +
 include/linux/rwlock_rt.h           |  140 ++++
 include/linux/rwlock_types.h        |   53 +-
 include/linux/rwsem.h               |   70 +-
 include/linux/sched.h               |  119 ++-
 include/linux/sched/wake_q.h        |    7 +-
 include/linux/spinlock.h            |   15 +-
 include/linux/spinlock_api_smp.h    |    3 +
 include/linux/spinlock_rt.h         |  159 ++++
 include/linux/spinlock_types.h      |   89 +-
 include/linux/spinlock_types_raw.h  |   73 ++
 include/linux/ww_mutex.h            |   50 +-
 kernel/Kconfig.locks                |    2 +-
 kernel/futex.c                      |  609 +++++++++-----
 kernel/locking/Makefile             |    3 +-
 kernel/locking/mutex-debug.c        |    5 +-
 kernel/locking/mutex-debug.h        |   29 -
 kernel/locking/mutex.c              |  541 ++----------
 kernel/locking/mutex.h              |   48 +-
 kernel/locking/rtmutex.c            | 1192 +++++++++++++--------------
 kernel/locking/rtmutex_api.c        |  590 +++++++++++++
 kernel/locking/rtmutex_common.h     |  135 ++-
 kernel/locking/rwbase_rt.c          |  288 +++++++
 kernel/locking/rwsem.c              |  111 +++
 kernel/locking/spinlock.c           |    7 +
 kernel/locking/spinlock_debug.c     |    5 +
 kernel/locking/spinlock_rt.c        |  263 ++++++
 kernel/locking/ww_mutex.h           |  569 +++++++++++++
 kernel/locking/ww_rt_mutex.c        |   76 ++
 kernel/rcu/tree_plugin.h            |    6 +-
 kernel/sched/core.c                 |  109 ++-
 lib/Kconfig.debug                   |   11 +-
 lib/test_lockup.c                   |    8 +-
 40 files changed, 4100 insertions(+), 1637 deletions(-)
 create mode 100644 include/linux/rbtree_types.h
 create mode 100644 include/linux/rwbase_rt.h
 create mode 100644 include/linux/rwlock_rt.h
 create mode 100644 include/linux/spinlock_rt.h
 create mode 100644 include/linux/spinlock_types_raw.h
 delete mode 100644 kernel/locking/mutex-debug.h
 create mode 100644 kernel/locking/rtmutex_api.c
 create mode 100644 kernel/locking/rwbase_rt.c
 create mode 100644 kernel/locking/spinlock_rt.c
 create mode 100644 kernel/locking/ww_mutex.h
 create mode 100644 kernel/locking/ww_rt_mutex.c
Edited by Waiman Long

Merge request reports