1. 18 Oct, 2018 1 commit
  2. 30 Aug, 2018 2 commits
    • Paul E. McKenney's avatar
      rcu: Define RCU-bh update API in terms of RCU · 65cfe358
      Paul E. McKenney authored
      Now that the main RCU API knows about softirq disabling and softirq's
      quiescent states, the RCU-bh update code can be dispensed with.
      This commit therefore removes the RCU-bh update-side implementation and
      defines RCU-bh's update-side API in terms of that of either RCU-preempt or
      RCU-sched, depending on the setting of the CONFIG_PREEMPT Kconfig option.
      
      In kernels built with CONFIG_RCU_NOCB_CPU=y this has the knock-on effect
      of reducing by one the number of rcuo kthreads per CPU.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      65cfe358
    • Paul E. McKenney's avatar
      rcu: Apply RCU-bh QSes to RCU-sched and RCU-preempt when safe · d28139c4
      Paul E. McKenney authored
      One necessary step towards consolidating the three flavors of RCU is to
      make sure that the resulting consolidated "one flavor to rule them all"
      correctly handles networking denial-of-service attacks.  One thing that
      allows RCU-bh to do so is that __do_softirq() invokes rcu_bh_qs() every
      so often, and so something similar has to happen for consolidated RCU.
      
      This must be done carefully.  For example, if a preemption-disabled
      region of code takes an interrupt which does softirq processing before
      returning, consolidated RCU must ignore the resulting rcu_bh_qs()
      invocations -- preemption is still disabled, and that means an RCU
      reader for the consolidated flavor.
      
      This commit therefore creates a new rcu_softirq_qs() that is called only
      from the ksoftirqd task, thus avoiding the interrupted-a-preempted-region
      problem.  This new rcu_softirq_qs() function invokes rcu_sched_qs(),
      rcu_preempt_qs(), and rcu_preempt_deferred_qs().  The latter call handles
      any deferred quiescent states.
      
      Note that __do_softirq() still invokes rcu_bh_qs().  It will continue to
      do so until a later stage of cleanup when the RCU-bh flavor is removed.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      [ paulmck: Fix !SMP issue located by kbuild test robot. ]
      d28139c4
  3. 03 Aug, 2018 1 commit
    • Frederic Weisbecker's avatar
      nohz: Fix missing tick reprogram when interrupting an inline softirq · 0a0e0829
      Frederic Weisbecker authored
      The full nohz tick is reprogrammed in irq_exit() only if the exit is not in
      a nesting interrupt. This stands as an optimization: whether a hardirq or a
      softirq is interrupted, the tick is going to be reprogrammed when necessary
      at the end of the inner interrupt, with even potential new updates on the
      timer queue.
      
      When soft interrupts are interrupted, it's assumed that they are executing
      on the tail of an interrupt return. In that case tick_nohz_irq_exit() is
      called after softirq processing to take care of the tick reprogramming.
      
      But the assumption is wrong: softirqs can be processed inline as well, ie:
      outside of an interrupt, like in a call to local_bh_enable() or from
      ksoftirqd.
      
      Inline softirqs don't reprogram the tick once they are done, as opposed to
      interrupt tail softirq processing. So if a tick interrupts an inline
      softirq processing, the next timer will neither be reprogrammed from the
      interrupting tick's irq_exit() nor after the interrupted softirq
      processing. This situation may leave the tick unprogrammed while timers are
      armed.
      
      To fix this, simply keep reprogramming the tick even if a softirq has been
      interrupted. That can be optimized further, but for now correctness is more
      important.
      
      Note that new timers enqueued in nohz_full mode after a softirq gets
      interrupted will still be handled just fine through self-IPIs triggered by
      the timer code.
      Reported-by: default avatarAnna-Maria Gleixner <anna-maria@linutronix.de>
      Signed-off-by: default avatarFrederic Weisbecker <frederic@kernel.org>
      Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Tested-by: default avatarAnna-Maria Gleixner <anna-maria@linutronix.de>
      Cc: stable@vger.kernel.org # 4.14+
      Link: https://lkml.kernel.org/r/1533303094-15855-1-git-send-email-frederic@kernel.org
      0a0e0829
  4. 17 Jul, 2018 1 commit
    • Linus Torvalds's avatar
      Mark HI and TASKLET softirq synchronous · 3c53776e
      Linus Torvalds authored
      Way back in 4.9, we committed 4cd13c21 ("softirq: Let ksoftirqd do
      its job"), and ever since we've had small nagging issues with it.  For
      example, we've had:
      
        1ff68820 ("watchdog: core: make sure the watchdog_worker is not deferred")
        8d5755b3 ("watchdog: softdog: fire watchdog even if softirqs do not get to run")
        217f6974 ("net: busy-poll: allow preemption in sk_busy_loop()")
      
      all of which worked around some of the effects of that commit.
      
      The DVB people have also complained that the commit causes excessive USB
      URB latencies, which seems to be due to the USB code using tasklets to
      schedule USB traffic.  This seems to be an issue mainly when already
      living on the edge, but waiting for ksoftirqd to handle it really does
      seem to cause excessive latencies.
      
      Now Hanna Hawa reports that this issue isn't just limited to USB URB and
      DVB, but also causes timeout problems for the Marvell SoC team:
      
       "I'm facing kernel panic issue while running raid 5 on sata disks
        connected to Macchiatobin (Marvell community board with Armada-8040
        SoC with 4 ARMv8 cores of CA72) Raid 5 built with Marvell DMA engine
        and async_tx mechanism (ASYNC_TX_DMA [=y]); the DMA driver (mv_xor_v2)
        uses a tasklet to clean the done descriptors from the queue"
      
      The latency problem causes a panic:
      
        mv_xor_v2 f0400000.xor: dma_sync_wait: timeout!
        Kernel panic - not syncing: async_tx_quiesce: DMA error waiting for transaction
      
      We've discussed simply just reverting the original commit entirely, and
      also much more involved solutions (with per-softirq threads etc).  This
      patch is intentionally stupid and fairly limited, because the issue
      still remains, and the other solutions either got sidetracked or had
      other issues.
      
      We should probably also consider the timer softirqs to be synchronous
      and not be delayed to ksoftirqd (since they were the issue with the
      earlier watchdog problems), but that should be done as a separate patch.
      This does only the tasklet cases.
      Reported-and-tested-by: default avatarHanna Hawa <hannah@marvell.com>
      Reported-and-tested-by: default avatarJosef Griebichler <griebichler.josef@gmx.at>
      Reported-by: default avatarMauro Carvalho Chehab <mchehab@s-opensource.com>
      Cc: Alan Stern <stern@rowland.harvard.edu>
      Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
      Cc: Eric Dumazet <edumazet@google.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      3c53776e
  5. 21 Jun, 2018 1 commit
    • Joel Fernandes (Google)'s avatar
      softirq: Reorder trace_softirqs_on to prevent lockdep splat · 1a63dcd8
      Joel Fernandes (Google) authored
      I'm able to reproduce a lockdep splat with config options:
      CONFIG_PROVE_LOCKING=y,
      CONFIG_DEBUG_LOCK_ALLOC=y and
      CONFIG_PREEMPTIRQ_EVENTS=y
      
      $ echo 1 > /d/tracing/events/preemptirq/preempt_enable/enable
      
      [   26.112609] DEBUG_LOCKS_WARN_ON(current->softirqs_enabled)
      [   26.112636] WARNING: CPU: 0 PID: 118 at kernel/locking/lockdep.c:3854
      [...]
      [   26.144229] Call Trace:
      [   26.144926]  <IRQ>
      [   26.145506]  lock_acquire+0x55/0x1b0
      [   26.146499]  ? __do_softirq+0x46f/0x4d9
      [   26.147571]  ? __do_softirq+0x46f/0x4d9
      [   26.148646]  trace_preempt_on+0x8f/0x240
      [   26.149744]  ? trace_preempt_on+0x4d/0x240
      [   26.150862]  ? __do_softirq+0x46f/0x4d9
      [   26.151930]  preempt_count_sub+0x18a/0x1a0
      [   26.152985]  __do_softirq+0x46f/0x4d9
      [   26.153937]  irq_exit+0x68/0xe0
      [   26.154755]  smp_apic_timer_interrupt+0x271/0x280
      [   26.156056]  apic_timer_interrupt+0xf/0x20
      [   26.157105]  </IRQ>
      
      The issue was this:
      
      preempt_count = 1 << SOFTIRQ_SHIFT
      
      	__local_bh_enable(cnt = 1 << SOFTIRQ_SHIFT) {
      		if (softirq_count() == (cnt && SOFTIRQ_MASK)) {
      			trace_softirqs_on() {
      				current->softirqs_enabled = 1;
      			}
      		}
      		preempt_count_sub(cnt) {
      			trace_preempt_on() {
      				tracepoint() {
      					rcu_read_lock_sched() {
      						// jumps into lockdep
      
      Where preempt_count still has softirqs disabled, but
      current->softirqs_enabled is true, and we get a splat.
      
      Link: http://lkml.kernel.org/r/20180607201143.247775-1-joel@joelfernandes.org
      
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
      Cc: Tom Zanussi <tom.zanussi@linux.intel.com>
      Cc: Namhyung Kim <namhyung@kernel.org>
      Cc: Thomas Glexiner <tglx@linutronix.de>
      Cc: Boqun Feng <boqun.feng@gmail.com>
      Cc: Paul McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Masami Hiramatsu <mhiramat@kernel.org>
      Cc: Todd Kjos <tkjos@google.com>
      Cc: Erick Reyes <erickreyes@google.com>
      Cc: Julia Cartwright <julia@ni.com>
      Cc: Byungchul Park <byungchul.park@lge.com>
      Cc: stable@vger.kernel.org
      Reviewed-by: default avatarSteven Rostedt (VMware) <rostedt@goodmis.org>
      Fixes: d5915816 ("tracing: Add support for preempt and irq enable/disable events")
      Signed-off-by: default avatarJoel Fernandes (Google) <joel@joelfernandes.org>
      Signed-off-by: default avatarSteven Rostedt (VMware) <rostedt@goodmis.org>
      1a63dcd8
  6. 15 May, 2018 1 commit
  7. 14 May, 2018 1 commit
  8. 09 Mar, 2018 2 commits
  9. 04 Dec, 2017 1 commit
  10. 16 Nov, 2017 1 commit
  11. 08 Nov, 2017 1 commit
  12. 11 Apr, 2017 1 commit
    • NeilBrown's avatar
      sched/core: Remove 'task' parameter and rename tsk_restore_flags() to current_restore_flags() · 717a94b5
      NeilBrown authored
      It is not safe for one thread to modify the ->flags
      of another thread as there is no locking that can protect
      the update.
      
      So tsk_restore_flags(), which takes a task pointer and modifies
      the flags, is an invitation to do the wrong thing.
      
      All current users pass "current" as the task, so no developers have
      accepted that invitation.  It would be best to ensure it remains
      that way.
      
      So rename tsk_restore_flags() to current_restore_flags() and don't
      pass in a task_struct pointer.  Always operate on current->flags.
      Signed-off-by: default avatarNeilBrown <neilb@suse.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Michal Hocko <mhocko@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: linux-kernel@vger.kernel.org
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      717a94b5
  13. 21 Oct, 2016 1 commit
  14. 10 Oct, 2016 1 commit
    • Emese Revfy's avatar
      latent_entropy: Mark functions with __latent_entropy · 0766f788
      Emese Revfy authored
      The __latent_entropy gcc attribute can be used only on functions and
      variables.  If it is on a function then the plugin will instrument it for
      gathering control-flow entropy. If the attribute is on a variable then
      the plugin will initialize it with random contents.  The variable must
      be an integer, an integer array type or a structure with integer fields.
      
      These specific functions have been selected because they are init
      functions (to help gather boot-time entropy), are called at unpredictable
      times, or they have variable loops, each of which provide some level of
      latent entropy.
      Signed-off-by: default avatarEmese Revfy <re.emese@gmail.com>
      [kees: expanded commit message]
      Signed-off-by: default avatarKees Cook <keescook@chromium.org>
      0766f788
  15. 30 Sep, 2016 1 commit
    • Eric Dumazet's avatar
      softirq: Let ksoftirqd do its job · 4cd13c21
      Eric Dumazet authored
      A while back, Paolo and Hannes sent an RFC patch adding threaded-able
      napi poll loop support : (https://patchwork.ozlabs.org/patch/620657/)
      
      The problem seems to be that softirqs are very aggressive and are often
      handled by the current process, even if we are under stress and that
      ksoftirqd was scheduled, so that innocent threads would have more chance
      to make progress.
      
      This patch makes sure that if ksoftirq is running, we let it
      perform the softirq work.
      
      Jonathan Corbet summarized the issue in https://lwn.net/Articles/687617/
      
      Tested:
      
       - NIC receiving traffic handled by CPU 0
       - UDP receiver running on CPU 0, using a single UDP socket.
       - Incoming flood of UDP packets targeting the UDP socket.
      
      Before the patch, the UDP receiver could almost never get CPU cycles and
      could only receive ~2,000 packets per second.
      
      After the patch, CPU cycles are split 50/50 between user application and
      ksoftirqd/0, and we can effectively read ~900,000 packets per second,
      a huge improvement in DOS situation. (Note that more packets are now
      dropped by the NIC itself, since the BH handlers get less CPU cycles to
      drain RX ring buffer)
      
      Since the load runs in well identified threads context, an admin can
      more easily tune process scheduling parameters if needed.
      Reported-by: default avatarPaolo Abeni <pabeni@redhat.com>
      Reported-by: default avatarHannes Frederic Sowa <hannes@stressinduktion.org>
      Signed-off-by: default avatarEric Dumazet <edumazet@google.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Cc: David Miller <davem@davemloft.net>
      Cc: Hannes Frederic Sowa <hannes@redhat.com>
      Cc: Jesper Dangaard Brouer <jbrouer@redhat.com>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Link: http://lkml.kernel.org/r/1472665349.14381.356.camel@edumazet-glaptop3.roam.corp.google.comSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
      4cd13c21
  16. 06 Sep, 2016 1 commit
  17. 25 Mar, 2016 1 commit
  18. 29 Feb, 2016 1 commit
  19. 14 Jan, 2015 3 commits
    • Paul E. McKenney's avatar
      ksoftirqd: Use new cond_resched_rcu_qs() function · 60479676
      Paul E. McKenney authored
      Simplify run_ksoftirqd() by using the new cond_resched_rcu_qs() function
      that conditionally reschedules, but unconditionally supplies an RCU
      quiescent state.  This commit is separate from the previous commit by
      Calvin Owens because Calvin's approach can be backported, while this
      commit cannot be.  The reason that this commit cannot be backported is
      that cond_resched_rcu_qs() does not always provide the needed quiescent
      state in earlier kernels.
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      60479676
    • Calvin Owens's avatar
      ksoftirqd: Enable IRQs and call cond_resched() before poking RCU · 28423ad2
      Calvin Owens authored
      While debugging an issue with excessive softirq usage, I encountered the
      following note in commit 3e339b5d ("softirq: Use hotplug thread
      infrastructure"):
      
          [ paulmck: Call rcu_note_context_switch() with interrupts enabled. ]
      
      ...but despite this note, the patch still calls RCU with IRQs disabled.
      
      This seemingly innocuous change caused a significant regression in softirq
      CPU usage on the sending side of a large TCP transfer (~1 GB/s): when
      introducing 0.01% packet loss, the softirq usage would jump to around 25%,
      spiking as high as 50%. Before the change, the usage would never exceed 5%.
      
      Moving the call to rcu_note_context_switch() after the cond_sched() call,
      as it was originally before the hotplug patch, completely eliminated this
      problem.
      Signed-off-by: default avatarCalvin Owens <calvinowens@fb.com>
      Cc: stable@vger.kernel.org
      Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      28423ad2
    • Heiko Carstens's avatar
      softirq/preempt: Add missing current->preempt_disable_ip update · 0f1ba9a2
      Heiko Carstens authored
      While debugging some "sleeping function called from invalid context" bug I
      realized that the debugging message "Preemption disabled at:" pointed to
      an incorrect function.
      
      In particular if the last function/action that disabled preemption was
      spin_lock_bh() then current->preempt_disable_ip won't be updated.
      
      The reason for this is that __local_bh_disable_ip() will increase
      preempt_count manually instead of calling preempt_count_add(), which
      would handle the update correctly.
      
      It look like the manual handling was done to work around some lockdep issue.
      
      So add the missing update of current->preempt_disable_ip to
      __local_bh_disable_ip() as well.
      Signed-off-by: default avatarHeiko Carstens <heiko.carstens@de.ibm.com>
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Link: http://lkml.kernel.org/r/20150107090441.GC4365@osirisSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
      0f1ba9a2
  20. 04 Nov, 2014 1 commit
  21. 07 Sep, 2014 1 commit
  22. 26 Aug, 2014 1 commit
  23. 05 May, 2014 1 commit
  24. 29 Apr, 2014 1 commit
  25. 28 Apr, 2014 1 commit
    • Thomas Gleixner's avatar
      genirq: x86: Ensure that dynamic irq allocation does not conflict · 62a08ae2
      Thomas Gleixner authored
      On x86 the allocation of irq descriptors may allocate interrupts which
      are in the range of the GSI interrupts. That's wrong as those
      interrupts are hardwired and we don't have the irq domain translation
      like PPC. So one of these interrupts can be hooked up later to one of
      the devices which are hard wired to it and the io_apic init code for
      that particular interrupt line happily reuses that descriptor with a
      completely different configuration so hell breaks lose.
      
      Inside x86 we allocate dynamic interrupts from above nr_gsi_irqs,
      except for a few usage sites which have not yet blown up in our face
      for whatever reason. But for drivers which need an irq range, like the
      GPIO drivers, we have no limit in place and we don't want to expose
      such a detail to a driver.
      
      To cure this introduce a function which an architecture can implement
      to impose a lower bound on the dynamic interrupt allocations.
      
      Implement it for x86 and set the lower bound to nr_gsi_irqs, which is
      the end of the hardwired interrupt space, so all dynamic allocations
      happen above.
      
      That not only allows the GPIO driver to work sanely, it also protects
      the bogus callsites of create_irq_nr() in hpet, uv, irq_remapping and
      htirq code. They need to be cleaned up as well, but that's a separate
      issue.
      Reported-by: default avatarJin Yao <yao.jin@linux.intel.com>
      Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Tested-by: default avatarMika Westerberg <mika.westerberg@linux.intel.com>
      Cc: Mathias Nyman <mathias.nyman@linux.intel.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Grant Likely <grant.likely@linaro.org>
      Cc: H. Peter Anvin <hpa@linux.intel.com>
      Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
      Cc: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
      Cc: Krogerus Heikki <heikki.krogerus@intel.com>
      Cc: Linus Walleij <linus.walleij@linaro.org>
      Link: http://lkml.kernel.org/r/alpine.DEB.2.02.1404241617360.28206@ionos.tec.linutronix.deSigned-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
      62a08ae2
  26. 19 Mar, 2014 1 commit
  27. 28 Jan, 2014 3 commits
  28. 15 Jan, 2014 1 commit
    • Frederic Weisbecker's avatar
      tick: Rename tick_check_idle() to tick_irq_enter() · 5acac1be
      Frederic Weisbecker authored
      This makes the code more symetric against the existing tick functions
      called on irq exit: tick_irq_exit() and tick_nohz_irq_exit().
      
      These function are also symetric as they mirror each other's action:
      we start to account idle time on irq exit and we stop this accounting
      on irq entry. Also the tick is stopped on irq exit and timekeeping
      catches up with the tickless time elapsed until we reach irq entry.
      
      This rename was suggested by Peter Zijlstra a long while ago but it
      got forgotten in the mass.
      
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Alex Shi <alex.shi@linaro.org>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: John Stultz <john.stultz@linaro.org>
      Cc: Kevin Hilman <khilman@linaro.org>
      Link: http://lkml.kernel.org/r/1387320692-28460-2-git-send-email-fweisbec@gmail.comSigned-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      5acac1be
  29. 13 Jan, 2014 2 commits
    • Peter Zijlstra's avatar
      sched/preempt, locking: Rework local_bh_{dis,en}able() · 0bd3a173
      Peter Zijlstra authored
      Currently local_bh_disable() is out-of-line for no apparent reason.
      So inline it to save a few cycles on call/return nonsense, the
      function body is a single add on x86 (a few loads and store extra on
      load/store archs).
      
      Also expose two new local_bh functions:
      
        __local_bh_{dis,en}able_ip(unsigned long ip, unsigned int cnt);
      
      Which implement the actual local_bh_{dis,en}able() behaviour.
      
      The next patch uses the exposed @Cnt argument to optimize bh lock
      functions.
      
      With build fixes from Jacob Pan.
      
      Cc: rjw@rjwysocki.net
      Cc: rui.zhang@intel.com
      Cc: jacob.jun.pan@linux.intel.com
      Cc: Mike Galbraith <bitbucket@online.de>
      Cc: hpa@zytor.com
      Cc: Arjan van de Ven <arjan@linux.intel.com>
      Cc: lenb@kernel.org
      Reviewed-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Link: http://lkml.kernel.org/r/20131119151338.GF3694@twins.programming.kicks-ass.netSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
      0bd3a173
    • Peter Zijlstra's avatar
      locking: Optimize lock_bh functions · 9ea4c380
      Peter Zijlstra authored
      Currently all _bh_ lock functions do two preempt_count operations:
      
        local_bh_disable();
        preempt_disable();
      
      and for the unlock:
      
        preempt_enable_no_resched();
        local_bh_enable();
      
      Since its a waste of perfectly good cycles to modify the same variable
      twice when you can do it in one go; use the new
      __local_bh_{dis,en}able_ip() functions that allow us to provide a
      preempt_count value to add/sub.
      
      So define SOFTIRQ_LOCK_OFFSET as the offset a _bh_ lock needs to
      add/sub to be done in one go.
      
      As a bonus it gets rid of the preempt_enable_no_resched() usage.
      
      This reduces a 1000 loops of:
      
        spin_lock_bh(&bh_lock);
        spin_unlock_bh(&bh_lock);
      
      from 53596 cycles to 51995 cycles. I didn't do enough measurements to
      say for absolute sure that the result is significant but the the few
      runs I did for each suggest it is so.
      Reviewed-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Cc: jacob.jun.pan@linux.intel.com
      Cc: Mike Galbraith <bitbucket@online.de>
      Cc: hpa@zytor.com
      Cc: Arjan van de Ven <arjan@linux.intel.com>
      Cc: lenb@kernel.org
      Cc: rjw@rjwysocki.net
      Cc: rui.zhang@intel.com
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Link: http://lkml.kernel.org/r/20131119151338.GF3694@twins.programming.kicks-ass.netSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
      9ea4c380
  30. 02 Dec, 2013 1 commit
    • Frederic Weisbecker's avatar
      nohz: Convert a few places to use local per cpu accesses · e8fcaa5c
      Frederic Weisbecker authored
      A few functions use remote per CPU access APIs when they
      deal with local values.
      
      Just do the right conversion to improve performance, code
      readability and debug checks.
      
      While at it, lets extend some of these function names with *_this_cpu()
      suffix in order to display their purpose more clearly.
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Oleg Nesterov <oleg@redhat.com>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      e8fcaa5c
  31. 27 Nov, 2013 1 commit
  32. 19 Nov, 2013 1 commit
    • Peter Zijlstra's avatar
      lockdep: Correctly annotate hardirq context in irq_exit() · f1a83e65
      Peter Zijlstra authored
      There was a reported deadlock on -rt which lockdep didn't report.
      
      It turns out that in irq_exit() we tell lockdep that the hardirq
      context ends and then do all kinds of locking afterwards.
      
      To fix it, move trace_hardirq_exit() to the very end of irq_exit(), this
      ensures all locking in tick_irq_exit() and rcu_irq_exit() are properly
      recorded as happening from hardirq context.
      
      This however leads to the 'fun' little problem of running softirqs
      while in hardirq context. To cure this make the softirq code a little
      more complex (in the CONFIG_TRACE_IRQFLAGS case).
      
      Due to stack swizzling arch dependent trickery we cannot pass an
      argument to __do_softirq() to tell it if it was done from hardirq
      context or not; so use a side-band argument.
      
      When we do __do_softirq() from hardirq context, 'atomically' flip to
      softirq context and back, so that no locking goes without being in
      either hard- or soft-irq context.
      
      I didn't find any new problems in mainline using this patch, but it
      did show the -rt problem.
      Reported-by: default avatarSebastian Andrzej Siewior <bigeasy@linutronix.de>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Link: http://lkml.kernel.org/n/tip-dgwc5cdksbn0jk09vbmcc9sa@git.kernel.orgSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
      f1a83e65
  33. 15 Nov, 2013 1 commit