1. 08 Jan, 2018 1 commit
  2. 12 Nov, 2017 1 commit
  3. 08 Nov, 2017 1 commit
  4. 31 Oct, 2017 1 commit
  5. 09 Oct, 2017 1 commit
  6. 23 Nov, 2015 1 commit
    • Peter Zijlstra's avatar
      treewide: Remove old email address · 90eec103
      Peter Zijlstra authored
      There were still a number of references to my old Red Hat email
      address in the kernel source. Remove these while keeping the
      Red Hat copyright notices intact.
      Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
      Cc: Jiri Olsa <jolsa@redhat.com>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Mike Galbraith <efault@gmx.de>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Stephane Eranian <eranian@google.com>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Vince Weaver <vincent.weaver@maine.edu>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      90eec103
  7. 29 Oct, 2014 1 commit
  8. 13 Sep, 2014 1 commit
    • Frederic Weisbecker's avatar
      irq_work: Force raised irq work to run on irq work interrupt · 76a33061
      Frederic Weisbecker authored
      The nohz full kick, which restarts the tick when any resource depend
      on it, can't be executed anywhere given the operation it does on timers.
      If it is called from the scheduler or timers code, chances are that
      we run into a deadlock.
      
      This is why we run the nohz full kick from an irq work. That way we make
      sure that the kick runs on a virgin context.
      
      However if that's the case when irq work runs in its own dedicated
      self-ipi, things are different for the big bunch of archs that don't
      support the self triggered way. In order to support them, irq works are
      also handled by the timer interrupt as fallback.
      
      Now when irq works run on the timer interrupt, the context isn't blank.
      More precisely, they can run in the context of the hrtimer that runs the
      tick. But the nohz kick cancels and restarts this hrtimer and cancelling
      an hrtimer from itself isn't allowed. This is why we run in an endless
      loop:
      
      	Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2
      	CPU: 2 PID: 7538 Comm: kworker/u8:8 Not tainted 3.16.0+ #34
      	Workqueue: btrfs-endio-write normal_work_helper [btrfs]
      	 ffff880244c06c88 000000001b486fe1 ffff880244c06bf0 ffffffff8a7f1e37
      	 ffffffff8ac52a18 ffff880244c06c78 ffffffff8a7ef928 0000000000000010
      	 ffff880244c06c88 ffff880244c06c20 000000001b486fe1 0000000000000000
      	Call Trace:
      	 <NMI[<ffffffff8a7f1e37>] dump_stack+0x4e/0x7a
      	 [<ffffffff8a7ef928>] panic+0xd4/0x207
      	 [<ffffffff8a1450e8>] watchdog_overflow_callback+0x118/0x120
      	 [<ffffffff8a186b0e>] __perf_event_overflow+0xae/0x350
      	 [<ffffffff8a184f80>] ? perf_event_task_disable+0xa0/0xa0
      	 [<ffffffff8a01a4cf>] ? x86_perf_event_set_period+0xbf/0x150
      	 [<ffffffff8a187934>] perf_event_overflow+0x14/0x20
      	 [<ffffffff8a020386>] intel_pmu_handle_irq+0x206/0x410
      	 [<ffffffff8a01937b>] perf_event_nmi_handler+0x2b/0x50
      	 [<ffffffff8a007b72>] nmi_handle+0xd2/0x390
      	 [<ffffffff8a007aa5>] ? nmi_handle+0x5/0x390
      	 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0
      	 [<ffffffff8a008062>] default_do_nmi+0x72/0x1c0
      	 [<ffffffff8a008268>] do_nmi+0xb8/0x100
      	 [<ffffffff8a7ff66a>] end_repeat_nmi+0x1e/0x2e
      	 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0
      	 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0
      	 [<ffffffff8a0cb7f8>] ? match_held_lock+0x8/0x1b0
      	 <<EOE><IRQ[<ffffffff8a0ccd2f>] lock_acquired+0xaf/0x450
      	 [<ffffffff8a0f74c5>] ? lock_hrtimer_base.isra.20+0x25/0x50
      	 [<ffffffff8a7fc678>] _raw_spin_lock_irqsave+0x78/0x90
      	 [<ffffffff8a0f74c5>] ? lock_hrtimer_base.isra.20+0x25/0x50
      	 [<ffffffff8a0f74c5>] lock_hrtimer_base.isra.20+0x25/0x50
      	 [<ffffffff8a0f7723>] hrtimer_try_to_cancel+0x33/0x1e0
      	 [<ffffffff8a0f78ea>] hrtimer_cancel+0x1a/0x30
      	 [<ffffffff8a109237>] tick_nohz_restart+0x17/0x90
      	 [<ffffffff8a10a213>] __tick_nohz_full_check+0xc3/0x100
      	 [<ffffffff8a10a25e>] nohz_full_kick_work_func+0xe/0x10
      	 [<ffffffff8a17c884>] irq_work_run_list+0x44/0x70
      	 [<ffffffff8a17c8da>] irq_work_run+0x2a/0x50
      	 [<ffffffff8a0f700b>] update_process_times+0x5b/0x70
      	 [<ffffffff8a109005>] tick_sched_handle.isra.21+0x25/0x60
      	 [<ffffffff8a109b81>] tick_sched_timer+0x41/0x60
      	 [<ffffffff8a0f7aa2>] __run_hrtimer+0x72/0x470
      	 [<ffffffff8a109b40>] ? tick_sched_do_timer+0xb0/0xb0
      	 [<ffffffff8a0f8707>] hrtimer_interrupt+0x117/0x270
      	 [<ffffffff8a034357>] local_apic_timer_interrupt+0x37/0x60
      	 [<ffffffff8a80010f>] smp_apic_timer_interrupt+0x3f/0x50
      	 [<ffffffff8a7fe52f>] apic_timer_interrupt+0x6f/0x80
      
      To fix this we force non-lazy irq works to run on irq work self-IPIs
      when available. That ability of the arch to trigger irq work self IPIs
      is available with arch_irq_work_has_interrupt().
      Reported-by: default avatarCatalin Iacob <iacobcatalin@gmail.com>
      Reported-by: default avatarDave Jones <davej@redhat.com>
      Acked-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      76a33061
  9. 26 Aug, 2014 1 commit
  10. 05 Jul, 2014 1 commit
  11. 16 Jun, 2014 2 commits
    • Frederic Weisbecker's avatar
      irq_work: Implement remote queueing · 47885016
      Frederic Weisbecker authored
      irq work currently only supports local callbacks. However its code
      is mostly ready to run remote callbacks and we have some potential user.
      
      The full nohz subsystem currently open codes its own remote irq work
      on top of the scheduler ipi when it wants a CPU to reevaluate its next
      tick. However this ad hoc solution bloats the scheduler IPI.
      
      Lets just extend the irq work subsystem to support remote queuing on top
      of the generic SMP IPI to handle this kind of user. This shouldn't add
      noticeable overhead.
      Suggested-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Acked-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Eric Dumazet <eric.dumazet@gmail.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Kevin Hilman <khilman@linaro.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Viresh Kumar <viresh.kumar@linaro.org>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      47885016
    • Frederic Weisbecker's avatar
      irq_work: Split raised and lazy lists · b93e0b8f
      Frederic Weisbecker authored
      An irq work can be handled from two places: from the tick if the work
      carries the "lazy" flag and the tick is periodic, or from a self IPI.
      
      We merge all these works in a single list and we use some per cpu latch
      to avoid raising a self-IPI when one is already pending.
      
      Now we could do away with this ugly latch if only the list was only made of
      non-lazy works. Just enqueueing a work on the empty list would be enough
      to know if we need to raise an IPI or not.
      
      Also we are going to implement remote irq work queuing. Then the per CPU
      latch will need to become atomic in the global scope. That's too bad
      because, here as well, just enqueueing a work on an empty list of
      non-lazy works would be enough to know if we need to raise an IPI or not.
      
      So lets take a way out of this: split the works in two distinct lists,
      one for the works that can be handled by the next tick and another
      one for those handled by the IPI. Just checking if the latter is empty
      when we queue a new work is enough to know if we need to raise an IPI.
      Suggested-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Acked-by: default avatarPeter Zijlstra <peterz@infradead.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Kevin Hilman <khilman@linaro.org>
      Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Viresh Kumar <viresh.kumar@linaro.org>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      b93e0b8f
  12. 21 Feb, 2014 1 commit
  13. 04 Feb, 2013 1 commit
  14. 18 Nov, 2012 1 commit
    • Frederic Weisbecker's avatar
      irq_work: Make self-IPIs optable · bc6679ae
      Frederic Weisbecker authored
      On irq work initialization, let the user choose to define it
      as "lazy" or not. "Lazy" means that we don't want to send
      an IPI (provided the arch can anyway) when we enqueue this
      work but we rather prefer to wait for the next timer tick
      to execute our work if possible.
      
      This is going to be a benefit for non-urgent enqueuers
      (like printk in the future) that may prefer not to raise
      an IPI storm in case of frequent enqueuing on short periods
      of time.
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      Acked-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
      bc6679ae
  15. 17 Nov, 2012 3 commits
    • Steven Rostedt's avatar
      irq_work: Warn if there's still work on cpu_down · 8aa2acce
      Steven Rostedt authored
      If we are in nohz and there's still irq_work to be done when the idle
      task is about to go offline, give a nasty warning. Everything should
      have been flushed from the CPU_DYING notifier already. Further attempts
      to enqueue an irq_work are buggy because irqs are disabled by
      __cpu_disable(). The best we can do is to report the issue to the user.
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      8aa2acce
    • Steven Rostedt's avatar
      irq_work: Flush work on CPU_DYING · c0e980a4
      Steven Rostedt authored
      In order not to offline a CPU with pending irq works, flush the
      queue from CPU_DYING. The notifier is called by stop_machine on
      the CPU that is going down. The code will not be called from irq context
      (so things like get_irq_regs() wont work) but I'm not sure what the
      requirements are for irq_work in that regard (Peter?). But irqs are
      disabled and the CPU is about to go offline. Might as well flush the work.
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      c0e980a4
    • Frederic Weisbecker's avatar
      irq_work: Don't stop the tick with pending works · 00b42959
      Frederic Weisbecker authored
      Don't stop the tick if we have pending irq works on the
      queue, otherwise if the arch can't raise self-IPIs, we may not
      find an opportunity to execute the pending works for a while.
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      Acked-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
      00b42959
  16. 14 Nov, 2012 2 commits
    • Frederic Weisbecker's avatar
      irq_work: Fix racy check on work pending flag · e0bbe2d8
      Frederic Weisbecker authored
      Work claiming wants to be SMP-safe.
      
      And by the time we try to claim a work, if it is already executing
      concurrently on another CPU, we want to succeed the claiming and queue
      the work again because the other CPU may have missed the data we wanted
      to handle in our work if it's about to complete there.
      
      This scenario is summarized below:
      
              CPU 1                                   CPU 2
              -----                                   -----
              (flags = 0)
              cmpxchg(flags, 0, IRQ_WORK_FLAGS)
              (flags = 3)
              [...]
              xchg(flags, IRQ_WORK_BUSY)
              (flags = 2)
              func()
                                                      if (flags & IRQ_WORK_PENDING)
                                                              (not true)
                                                      cmpxchg(flags, flags, IRQ_WORK_FLAGS)
                                                      (flags = 3)
                                                      [...]
              cmpxchg(flags, IRQ_WORK_BUSY, 0);
              (fail, pending on CPU 2)
      
      This state machine is synchronized using [cmp]xchg() on the flags.
      As such, the early IRQ_WORK_PENDING check in CPU 2 above is racy.
      By the time we check it, we may be dealing with a stale value because
      we aren't using an atomic accessor. As a result, CPU 2 may "see"
      that the work is still pending on another CPU while it may be
      actually completing the work function exection already, leaving
      our data unprocessed.
      
      To fix this, we start by speculating about the value we wish to be
      in the work->flags but we only make any conclusion after the value
      returned by the cmpxchg() call that either claims the work or let
      the current owner handle the pending work for us.
      Changelog-heavily-inspired-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      Acked-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
      Cc: Anish Kumar <anish198519851985@gmail.com>
      e0bbe2d8
    • Frederic Weisbecker's avatar
      irq_work: Fix racy IRQ_WORK_BUSY flag setting · c8446b75
      Frederic Weisbecker authored
      The IRQ_WORK_BUSY flag is set right before we execute the
      work. Once this flag value is set, the work enters a
      claimable state again.
      
      So if we have specific data to compute in our work, we ensure it's
      either handled by another CPU or locally by enqueuing the work again.
      This state machine is guanranteed by atomic operations on the flags.
      
      So when we set IRQ_WORK_BUSY without using an xchg-like operation,
      we break this guarantee as in the following summarized scenario:
      
              CPU 1                                   CPU 2
              -----                                   -----
                                                      (flags = 0)
                                                      old_flags = flags;
              (flags = 0)
              cmpxchg(flags, old_flags,
                      old_flags | IRQ_WORK_FLAGS)
              (flags = 3)
              [...]
              flags = IRQ_WORK_BUSY
              (flags = 2)
              func()
                                                      (sees flags = 3)
                                                      cmpxchg(flags, old_flags,
                                                              old_flags | IRQ_WORK_FLAGS)
                                                      (give up)
      
              cmpxchg(flags, 2, 0);
              (flags = 0)
      
      CPU 1 claims a work and executes it, so it sets IRQ_WORK_BUSY and
      the work is again in a claimable state. Now CPU 2 has new data to process
      and try to claim that work but it may see a stale value of the flags
      and think the work is still pending somewhere that will handle our data.
      This is because CPU 1 doesn't set IRQ_WORK_BUSY atomically.
      
      As a result, the data expected to be handle by CPU 2 won't get handled.
      
      To fix this, use xchg() to set IRQ_WORK_BUSY, this way we ensure the CPU 2
      will see the correct value with cmpxchg() using the expected ordering.
      Changelog-heavily-inspired-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      Acked-by: default avatarSteven Rostedt <rostedt@goodmis.org>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
      Cc: Anish Kumar <anish198519851985@gmail.com>
      c8446b75
  17. 13 Apr, 2012 1 commit
  18. 02 Apr, 2012 1 commit
  19. 31 Oct, 2011 2 commits
    • Paul Gortmaker's avatar
      kernel: fix two implicit header assumptions in irq_work.c · 967d1f90
      Paul Gortmaker authored
      Up until now, this file was getting percpu.h because nearly every
      file was implicitly getting module.h (and all its sub-includes).
      But we want to clean that up, so call out percpu.h explicitly.
      Otherwise we'll get things like this on an ARM build:
      
      kernel/irq_work.c:48: error: expected declaration specifiers or '...' before 'irq_work_list'
      kernel/irq_work.c:48: warning: type defaults to 'int' in declaration of 'DEFINE_PER_CPU'
      
      The same thing was happening for builds on ARM for asm/processor.h
      
      kernel/irq_work.c: In function 'irq_work_sync':
      kernel/irq_work.c:166: error: implicit declaration of function 'cpu_relax'
      Signed-off-by: default avatarPaul Gortmaker <paul.gortmaker@windriver.com>
      967d1f90
    • Paul Gortmaker's avatar
      kernel: Map most files to use export.h instead of module.h · 9984de1a
      Paul Gortmaker authored
      The changed files were only including linux/module.h for the
      EXPORT_SYMBOL infrastructure, and nothing else.  Revector them
      onto the isolated export header for faster compile times.
      
      Nothing to see here but a whole lot of instances of:
      
        -#include <linux/module.h>
        +#include <linux/export.h>
      
      This commit is only changing the kernel dir; next targets
      will probably be mm, fs, the arch dirs, etc.
      Signed-off-by: default avatarPaul Gortmaker <paul.gortmaker@windriver.com>
      9984de1a
  20. 04 Oct, 2011 2 commits
  21. 18 Dec, 2010 1 commit
    • Christoph Lameter's avatar
      irq_work: Use per cpu atomics instead of regular atomics · 20b87691
      Christoph Lameter authored
      The irq work queue is a per cpu object and it is sufficient for
      synchronization if per cpu atomics are used. Doing so simplifies
      the code and reduces the overhead of the code.
      
      Before:
      
      christoph@linux-2.6$ size kernel/irq_work.o
         text	   data	    bss	    dec	    hex	filename
          451	      8	      1	    460	    1cc	kernel/irq_work.o
      
      After:
      
      christoph@linux-2.6$ size kernel/irq_work.o 
         text	   data	    bss	    dec	    hex	filename
          438	      8	      1	    447	    1bf	kernel/irq_work.o
      
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Signed-off-by: default avatarChristoph Lameter <cl@linux.com>
      20b87691
  22. 18 Nov, 2010 1 commit
  23. 18 Oct, 2010 1 commit
    • Peter Zijlstra's avatar
      irq_work: Add generic hardirq context callbacks · e360adbe
      Peter Zijlstra authored
      Provide a mechanism that allows running code in IRQ context. It is
      most useful for NMI code that needs to interact with the rest of the
      system -- like wakeup a task to drain buffers.
      
      Perf currently has such a mechanism, so extract that and provide it as
      a generic feature, independent of perf so that others may also
      benefit.
      
      The IRQ context callback is generated through self-IPIs where
      possible, or on architectures like powerpc the decrementer (the
      built-in timer facility) is set to generate an interrupt immediately.
      
      Architectures that don't have anything like this get to do with a
      callback from the timer tick. These architectures can call
      irq_work_run() at the tail of any IRQ handlers that might enqueue such
      work (like the perf IRQ handler) to avoid undue latencies in
      processing the work.
      Signed-off-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Acked-by: default avatarKyle McMartin <kyle@mcmartin.ca>
      Acked-by: default avatarMartin Schwidefsky <schwidefsky@de.ibm.com>
      [ various fixes ]
      Signed-off-by: default avatarHuang Ying <ying.huang@intel.com>
      LKML-Reference: <1287036094.7768.291.camel@yhuang-dev>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      e360adbe