percpu: add optimized generic percpu accessors
authorIngo Molnar <mingo@elte.hu>
Thu, 15 Jan 2009 13:15:53 +0000 (22:15 +0900)
committerIngo Molnar <mingo@elte.hu>
Fri, 16 Jan 2009 13:20:31 +0000 (14:20 +0100)
commit6dbde3530850d4d8bfc1b6bd4006d92786a2787f
tree08c6dd55e860827311b889e2ecfe3de9f51421a0
parent004aa322f855a765741d9437a98dd8fe2e4f32a6
percpu: add optimized generic percpu accessors

It is an optimization and a cleanup, and adds the following new
generic percpu methods:

  percpu_read()
  percpu_write()
  percpu_add()
  percpu_sub()
  percpu_and()
  percpu_or()
  percpu_xor()

and implements support for them on x86. (other architectures will fall
back to a default implementation)

The advantage is that for example to read a local percpu variable,
instead of this sequence:

 return __get_cpu_var(var);

 ffffffff8102ca2b: 48 8b 14 fd 80 09 74  mov    -0x7e8bf680(,%rdi,8),%rdx
 ffffffff8102ca32: 81
 ffffffff8102ca33: 48 c7 c0 d8 59 00 00  mov    $0x59d8,%rax
 ffffffff8102ca3a: 48 8b 04 10           mov    (%rax,%rdx,1),%rax

We can get a single instruction by using the optimized variants:

 return percpu_read(var);

 ffffffff8102ca3f: 65 48 8b 05 91 8f fd  mov    %gs:0x7efd8f91(%rip),%rax

I also cleaned up the x86-specific APIs and made the x86 code use
these new generic percpu primitives.

tj: * fixed generic percpu_sub() definition as Roel Kluin pointed out
    * added percpu_and() for completeness's sake
    * made generic percpu ops atomic against preemption

Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Tejun Heo <tj@kernel.org>
15 files changed:
arch/x86/include/asm/current.h
arch/x86/include/asm/irq_regs_32.h
arch/x86/include/asm/mmu_context_32.h
arch/x86/include/asm/pda.h
arch/x86/include/asm/percpu.h
arch/x86/include/asm/smp.h
arch/x86/kernel/process_32.c
arch/x86/kernel/tlb_32.c
arch/x86/mach-voyager/voyager_smp.c
arch/x86/xen/enlighten.c
arch/x86/xen/irq.c
arch/x86/xen/mmu.c
arch/x86/xen/multicalls.h
arch/x86/xen/smp.c
include/asm-generic/percpu.h