releases/2.6.32.12/x86-64-rwsem-avoid-store-forwarding-hazard-in-__downgrade_write.patch

   1 From 0d1622d7f526311d87d7da2ee7dd14b73e45d3fc Mon Sep 17 00:00:00 2001
   2 From: Avi Kivity <avi@redhat.com>
   3 Date: Sat, 13 Feb 2010 10:33:12 +0200
   4 Subject: x86-64, rwsem: Avoid store forwarding hazard in __downgrade_write
   5
   6 From: Avi Kivity <avi@redhat.com>
   7
   8 commit 0d1622d7f526311d87d7da2ee7dd14b73e45d3fc upstream.
   9
  10 The Intel Architecture Optimization Reference Manual states that a short
  11 load that follows a long store to the same object will suffer a store
  12 forwading penalty, particularly if the two accesses use different addresses.
  13 Trivially, a long load that follows a short store will also suffer a penalty.
  14
  15 __downgrade_write() in rwsem incurs both penalties:  the increment operation
  16 will not be able to reuse a recently-loaded rwsem value, and its result will
  17 not be reused by any recently-following rwsem operation.
  18
  19 A comment in the code states that this is because 64-bit immediates are
  20 special and expensive; but while they are slightly special (only a single
  21 instruction allows them), they aren't expensive: a test shows that two loops,
  22 one loading a 32-bit immediate and one loading a 64-bit immediate, both take
  23 1.5 cycles per iteration.
  24
  25 Fix this by changing __downgrade_write to use the same add instruction on
  26 i386 and on x86_64, so that it uses the same operand size as all the other
  27 rwsem functions.
  28
  29 Signed-off-by: Avi Kivity <avi@redhat.com>
  30 LKML-Reference: <1266049992-17419-1-git-send-email-avi@redhat.com>
  31 Signed-off-by: H. Peter Anvin <hpa@zytor.com>
  32 Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
  33
  34 ---
  35  arch/x86/include/asm/rwsem.h |   25 +++++--------------------
  36  1 file changed, 5 insertions(+), 20 deletions(-)
  37
  38 --- a/arch/x86/include/asm/rwsem.h
  39 +++ b/arch/x86/include/asm/rwsem.h
  40 @@ -232,34 +232,19 @@ static inline void __up_write(struct rw_
  41   */
  42  static inline void __downgrade_write(struct rw_semaphore *sem)
  43  {
  44 -#ifdef CONFIG_X86_64
  45 -# if RWSEM_WAITING_BIAS != -0x100000000
  46 -#  error "This code assumes RWSEM_WAITING_BIAS == -2^32"
  47 -# endif
  48 -
  49 -       /* 64-bit immediates are special and expensive, and not needed here */
  50 -       asm volatile("# beginning __downgrade_write\n\t"
  51 -                    LOCK_PREFIX "incl 4(%1)\n\t"
  52 -                    /* transitions 0xZZZZZZZZ00000001 -> 0xYYYYYYYY00000001 */
  53 -                    "  jns       1f\n\t"
  54 -                    "  call call_rwsem_downgrade_wake\n"
  55 -                    "1:\n\t"
  56 -                    "# ending __downgrade_write\n"
  57 -                    : "+m" (sem->count)
  58 -                    : "a" (sem)
  59 -                    : "memory", "cc");
  60 -#else
  61         asm volatile("# beginning __downgrade_write\n\t"
  62                      LOCK_PREFIX _ASM_ADD "%2,(%1)\n\t"
  63 -                    /* transitions 0xZZZZ0001 -> 0xYYYY0001 */
  64 +                    /*
  65 +                     * transitions 0xZZZZ0001 -> 0xYYYY0001 (i386)
  66 +                     *     0xZZZZZZZZ00000001 -> 0xYYYYYYYY00000001 (x86_64)
  67 +                     */
  68                      "  jns       1f\n\t"
  69                      "  call call_rwsem_downgrade_wake\n"
  70                      "1:\n\t"
  71                      "# ending __downgrade_write\n"
  72                      : "+m" (sem->count)
  73 -                    : "a" (sem), "i" (-RWSEM_WAITING_BIAS)
  74 +                    : "a" (sem), "er" (-RWSEM_WAITING_BIAS)
  75                      : "memory", "cc");
  76 -#endif
  77  }
  78
  79  /*