Fix segmentation violation caused by stack misalignment when vgdb use ptrace to force...

On amd64, on a big application, a vgdb call that wakes up the application
using ptrace fails unfrequently (we speak about one failure every few thousands vgdb calls).
The failure started to appear when valgrind was compiled with gcc 7.3 instead of gcc 6.4

After investigation:
* gcc 7.3 is using (more) sse instructions
* Such instructions imply to have a stack pointer aligned on 16 bytes.
* vgdb-invoker-ptrace.c 'ptrace' modification of the stack pointer was
  not respecting the amd64 ABI convention to align on 16 bytes.
  It was also not protecting the red zone (unclear if this could cause
  the problem, but in any case, this ptrace logic is similar to a
  signal handler, and cannot modify the redzone.

The fix consists in respecting the ABI.

Without the patch, segmentation violation due to an sse instruction
being executed with an address on the stack not aligned on 16 bytes,
happening something like every 5000 vgdb execution.
With the patch, 250_000 executions without problems.

diff --git a/coregrind/vgdb-invoker-ptrace.c b/coregrind/vgdb-invoker-ptrace.c
index 98092c72cc82041d6f2a315eeeddae4e1674de43..bd640ed0994eb14e4d587432be41804406844a32 100644 (file)
--- a/coregrind/vgdb-invoker-ptrace.c
+++ b/coregrind/vgdb-invoker-ptrace.c
@@ -970,14 +970,16 @@ Bool invoker_invoke_gdbserver (pid_t pid)
       // vgdb speaking with a 64 bit executable.
       const int regsize = 8;
       int rw;
-      
+
       /* give check arg in rdi */
       user_mod.regs.rdi = check;
 
       /* push return address on stack : return to breakaddr */
+      sp &= ~0xf; // keep the stack aligned on 16 bytes ...
+      sp = sp - 128; // do not touch the amd64 redzone
       sp = sp - regsize;
       DEBUG(1, "push bad_return return address ptrace_write_memory\n");
-      rw = ptrace_write_memory(pid, sp, 
+      rw = ptrace_write_memory(pid, sp,
                                &bad_return,
                                sizeof(bad_return));
       if (rw != 0) {