gdbserver/linux-low: carry over stop_expected flag after exec (avoid spurious SIGSTOPs)

Running test gdb.base/vfork-follow-parent.exp on the native-gdbserver
board is flaky.  This test has GDB debugging a vfork parent and child.
When resuming the child, we expect it to run through an exec, and then
exit, like so:

    continue
    Continuing.
    [New inferior 2 (process 3286309)]
    process 3286309 is executing new program: /home/simark/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/vfork-follow-parent/vforked-prog
    [Inferior 2 (process 3286309) exited normally]
    (gdb) PASS: gdb.base/vfork-follow-parent.exp: exec_file=vfork-follow-parent-exec: target-non-stop=off: non-stop=off: resolution_method=schedule-multiple: continue to end of inferior 2

Instead, we sometimes see it reporting a spurious SIGSTOP:

    continue
    Continuing.
    [New inferior 2 (process 3281383)]
    process 3281383 is executing new program: /home/simark/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/vfork-follow-parent/vforked-prog

    Thread 2.1 "vforked-prog" received signal SIGSTOP, Stopped (signal).
    Cannot remove breakpoints because program is no longer writable.
    Further execution is probably impossible.
    0x00007ffff7fe0300 in ?? () from /lib64/ld-linux-x86-64.so.2
    (gdb) FAIL: gdb.base/vfork-follow-parent.exp: exec_file=vfork-follow-parent-exec: target-non-stop=off: non-stop=off: resolution_method=schedule-multiple: continue to end of inferior 2

Note that this case uses "target-non-stop=off", meaning that when one
thread reports a stop, GDBserver attempts to stop the other threads
itself before reporting the stop to GDB.  The bug is related to
GDBserver's bookkeeping of the threads.

When running the test 100 times, it failed 6 times.  Using taskset to
pin the process subtree to a single CPU seemed to make it more likely
to fail: on 100 runs, it failed 30 times.

The cause
=========

The steps occurring in the "pass" case are:

 1. Process P (Parent) is blocked inside the vfork system call, waiting
    for process C (Child) to exec or exit.
 2. Process C is currently stopped at its first instruction (right out
    of vfork).
 3. We resume process C (the "continue" seen in the logs above).
 4. Process C calls exec, as a result the kernel produces two events to
    be consumed by the tracer (GDBserver):

     - For P, a vfork done event (PTRACE_EVENT_VFORK_DONE)
     - For C, an exec event (PTRACE_EVENT_EXEC)

 5. Let's suppose here that waitpid happens to return the
    PTRACE_EVENT_EXEC for C first.
 6. Upon receiving the event, GDBserver calls stop_all_lwps, which sends
    a SIGSTOP to P, and sets `lwp->stop_expected` for P.
 7. GDBserver waits for P to stop, waitpid returns the
    PTRACE_EVENT_VFORK_DONE event.  GDBserver stashes it in
    lwp->status_pending.  The SIGSTOP for P is still pending at the
    kernel level.
 8. When we resume C, everything is fine, it runs until exit.
 9. When we resume P, GDBserver receives the SIGSTOP event for P, but
    suppresses it because `lwp->stop_expected` is set for P.

Now, for the failing case, imagine that at step 5 the kernel decided to
return the PTRACE_EVENT_VFORK_DONE event for P first.  The following
steps would play out like this:

 6. Upon receiving the event, GDBserver calls stop_all_lwps, which sends
    a SIGSTOP to C, and sets `lwp->stop_expected` for C.
 7. GDBserver waits for C to stop, waitpid returns the PTRACE_EVENT_EXEC
    event.
 8. In the handling of PTRACE_EVENT_EXEC (in
    linux_process_target::handle_extended_wait), GDBserver deletes the
    process and its threads (the mourn call), and creates a brand new
    process and lwp_info for C.  Note that this loses the previously set
    `lwp->stop_expected`.  The SIGSTOP for C is still pending at the
    kernel level.
 9. When we resume C, GDBserver receives the SIGSTOP event for C, and
    because `lwp->stop_expected` is not set, GDBserver doesn't recognize
    it as its own, and the stop is presented to the user.

The fix
=======

The (simplest) fix is to transfer the `lwp->stop_expected` from the old
lwp_info to the new one.  When doing so, when we resume C and GDBserver
receives the SIGSTOP event for C, it recognizes it as its own and
suppresses it.

Some care is needed in case process C is multi-threaded and the exec is
done by a non-leader thread.  When a non-leader thread execs, the kernel
scraps all other threads and renumbers this one to the tgid, so that it
becomes the new leader.  The PTRACE_EVENT_EXEC event is reported using
that new renumbered id.  But if a SIGSTOP was pending for the non-leader
exec'ing thread when the exec happened, it will still be pending
post-exec for that thread under its new post-exec leader identity.

Here is a hypothetical but more concrete scenario:

 - There are two threads, 100.100 (the leader) and 100.101.
 - Thread 100.101 is stopped at the entry of the execve system call (so
   the effects of execve haven't occurred yet) when GDBserver sends it a
   SIGSTOP and sets `lwp->stop_expected`.  The SIGSTOP becomes pending
   in the kernel.
 - When 100.101 is resumed, the exec occurs, the kernel deletes thread
   100.100 and renumbers 100.101 to 100.100.  The latter still has the
   SIGSTOP pending.
 - GDBserver receives a PTRACE_EVENT_EXEC event for thread 100.100.
 - Upon resumption, GDBserver then receives an event for the SIGSTOP,
   for thread 100.100.

All this to say that when transferring the `lwp->stop_expected` flag
from the old lwp_info to the new, we must take care to read the exec'ing
thread's flag.  If we use the id reported for the PTRACE_EVENT_EXEC to
look up an lwp_info, then we'll get the leader's lwp_info, which may not
have `lwp->stop_expected` set.  Instead, we must get the exec'ing
thread's original id using PTRACE_GETEVENTMSG, and use that to source
the right lwp_info to transfer the `lwp->stop_expected` flag.

There is a comment about using PTRACE_GETEVENTMSG with PTRACE_EVENT_EXEC
in gdb/linux-nat.c:

    ...
    tid to the tgid, and the previous leader vanishes.  Since
    Linux 3.0, the former thread ID can be retrieved with
    PTRACE_GETEVENTMSG, but since we support older kernels, don't
    bother with it, and just walk the LWP list.  Even with
    ...

Linux 3.0 was released in 2011, so I think it's fine to use that.

With the fix, I don't get any failures after 100 test runs (even with
taskset).

Kernel behavior experiments
===========================

The fix relies on assumptions about how the kernel orders ptrace events
and handles pending signals across exec.  I experimented with these
using standalone ptrace programs (mostly written by my buddy Claude).
The programs would drive a thread of the tracee to the execve syscall
entry, deliver a SIGSTOP to a thread, resume things and then look at
what events would come out of waitpid.  Here are the scenarios I tried:

  - Single-threaded: a tracee stopped at execve syscall entry is sent a
    SIGSTOP, then allowed to exec.  The kernel reports PTRACE_EVENT_EXEC first,
    then the SIGSTOP.

  - Multi-threaded, non-leader exec: a non-leader thread stopped at
    execve syscall entry is sent a SIGSTOP, then allowed to exec.  The
    kernel reports PTRACE_EVENT_EXEC, then the SIGSTOP, both under the
    leader id (the exec'ing non-leader got renumbered).

  - Multi-threaded, leader has a pending SIGSTOP while a non-leader
    execs: the leader is sent a SIGSTOP, which stays pending, then we
    let the non-leader thread exec (giving it a few seconds to be sure).
    The kernel reports PTRACE_EVENT_EXEC under the leader id (the
    exec'ing non-leader got renumbered) and the SIGSTOP has vanished.

What about GDB
==============

I tried to check if the same bug could happen with GDB's linux-nat
target, and if the same fix was needed.  linux-nat takes a different
approach when handling PTRACE_EVENT_EXEC.  It wipes all lwp_infos except
the leader:

      for (lwp_info &other_lp : all_lwps_safe ())
	if (&other_lp != lp && other_lp.ptid.pid () == lp->ptid.pid ())
	  exit_lwp (&other_lp);

Here, LP is an lwp_info obtained using the event ptid of the
PTRACE_EVENT_EXEC, therefore the leader's lwp_info (even if the exec was
done by a non-leader).

If the exec is done by the leader, as is the case with
gdb.base/vfork-follow-parent.exp, we are ok.  Because GDB doesn't delete
and re-create the lwp_info, the equivalent of GDBserver's
`lwp_info::stop_expected`, `lwp_info::signalled`, survives the exec.

If the exec is done by a non-leader, then we could be in trouble.  If
the leader's signalled flag is not set, but the exec'ing non-leader's
flag is set, then we'll lose it.  I suppose we could fix GDB to use
PTRACE_GETEVENTMSG to get the exec'ing thread former id, look up the
lwp_info for that id, and preserve that lwp_info.

Change-Id: Iaebd1d2cf813dcad35d7d8639bbaed80d40b7d1e
Approved-By: Pedro Alves <pedro@palves.net>

diff --git a/gdbserver/linux-low.cc b/gdbserver/linux-low.cc
index ade5e9e2a1c0f6146f9cff89e85772d2b8d4bef3..86ea2451c1c8c9afe38b094551c71135324f6d47 100644 (file)
--- a/gdbserver/linux-low.cc
+++ b/gdbserver/linux-low.cc
@@ -732,6 +732,40 @@ linux_process_target::handle_extended_wait (lwp_info **orig_event_lwp,
       event_ptid = event_thr->id;
       event_pid = event_ptid.pid ();
 
+      /* If GDBserver had sent the exec'ing thread a SIGSTOP and the thread
+        exec'ed before consuming it, the SIGSTOP stays pending across the exec.
+        The pending SIGSTOP is reported once the post-exec thread is resumed.
+        Carry the expectation over to the new lwp_info below, so that it is
+        filtered out instead of being reported to GDB.
+
+        Things get interesting if a non-leader thread does the exec.  The
+        kernel deletes the other threads, changes the exec'ing thread's id
+        so it matches the tgid (becomes the new leader), and reports the
+        PTRACE_EVENT_EXEC for that leader id.  If a SIGSTOP was pending on
+        the exec'ing non-leader as the exec happened, it will still be pending
+        post-exec, but it will be reported under leader id.
+
+        Since EVENT_LWP was looked up using the leader id, it represents the
+        former leader, not necessarily the exec'ing thread.  To properly carry
+        the stop expectation from the exec'ing thread's lwp_info to the new
+        lwp_info, use PTRACE_GETEVENTMSG to obtain the exec'ing thread's former
+        id, and look up the lwp_info from that.
+
+        If a SIGSTOP is pending (in the kernel) on the leader when a non-leader
+        execs, then that SIGSTOP disappears with the thread.  */
+      unsigned long execing_tid;
+      if (ptrace (PTRACE_GETEVENTMSG, event_ptid.lwp (), (PTRACE_TYPE_ARG3) 0,
+                 &execing_tid) != 0)
+       {
+         /* If ptrace fails, fall back to using the leader.  */
+         execing_tid = event_ptid.lwp ();
+       }
+
+      lwp_info *execing_lwp
+       = find_lwp_pid (ptid_t (event_pid, execing_tid));
+      bool stop_expected
+       = execing_lwp != nullptr && execing_lwp->stop_expected;
+
       /* Save the syscall list from the execing process.  */
       process_info *proc = event_thr->process ();
       syscalls_to_catch = std::move (proc->syscalls_to_catch);
@@ -747,6 +781,9 @@ linux_process_target::handle_extended_wait (lwp_info **orig_event_lwp,
       gdb_assert (current_thread == event_thr);
       arch_setup_thread (event_thr);
 
+      /* Carry over a possible SIGSTOP expectation.  */
+      event_lwp->stop_expected = stop_expected;
+
       /* Set the event status.  */
       event_lwp->waitstatus.set_execd
        (make_unique_xstrdup