single-stepping and internal breakpoints on a multi-threaded program

single-stepping and internal breakpoints on a multi-threaded program

[Emi SUZUKI]

[-- Attachment #1: Type: Text/Plain, Size: 2683 bytes --]

Hello members, 

My colleague has found a problem about handling single-stepping on a
multi-threaded target running on Linux box.  

The attached program makes two sub threads, and we do single-stepping
on one of them.  
When the target get stopped by a internal breakpoint hit (ex. thread
creation, thread death, shared library loading) during singe-stepping,
GDB unconditionally resume all the thread and it result in running
over the range of single-stepping.  

The session log below shows one of the phenomena reproduced on X86
FC6; the problem occured when the second thread was created.  
I can see them 10-20% of the time on our Cell hardware running FC5
(note that the program is compiled just for PPC64) and X86 FC5/FC6.  

----
GNU gdb 6.6.50.20070330-cvs
Copyright (C) 2007 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) b 14
Breakpoint 1 at 0x80484fa: file mt.c, line 14.
(gdb) l 14
9       {
10        int i;
11        int cnt = *((int *) arg);
12        pid_t pid = getpid();
13
14        for (i=0; i<cnt; i++)
15          printf ("[%d]counter: %d\n", pid, i);
16
17        return NULL;
18      }
(gdb) r
Starting program: /project/07/user/suzuki/test/mt/mt
[Thread debugging using libthread_db enabled]
[New Thread -1208011072 (LWP 28472)]
[28472]start
[New Thread -1208013936 (LWP 28475)]
[Switching to Thread -1208013936 (LWP 28475)]

Breakpoint 1, counter (arg=0xbfd491b0) at mt.c:14
14        for (i=0; i<cnt; i++)
(gdb) s
[New Thread -1218503792 (LWP 28476)]
[28472]no_counter
[Thread -1218503792 (LWP 28476) exited]
[28472]counter: 0
[28472]counter: 1
[28472]counter: 2
[28472]counter: 3
[28472]counter: 4
[28472]counter: 5
[28472]counter: 6
[28472]counter: 7
[28472]counter: 8
[28472]counter: 9
[Thread -1208013936 (LWP 28475) exited]
Thread Event Breakpoint: gdb should not stop!
(gdb) 
---- 

I saw that the context of single-stepping thread is preserved by
context_switch called in handle_inferior_event, but GDB doesn't go
back to that context till the thread is trapped in another reason
(in the above case, the death of the thread).  

I think it should either stop immediately and prompt the user that the
single-stepping has been cancelled, or goes back to the preserved
context somehow and continue single-stepping.  
How do you think of it? 

Thanks in advance, 
-- 
Emi SUZUKI

[-- Attachment #2: mt.c --]
[-- Type: Text/Plain, Size: 1077 bytes --]

#include <stdio.h>
#include <sys/types.h>
#include <pthread.h>
#include <unistd.h>
#include <string.h>

void *
counter(void *arg)
{
  int i;
  int cnt = *((int *) arg);
  pid_t pid = getpid();

  for (i=0; i<cnt; i++)
    printf ("[%d]counter: %d\n", pid, i);

  return NULL;
}

void *
no_counter (void *arg)
{
  pid_t pid = getpid();

  printf ("[%d]no_counter\n", pid);

  return NULL;
}

int
main (void)
{
  pid_t p_pid;
  pthread_t thread_id1, thread_id2;
  int status;
  int count = 10;

  p_pid = getpid ();
  printf ("[%d]start\n", p_pid);

  status = pthread_create (&thread_id1, NULL, counter, (void *)&count);
  if (status != 0)
    {
      fprintf (stderr, "pthread_create : %s", strerror(status));
      return (1);
    }

  status = pthread_create (&thread_id2, NULL, no_counter, (void *)NULL);
  if (status != 0)
    {
      fprintf (stderr, "pthread_create : %s", strerror(status));
      pthread_join (thread_id1, NULL);
      return (1);
    }

  pthread_join (thread_id2, NULL);
  pthread_join (thread_id1, NULL);

  printf ("[%d]end\n",p_pid);
  return 0;
}

Re: single-stepping and internal breakpoints on a multi-threaded program

[Jim Blandy]

Emi SUZUKI <emi-suzuki@tjsys.co.jp> writes:
> The attached program makes two sub threads, and we do single-stepping
> on one of them.  
> When the target get stopped by a internal breakpoint hit (ex. thread
> creation, thread death, shared library loading) during singe-stepping,
> GDB unconditionally resume all the thread and it result in running
> over the range of single-stepping.  

I'm able to reproduce this problem on my FC6 x86 dual core system.
Thanks very much for the clear reproduction instructions.

> I saw that the context of single-stepping thread is preserved by
> context_switch called in handle_inferior_event, but GDB doesn't go
> back to that context till the thread is trapped in another reason
> (in the above case, the death of the thread).  
>
> I think it should either stop immediately and prompt the user that the
> single-stepping has been cancelled, or goes back to the preserved
> context somehow and continue single-stepping.  
> How do you think of it? 

So, if I'm understanding correctly, we single-step the 'counter'
thread, which proceeds normally until the 'no_counter' thread hits its
TD_CREATE event, to tell GDB about its existence.  Then, GDB fails to
resume single-stepping the 'counter' thread, letting it run until it
exits and reports its TD_DEATH event.

The fact that the inferior uses breakpoints to report thread creation
and thread death is supposed to be an internal detail of the thread
library --- or, more properly, of the thread debugging interface.  One
thread stepping shouldn't be affected by other threads being created
(although GDB should still print the 'new thread' messages).

So I think the right behavior would be for GDB to continue
single-stepping after processing creation and death events for other
threads.

I have forgetten exactly how GDB handles multi-threaded single
stepping; I'm afraid I can't suggest how to do this.  If you're unable
to fix it yourself, please file a bug report, and include your test
program.

Re: single-stepping and internal breakpoints on a multi-threaded program

[Michael Snyder]

On Fri, 2007-04-06 at 15:12 -0700, Jim Blandy wrote:
> Emi SUZUKI <emi-suzuki@tjsys.co.jp> writes:
> > The attached program makes two sub threads, and we do single-stepping
> > on one of them.  
> > When the target get stopped by a internal breakpoint hit (ex. thread
> > creation, thread death, shared library loading) during singe-stepping,
> > GDB unconditionally resume all the thread and it result in running
> > over the range of single-stepping.  
> 
> I'm able to reproduce this problem on my FC6 x86 dual core system.
> Thanks very much for the clear reproduction instructions.
> 
> > I saw that the context of single-stepping thread is preserved by
> > context_switch called in handle_inferior_event, but GDB doesn't go
> > back to that context till the thread is trapped in another reason
> > (in the above case, the death of the thread).  
> >
> > I think it should either stop immediately and prompt the user that the
> > single-stepping has been cancelled, or goes back to the preserved
> > context somehow and continue single-stepping.  
> > How do you think of it? 
> 
> So, if I'm understanding correctly, we single-step the 'counter'
> thread, which proceeds normally until the 'no_counter' thread hits its
> TD_CREATE event, to tell GDB about its existence.  Then, GDB fails to
> resume single-stepping the 'counter' thread, letting it run until it
> exits and reports its TD_DEATH event.
> 
> The fact that the inferior uses breakpoints to report thread creation
> and thread death is supposed to be an internal detail of the thread
> library --- or, more properly, of the thread debugging interface.  One
> thread stepping shouldn't be affected by other threads being created
> (although GDB should still print the 'new thread' messages).
> 
> So I think the right behavior would be for GDB to continue
> single-stepping after processing creation and death events for other
> threads.
> 
> I have forgetten exactly how GDB handles multi-threaded single
> stepping; I'm afraid I can't suggest how to do this.  If you're unable
> to fix it yourself, please file a bug report, and include your test
> program.

I'm thinking that gdb just loses the "I'm stepping this thread" 
bit of state info, which is normally handled in "proceed" and 
stored in the infrun-state struct - whereupon it just continues
both threads without stepping.


But that's just a hunch.

Re: single-stepping and internal breakpoints on a multi-threaded program

[Emi SUZUKI]

Jim and Michael,

Thank you for your comments.  

From: Michael Snyder <Michael dot Snyder at access-company dot com>
Subject: Re: single-stepping and internal breakpoints on a multi-threaded program
Date: Fri, 06 Apr 2007 15:25:09 -0700

> > I have forgetten exactly how GDB handles multi-threaded single
> > stepping; I'm afraid I can't suggest how to do this.  If you're unable
> > to fix it yourself, please file a bug report, and include your test
> > program.
> 
> I'm thinking that gdb just loses the "I'm stepping this thread" 
> bit of state info, which is normally handled in "proceed" and 
> stored in the infrun-state struct - whereupon it just continues
> both threads without stepping.

I think it's true in some ways: it seemed that the clue for GDB to
decide that "I'm stepping this thread" is eventually only the value of
step_range_end.  

When the target get stopped on a thread different from which GDB
expected to stop, it was preserved in the thread_info structure by
context_switch, but never had an oppotunity to be looked back till the
target get stopped on that thread again.  It is not just lost, but no
better than lost.  

I think I should work more on the problem, but unfortunately, my
copyright assignment to FSF is still not ready.  So I will file it to
the bug tracking system, following Jim's advice.  
I will be back on it when I find that anything is not done for it
after my assignment is ready.  I hope not, though :-)

Thanks anyway, 
-- 
Emi SUZUKI