Suggestion: backtrace stop guard for threads callstacks

Suggestion: backtrace stop guard for threads callstacks

[Joel Brobecker]

Hello,

We have recently implemented a small feature in our version of the
debugger. While the implementation itself is not ready for inclusion
here, and not directly interesting in any case (lacking support for
Ada tasks for now, see below), I thought the general idea might be
interesting to the group. And I think it's not going to be too time
consuming, so could probably produce a patch.

One of our customers reported that he saw a warning from the debugger
saying that a frame was identical to the previous frame. Here is the
kind of output he would see:

> #3  0x000370d4 in system.tasking.stages.task_wrapper ()
> #4  0xff34b124 in _thread_start () from /usr/lib/libthread.so.1
> #5  0xff34b124 in _thread_start () from /usr/lib/libthread.so.1
> warning: Previous frame identical to this frame (corrupt stack?)

In this case, the user tried to print a backtrace from inside an Ada
task, which is mapped over a thread, and that the unwinder didn't
(couldn't?) know that it was time to stop at frame #4. The same sort
of situation happens with the main process, outside of a thread context,
where we sometimes cannot know that we reached the top of the callstack.

For the main process, we have a guard put in place that will stop it
when reaching the "main program". See main_name(), inside_main_func(),
and get_prev_frame().

What we've done for Ada tasks, is to take advantage of the fact that
we know that tasks are run from a known routine in the GNAT runtime,
called System.Tasking.[...].Task_Wrapper. So we added a small piece
of code to detect Ada frames corresponding to this routine (we actually
have two, but the principle is the same), and declare that we're
inside the main func if the task_wrapper is detected.

The complete backtrace can be obtained using the already implemented

        (gdb) set backtrace past-main

It's a little bit less simple for the general case for C, C++ and other
languages, but I suggest we implement something similar. How about we
add some code that recognizes some of the thread entry-points? For
the GNU/Linux NPTL for instance, we could recognize clone() or perhaps
_start_thread? We could make the check as stringent as we like, for
instance not only check the funtion name, but also the object name,
to make sure we're inside the pthread library.

Just throwing some ideas.

My overall idea is to implement a mechanism where various modules
in GDB could register their own routine that would be able to tell
whether they recognize the given function as a main or not.

For instance, sol-thread.c could implement its routine, and register
it during the init phase... inside_main_func() would then go through
this list, and call each one of them until one recognizes it or until
it reaches its end.

Even grander, have a language-specific-hook, followed by list of
registered sniffers. So inside_main_func() would look like this:

    msymbol = lookup_minimal_symbol (main_name (), NULL, symfile_objfile);
    [check this symbol to see if it's the main]

    /* Check this frame using the language-specific sniffer */
    symtab = find_pc_symtab (pc);
    if (symtab != NULL)
      if (language_main_name_sniffer[language] (this_frame))
        return 1;

    /* Check using the other sniffers  */
    FOR_ALL_MAIN_FUNC_SNIFFERS (sniffer)
      if (sniffer (this_frame))
        return 1;

    return 0;

Thoughts?
-- 
Joel

Re: Suggestion: backtrace stop guard for threads callstacks

[Daniel Jacobowitz]

On Wed, Mar 01, 2006 at 05:29:43PM -0800, Joel Brobecker wrote:
> It's a little bit less simple for the general case for C, C++ and other
> languages, but I suggest we implement something similar. How about we
> add some code that recognizes some of the thread entry-points? For
> the GNU/Linux NPTL for instance, we could recognize clone() or perhaps
> _start_thread? We could make the check as stringent as we like, for
> instance not only check the funtion name, but also the object name,
> to make sure we're inside the pthread library.

FYI, this is discussed here about five times a year in one form or
another.  For systems we control, the correct solution is not this at
all, but to add unwinding information or suitable prologues that
indicate the end of the stack.  Almost every architecture has a
suitable sequence, and we have a recently-decided-upon DWARF convention
to represent it also.

Whatever platform you're looking at, does the beginning of
_thread_start make sufficiently clear that we're at the end of the
stack?  If not, perhaps you should register an appropriate unwinder to
notice it and stop.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Suggestion: backtrace stop guard for threads callstacks

[Andrew STUBBS]

Daniel Jacobowitz wrote:
> FYI, this is discussed here about five times a year in one form or
> another.  For systems we control, the correct solution is not this at
> all, but to add unwinding information or suitable prologues that
> indicate the end of the stack.  Almost every architecture has a
> suitable sequence, and we have a recently-decided-upon DWARF convention
> to represent it also.

Yes, I started this same discussion last autumn. We finally decided that 
the Right Thing would be to do something to the library that creates the 
thread stack.

But, while we are having ideas....

How about:

set backtrace main-functions main;mytaskfunc;_start

or perhaps:

set backtrace add-main-function mytaskfunc

so that the user can stop backtraces whereever is appropriate for their 
environment.


Just a thought.

Andrew

Re: Suggestion: backtrace stop guard for threads callstacks

[Jim Blandy]

On 3/2/06, Andrew STUBBS <andrew.stubbs@st.com> wrote:
> How about:
>
> set backtrace main-functions main;mytaskfunc;_start
>
> or perhaps:
>
> set backtrace add-main-function mytaskfunc
>
> so that the user can stop backtraces whereever is appropriate for their
> environment.

That's certainly easy to do.  But it's clearly preferable for people
to properly mark their thread start functions.  If I'm remembering
right, we haven't yet seen any reports of this problem in cases where
we couldn't get the thread start function marked.  So I'd argue we
don't have any situations where this command was needed --- only cases
where using it would have helped people paper over an underlying
problem that needs to be fixed elsewhere.

Given how often this comes up, would it make sense to add an appendix
to the GDB manual explaining how to terminate backtraces?

Re: Suggestion: backtrace stop guard for threads callstacks

[Joel Brobecker]

> FYI, this is discussed here about five times a year in one form or
> another.

Yes, I remember now. But ...

> For systems we control, the correct solution is not this at
> all, but to add unwinding information or suitable prologues that
> indicate the end of the stack.  Almost every architecture has a
> suitable sequence, and we have a recently-decided-upon DWARF convention
> to represent it also.

I understand. But I also thought that it would be nice to avoid printing
the frames that are relative to thread startup code, and start the
backtrace at the function the code used in the pthread_create call
for instance. I think that the user won't be interested in them most
of the time.

That was part of my motivation. No big deal, I'll solve the other
issues the recommended way the next time I come across them.

-- 
Joel

Re: Suggestion: backtrace stop guard for threads callstacks

[Daniel Jacobowitz]

On Thu, Mar 02, 2006 at 09:55:20AM -0800, Joel Brobecker wrote:
> I understand. But I also thought that it would be nice to avoid printing
> the frames that are relative to thread startup code, and start the
> backtrace at the function the code used in the pthread_create call
> for instance. I think that the user won't be interested in them most
> of the time.

Oh - actually stop the backtrace _before_ the library instead of _in_
it?

I'm not sure I agree; while the user may not be interested in the
library functions, you can call thread startup functions as normal
functions also; having backtraces stop without a clear indication
of why would be a little confusing too.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Suggestion: backtrace stop guard for threads callstacks

[Daniel Jacobowitz]

On Thu, Mar 02, 2006 at 09:55:20AM -0800, Joel Brobecker wrote:
> I understand. But I also thought that it would be nice to avoid printing
> the frames that are relative to thread startup code, and start the
> backtrace at the function the code used in the pthread_create call
> for instance. I think that the user won't be interested in them most
> of the time.

Oh - actually stop the backtrace _before_ the library instead of _in_
it?

I'm not sure I agree; while the user may not be interested in the
library functions, you can call thread startup functions as normal
functions also; having backtraces stop without a clear indication
of why would be a little confusing too.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Suggestion: backtrace stop guard for threads callstacks

[Jim Blandy]

On 3/2/06, Daniel Jacobowitz <drow@false.org> wrote:
> On Thu, Mar 02, 2006 at 09:55:20AM -0800, Joel Brobecker wrote:
> > I understand. But I also thought that it would be nice to avoid printing
> > the frames that are relative to thread startup code, and start the
> > backtrace at the function the code used in the pthread_create call
> > for instance. I think that the user won't be interested in them most
> > of the time.
>
> Oh - actually stop the backtrace _before_ the library instead of _in_
> it?
>
> I'm not sure I agree; while the user may not be interested in the
> library functions, you can call thread startup functions as normal
> functions also; having backtraces stop without a clear indication
> of why would be a little confusing too.

I guess part of the problem here is that we're defining the problem in
terms of recognizing frames after which we don't unwind, instead of
saying, "These functions are part of the run-time and their frames
shouldn't be listed normally."

That way, you'd just mark _start and the thread startup trampoline,
and you'd never get confused by recursive uses of main or calls to the
thread initial functions from other places.

In other words, we're somewhat off-by-one with our predicates.

Re: Suggestion: backtrace stop guard for threads callstacks

[Jim Blandy]

On 3/2/06, Daniel Jacobowitz <drow@false.org> wrote:
> On Thu, Mar 02, 2006 at 09:55:20AM -0800, Joel Brobecker wrote:
> > I understand. But I also thought that it would be nice to avoid printing
> > the frames that are relative to thread startup code, and start the
> > backtrace at the function the code used in the pthread_create call
> > for instance. I think that the user won't be interested in them most
> > of the time.
>
> Oh - actually stop the backtrace _before_ the library instead of _in_
> it?
>
> I'm not sure I agree; while the user may not be interested in the
> library functions, you can call thread startup functions as normal
> functions also; having backtraces stop without a clear indication
> of why would be a little confusing too.

I guess part of the problem here is that we're defining the problem in
terms of recognizing frames after which we don't unwind, instead of
saying, "These functions are part of the run-time and their frames
shouldn't be listed normally."

That way, you'd just mark _start and the thread startup trampoline,
and you'd never get confused by recursive uses of main or calls to the
thread initial functions from other places.

In other words, we're somewhat off-by-one with our predicates.