Using gdb as a trace agent

Using gdb as a trace agent

[Alexandre Courbot]

Hello everyone,

In my quest to get datas out of a program as non-intrusively as possible 
(I'm trying to get graphes out of the memory manager of an embedded 
operating system), I'm having a close and interested look at gdb 
tracepoints. It looks like a great solution, but unfortunately:

- It only works on remote targets, dixit the manual,
- Anyway, no remote target supports them.

Traditional breakpoints with additional commands are just too intrusive 
to me, unfortunately. Is there anything I missed on the tracepoints 
front? Or would anyone know a tool that I could use in replacement to 
gdb to do this? (i.e. getting the values of some variables at some 
points of the program, without slaughtering the source code).

Thanks for any information,
Alex.
-- 
Alexandre Courbot - PhD student
RD2P/LIFL
http://www.lifl.fr/~courbot

Re: Using gdb as a trace agent

[Jim Blandy]

Alexandre Courbot <Alexandre.Courbot@lifl.fr> writes:
> Hello everyone,
> 
> In my quest to get datas out of a program as non-intrusively as
> possible (I'm trying to get graphes out of the memory manager of an
> embedded operating system), I'm having a close and interested look at
> gdb tracepoints. It looks like a great solution, but unfortunately:
> 
> - It only works on remote targets, dixit the manual,

How are you communicating with the target?

> - Anyway, no remote target supports them.

I remember Michael talking about getting the stub tracing code
released to the public, but I don't remember what came of that.
Michael?

Re: Using gdb as a trace agent

[Alexandre Courbot]

>>In my quest to get datas out of a program as non-intrusively as
>>possible (I'm trying to get graphes out of the memory manager of an
>>embedded operating system), I'm having a close and interested look at
>>gdb tracepoints. It looks like a great solution, but unfortunately:
>>
>>- It only works on remote targets, dixit the manual,
> 
> 
> How are you communicating with the target?

Right now, preferred ways to use GDB with this platform is either to 
generate a native binary for your GNU environment (in which case GDB can 
run non-remotely, but tracepoints seems not to be supported at all in 
this case), or to generate a Game Boy Advance ROM that you run with some 
emulator that supports GDB. But that does not support tracepoints, 
unfortunately.

The ideal for me would be to be able to use tracepoints in both cases 
(i.e. remote and local). What would prevent a local support for tracepoints?

> I remember Michael talking about getting the stub tracing code
> released to the public, but I don't remember what came of that.
> Michael?

That would be nice indeed. But no chance to get tracepoints to work 
locally? That would turn GDB into a great analysis tool. I'm already 
successfully using it to generate Gnuplot data, and it would be just 
great if I could do it less intrusively so I can exploit these datas 
along with time.

Maybe if the task is not to huge I could try myself to it, for it would 
dramatically ease my life.

Thanks for your reply,
Alex.
-- 
Alexandre Courbot - PhD student
RD2P/LIFL
http://www.lifl.fr/~courbot

Re: Using gdb as a trace agent

[Eli Zaretskii]

> Date: Mon, 17 May 2004 18:47:51 +0200
> From: Alexandre Courbot <Alexandre.Courbot@lifl.fr>
> 
> The ideal for me would be to be able to use tracepoints in both cases 
> (i.e. remote and local). What would prevent a local support for tracepoints?

Nothing, except the fact that the code to do that has not been
written yet.

If you fill that void, you will have my eternal gratitude.

> no chance to get tracepoints to work locally? That would turn GDB
> into a great analysis tool.

This was discussed in the past, and everybody agreed it would be
great.  Now all we need is someone to step forward and code that.

TIA

Re: Using gdb as a trace agent

[Jim Blandy]

"Eli Zaretskii" <eliz@gnu.org> writes:

> > Date: Mon, 17 May 2004 18:47:51 +0200
> > From: Alexandre Courbot <Alexandre.Courbot@lifl.fr>
> > 
> > The ideal for me would be to be able to use tracepoints in both cases 
> > (i.e. remote and local). What would prevent a local support for tracepoints?
> 
> Nothing, except the fact that the code to do that has not been
> written yet.
> 
> If you fill that void, you will have my eternal gratitude.
> 
> > no chance to get tracepoints to work locally? That would turn GDB
> > into a great analysis tool.
> 
> This was discussed in the past, and everybody agreed it would be
> great.  Now all we need is someone to step forward and code that.

Yep.  Here's why it's only been implemented remotely:

On a typical remote embedded system communicating with GDB via a stub,
the stub is installed as a trap handler: when the debuggee hits a
breakpoint, the processor simply transfers control to the stub.  The
stub then chats with GDB for as long as is necessary, and when it
receives a 'continue' packet, it returns from the trap handler.

This means that it's pretty easy to add tracepoints in a low-overhead
way.  The stub just looks up the address of the instruction at which
the trap occurred in a table of tracepoints; if it finds a tracepoint
at that address, it collects the associated data, and continues
immediately.

So for native configurations, the challenge is to find a low-overhead
way to handle those traps.  For example, one could add tracepoint
support to gdbserver, but then you'd have one process collecting data
from another process via ptrace or some other system call, and it
wouldn't be very lightweight.

Re: Using gdb as a trace agent

[Alexandre Courbot]

> Yep.  Here's why it's only been implemented remotely:

Is it possible to use tracepoints remotely as of today? I mean, the code 
seems to be here in GDB, but is there a target that supports them?

Say I have 2 computers running GNU/Linux and a network connection 
between them. Can I then use tracepoints to analyze my program as of today?

> So for native configurations, the challenge is to find a low-overhead
> way to handle those traps.  For example, one could add tracepoint
> support to gdbserver, but then you'd have one process collecting data
> from another process via ptrace or some other system call, and it
> wouldn't be very lightweight.

If gdbserver is running on a different host than gdb, then I guess the 
overhead would be comparable to any other system, right? (just a 
question, I'm just a regular gdb user for now). So maybe the first thing 
to do would be to add tracepoint support to gdbserver, since you seem to 
say it's missing.

On a completely local configuration, the problem is different indeed. 
Maybe if you can evaluate the amount of data to be collected, you can 
reserve a large enough area of memory that would receive all the traced 
datas, without any expression evaluation during collection. Evaluation 
would occur during data examination, so the program gets disturbed as 
less as possible while running. Problem: make sure memory doesn't blow 
up and avoid allocations while debugging. These two reasons are probably 
what makes tracepoints implementation easier on a distributed architecture.

Well, these are just random ideas. As I said, I don't have a clear idea 
yet of gdb's internals.

I'm very interested in that anyway. We need here to be able to evaluate 
our OSes in an easy, non-intrusive and efficient way. GDB has proven it 
was great even for non-debugging purposes (for instance, introspecting a 
program and get some datas out of it to be exploited by gnuplot), with 
tracepoints it would be just *fine*. Oh, and file IOs to write the 
collected datas into files without debuggee and GDB output with them. ;)

Just saw that there has been some discussion about this subject recently:

http://sources.redhat.com/ml/gdb/2004-01/msg00012.html

Seems the ideas were nearly the same. Did it give some results (i.e. code)?

Alex.

-- 
Alexandre Courbot - PhD student
RD2P/LIFL
http://www.lifl.fr/~courbot

Re: Using gdb as a trace agent

[Ramana Radhakrishnan]

Hi Alex,

Alexandre Courbot wrote:

>> Yep.  Here's why it's only been implemented remotely:
>
>
> Is it possible to use tracepoints remotely as of today? I mean, the 
> code seems to be here in GDB, but is there a target that supports them?
>
> Say I have 2 computers running GNU/Linux and a network connection 
> between them. Can I then use tracepoints to analyze my program as of 
> today?

AFAIK no .

>
>> So for native configurations, the challenge is to find a low-overhead
>> way to handle those traps.  For example, one could add tracepoint
>> support to gdbserver, but then you'd have one process collecting data
>> from another process via ptrace or some other system call, and it
>> wouldn't be very lightweight.
>
>
> If gdbserver is running on a different host than gdb, then I guess the 
> overhead would be comparable to any other system, right? (just a 
> question, I'm just a regular gdb user for now). So maybe the first 
> thing to do would be to add tracepoint support to gdbserver, since you 
> seem to say it's missing.
>
> On a completely local configuration, the problem is different indeed. 
> Maybe if you can evaluate the amount of data to be collected, you can 
> reserve a large enough area of memory that would receive all the 
> traced datas, without any expression evaluation during collection. 
> Evaluation would occur during data examination, so the program gets 
> disturbed as less as possible while running. Problem: make sure memory 
> doesn't blow up and avoid allocations while debugging. These two 
> reasons are probably what makes tracepoints implementation easier on a 
> distributed architecture.


>
> Well, these are just random ideas. As I said, I don't have a clear 
> idea yet of gdb's internals.
>
> I'm very interested in that anyway. We need here to be able to 
> evaluate our OSes in an easy, non-intrusive and efficient way. GDB has 
> proven it was great even for non-debugging purposes (for instance, 
> introspecting a program and get some datas out of it to be exploited 
> by gnuplot), with tracepoints it would be just *fine*. Oh, and file 
> IOs to write the collected datas into files without debuggee and GDB 
> output with them. ;)
>
> Just saw that there has been some discussion about this subject recently:
>
> http://sources.redhat.com/ml/gdb/2004-01/msg00012.html
> Seems the ideas were nearly the same. Did it give some results (i.e. 
> code)?

Well we internally have some code which is slightly dated but should not 
be too much effort to integrate into the current CVS tree. (The idea was 
to make gdbserver multithreaded and have a separate thread running which 
would collect the trace data. That was simpler and easier to implement 
in a hurry.It should not be too much of a problem to do asynchronous IO 
with gdbserver too.  ) . So that is the current status with respect to 
the thread that you are referring to . Let me check with the code  we 
have and get back to you. What is the target that you need this for ? 
good old x86?

cheers
Ramana

Re: Using gdb as a trace agent

[Alexandre Courbot]

> Well we internally have some code which is slightly dated but should not 
> be too much effort to integrate into the current CVS tree. (The idea was 
> to make gdbserver multithreaded and have a separate thread running which 
> would collect the trace data. That was simpler and easier to implement 
> in a hurry.It should not be too much of a problem to do asynchronous IO 
> with gdbserver too.  ) . So that is the current status with respect to 
> the thread that you are referring to . Let me check with the code  we 
> have and get back to you. What is the target that you need this for ? 
> good old x86?

At first, yes, and I think your implementation would do the trick. Your 
code sounds interesting to me, and if you make it available I might use 
it as a starting point (even though I'm sure it would perform quite well 
as is).

Thanks,
Alex.
-- 
Alexandre Courbot - PhD student
RD2P/LIFL
http://www.lifl.fr/~courbot