Re: Non-stop multi-threaded debugging

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Hello,
I am collaborating with Ericsson on implementing GDB support in Eclipse 
and I'm very much looking forward to seeing this functionality 
implemented in GDB.   With the Eclipse use of the MI protocol in mind I 
have a couple of comments and questions on the proposal:

> P1) Non-stop multi-threaded native debugging
<snip>
>     - The MI '-exec-interrupt' command shall stop all threads.  This
>       will always generate an 'EXEC-ASYNC-OUTPUT' record, even if all
>       threads were already stopped.  (This helps users handle the case
>       where the thread stops of its own accord just as the user sends
>       it an '-exec-interrupt' command.)
>
>     - The MI '-thread-select' command shall stop the thread selected,
>       if it is running.  The previously selected thread is left in its
>       former state, either stopped or running.  A '-thread-select'
>       command shall always generate an 'EXEC-ASYNC-OUTPUT' record,
>       even if the thread was already stopped.
I think using -thread-select to suspend a thread will have negative 
consequences to the MI-client.  The -thread-select is used to change the 
protocol state before requesting data related to that thread.  Adding 
the side-effect of suspending the thread, will force the client to add 
additional logic to avoid suspending threads accidentally as a result of 
attempting to read data.  I think it would be much safer to use an 
explicit -exec-interrupt request (after a -thread-select) to suspend a 
thread. 
Note that either way GDB will have to handle and return an errors to 
requests for data (such as stack or expressions) while a selected thread 
is running.  If a client sends a -exec-continue, I assume that whatever 
thread was selected last will still be selected.  So a 
-stack-list-frames should result in an error.

> P7) Non-stop multi-threaded multi-process native debugging
<snip>
>     - MI shall provide a command to stop all the threads in a given
>       process, and a command to resume all stopped threads in a given
>       process.

I would like to make a suggestion here.  Instead of inventing a new set 
of parallel commands and events to deal with process-wide events, it 
would be cleaner to simply assign thread IDs to processes and use 
existing commands and events to operate on them.  A -thread-select on an 
ID of a process followed by an -exec-continue would resume an entire 
process, while a -thread-select of a thread's ID followed by a continue 
would resume only that thread.  This could also be applied to all other 
commands that need to operate on a process, such as -thread-list-ids, 
-break-insert, etc.

While events could also be re-used to report without any changes to the 
event format, it would be very helpful for the client if the events 
included both the process's and thread's thread ID, as well as whether 
all threads stopped as the result of the event:
...
-thread-select 2
^done
-exec-interrupt
*stopped, thread-id="2", process-thread-id="1", all-threads="false", ....
^done
...
-thread-select 1
^done
-exec-continue
^running, thread-id="1", process-thread-id="1", all-threads="true"
...

Cheers,
Pawel

Re: Non-stop multi-threaded debugging

[Nick Roberts]

 >                                           ... A -thread-select on an 
 > ID of a process followed by an -exec-continue would resume an entire 
 > process, while a -thread-select of a thread's ID followed by a continue 
 > would resume only that thread.  This could also be applied to all other 
 > commands that need to operate on a process, such as -thread-list-ids, 
 > -break-insert, etc.

This would change the current behaviour of these commands.  If a new command
is undesirable then perhaps optional parameters could be used:

     -exec-continue [ -p THREAD-ID/PROCESS-ID ]
     -exec-interrupt [ -p THREAD-ID/PROCESS-ID ]

It appears that -break-insert already has such an option for threads.

-- 
Nick                                           http://www.inet.net.nz/~nickrob

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Nick Roberts wrote:
>  >                                           ... A -thread-select on an 
>  > ID of a process followed by an -exec-continue would resume an entire 
>  > process, while a -thread-select of a thread's ID followed by a continue 
>  > would resume only that thread.  This could also be applied to all other 
>  > commands that need to operate on a process, such as -thread-list-ids, 
>  > -break-insert, etc.
>
> This would change the current behaviour of these commands.  If a new command
> is undesirable then perhaps optional parameters could be used:
>
>      -exec-continue [ -p THREAD-ID/PROCESS-ID ]
>      -exec-interrupt [ -p THREAD-ID/PROCESS-ID ]
>
> It appears that -break-insert already has such an option for threads.
>
>   
 From Eclipse's point of view it actually doesn't make much difference 
whether -thread-select or -p option is used to specify the thread. 

That said, I would argue that adding the non-stop debugging feature 
changes the behavior of the entire system, so it could be expected that 
some commands will behave somewhat differently as they relate to this 
new feature.  Actually, with non-stop debugging feature turned off, and 
without attaching to multiple processes, these commands would still 
behave exactly as they do now. 

In fact, the original post states:
>     - Execution commands like '-exec-continue' and '-exec-step' shall
>       resume only the selected thread, without affecting the other
>       threads in the process.
So it seems that Jim's proposal would already change the existing 
behavior of -exec-continue and -exec-step commands.


Also:
> At the user interface level:
>
>     - MI shall provide a command to stop all the threads in a given
>       process, and a command to resume all stopped threads in a given
>       process.
Which to me implies that -exec-continue and -exec-interrupt would act on 
a single thread, although this is contradicted by the other statement 
about -exec-interrupt.


Lastly, since my post I thought a little about the CLI aspect in my 
suggested use of thread IDs.  It would probably be a little awkward to 
use the "thread <id>" command to switch to a different process, or to 
use it with a process-id to initiate process-wide operations.  So for 
the CLI interface, while I still think it would make sense to use the 
same ID name-space for processes and threads, it would help to add a 
couple of new commands:

"process <id>" - To select a new active process where the active thread 
would be the last active thread in this process.  This would be the 
equivalent of a -thread-select with the last active thread in that process.

"thread all" - To un-select the current thread in order to perform 
operations on all threads in the process.  Equivalent of -thread-select 
with the ID of the process.

Cheers,
Pawel

gdb over RNDIS to PDA

[Steve DeLaney]

 
anyone doing anything with gdb over RNDIS?  
I'm working on a windows mobile project using cegcc and am trying to get
gdbserver to work
the PDA is sync'd to linux FC6 with RNDIS/USB using SynCE

details posted in cegcc-devel@lists.sourceforge.net 
topic "mingw32ce gdb to wm2005 over RNDIS"

thanks!

/steverino2

Re: Non-stop multi-threaded debugging

[Vladimir Prus]

Pawel Piech wrote:

> Nick Roberts wrote:
>>  >                                           ... A -thread-select on an
>>  > ID of a process followed by an -exec-continue would resume an entire
>>  > process, while a -thread-select of a thread's ID followed by a
>>  > continue
>>  > would resume only that thread.  This could also be applied to all
>>  > other commands that need to operate on a process, such as
>>  > -thread-list-ids, -break-insert, etc.
>>
>> This would change the current behaviour of these commands.  If a new
>> command is undesirable then perhaps optional parameters could be used:
>>
>>      -exec-continue [ -p THREAD-ID/PROCESS-ID ]
>>      -exec-interrupt [ -p THREAD-ID/PROCESS-ID ]
>>
>> It appears that -break-insert already has such an option for threads.
>>
>>   
>  From Eclipse's point of view it actually doesn't make much difference
> whether -thread-select or -p option is used to specify the thread.
> 
> That said, I would argue that adding the non-stop debugging feature
> changes the behavior of the entire system, so it could be expected that
> some commands will behave somewhat differently as they relate to this
> new feature.  Actually, with non-stop debugging feature turned off, and
> without attaching to multiple processes, these commands would still
> behave exactly as they do now.

Given the choice between:

1. Changing the behaviour of the existing command, and adding new one
that behaves like existing one, and

2. Adding new command

I think adding new command (or option to existing command), is a smaller change.
So yes, -exec-continue -t <xxx> might be a better choice.

- Volodya

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Vladimir Prus wrote:
> Pawel Piech wrote:
>
>   
>> Nick Roberts wrote:
>>     
>>>  >                                           ... A -thread-select on an
>>>  > ID of a process followed by an -exec-continue would resume an entire
>>>  > process, while a -thread-select of a thread's ID followed by a
>>>  > continue
>>>  > would resume only that thread.  This could also be applied to all
>>>  > other commands that need to operate on a process, such as
>>>  > -thread-list-ids, -break-insert, etc.
>>>
>>> This would change the current behaviour of these commands.  If a new
>>> command is undesirable then perhaps optional parameters could be used:
>>>
>>>      -exec-continue [ -p THREAD-ID/PROCESS-ID ]
>>>      -exec-interrupt [ -p THREAD-ID/PROCESS-ID ]
>>>
>>> It appears that -break-insert already has such an option for threads.
>>>
>>>   
>>>       
>>  From Eclipse's point of view it actually doesn't make much difference
>> whether -thread-select or -p option is used to specify the thread.
>>
>> That said, I would argue that adding the non-stop debugging feature
>> changes the behavior of the entire system, so it could be expected that
>> some commands will behave somewhat differently as they relate to this
>> new feature.  Actually, with non-stop debugging feature turned off, and
>> without attaching to multiple processes, these commands would still
>> behave exactly as they do now.
>>     
>
> Given the choice between:
>
> 1. Changing the behaviour of the existing command, and adding new one
> that behaves like existing one, and
>
> 2. Adding new command
>
> I think adding new command (or option to existing command), is a smaller change.
> So yes, -exec-continue -t <xxx> might be a better choice.
>
> - Volodya
>   
I don't believe that I suggested adding any new command.  Instead, I 
suggested only changing the behavior of some of the existing commands to 
use the currently selected thread.  It was Jim's proposal that included 
adding new commands.  My point was that instead of adding new commands 
it would be cleaner to extend the functionality of -thread-select in 
order to select a process context, i.e. a context that includes all 
threads. This way existing commands, which currently can only operate on 
a global context, could now operate on a process or a thread context.

Cheers,
Pawel

Re: Non-stop multi-threaded debugging

[Vladimir Prus]

On Thursday 29 November 2007 19:39:18 Pawel Piech wrote:

>  I don't believe that I suggested adding any new command.
>  Instead, I suggested only changing the behavior of 
>  some of the existing commands to use the currently selected thread.
>  It was Jim's proposal that included adding new commands.
>  My point was that instead of adding new commands it would be cleaner
>  to extend the functionality of -thread-select in order to select 
>  a process context, i.e. a context that includes all threads. 
>  This way existing commands, which currently can only operate
>  on a global context, could now operate on a process or a thread context.        

Ok, we have those choices:

1. Make -exec-continue work on all threads, until -thread-select is used.
2. Make -exec-continue work on one thread, and add another command
to operate on all threads.
3. Make -exec-continue still operate on all threads, unless
and explicit option to make it operate on a thread is given.

You've indicated that (1) and (3) are about the same in complexity for you --
am I right? I personally prefer (3), since it does not implicitly changes
the meaning of existing commands.

Surely, non-stop mode does require some changes in frontend, but the
fewer changes are, the better, IMO.

- Volodya

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Vladimir Prus wrote:
> On Thursday 29 November 2007 19:39:18 Pawel Piech wrote:
>
>   
>>  I don't believe that I suggested adding any new command.
>>  Instead, I suggested only changing the behavior of 
>>  some of the existing commands to use the currently selected thread.
>>  It was Jim's proposal that included adding new commands.
>>  My point was that instead of adding new commands it would be cleaner
>>  to extend the functionality of -thread-select in order to select 
>>  a process context, i.e. a context that includes all threads. 
>>  This way existing commands, which currently can only operate
>>  on a global context, could now operate on a process or a thread context.        
>>     
>
> Ok, we have those choices:
>
> 1. Make -exec-continue work on all threads, until -thread-select is used.
> 2. Make -exec-continue work on one thread, and add another command
> to operate on all threads.
> 3. Make -exec-continue still operate on all threads, unless
> and explicit option to make it operate on a thread is given.
>
> You've indicated that (1) and (3) are about the same in complexity for you --
> am I right? 
This is correct.
> I personally prefer (3), since it does not implicitly changes
> the meaning of existing commands.
>
> Surely, non-stop mode does require some changes in frontend, but the
> fewer changes are, the better, IMO.
>
> - Volodya
>   

In that case I'll try to convince you otherwise :-)

-exec-continue is not the only command that would need to be modified.  
-exec-interrupt, would all need to take the -p parameter, and in order 
to implement multi-process debugging, many of the commands that 
currently operate on a global context (too many to try to list) would 
all require an additional parameter to specify which process they are to 
act on.  There seems to be a well established paradigm in the MI (and 
CLI) protocol, where special commands: -thread-select and 
-stack-frame-select change the state of the protocol so that commands 
following these operate on the context selected by these commands.  My 
main point is to extend the functionality of these state-changing 
commands in order to add the ability to select an active context, and to 
select a context which will allow commands to operate on all the threads 
of a process. 

IMO, the question of whether -exec-continue takes a -p argument is a 
rather minor one.  But for sake of consistency with other -exec-* 
commands I think it would be a mistake to add this parameter.  That's 
because the stepping commands already do operate on the currently 
selected thread.  While all the threads are resumed when stepping, 
execution does not stop until the next line of code is reached by the 
thread that was selected.  With non-stop debugging, stepping commands 
will continue to operate on the selected thread with the difference that 
other suspended threads will remain suspended.  It seems logical to me 
that -exec-continue and -exec-interrupt should operate in the same way.  
In the degenerate case where the target is not capable non-stop 
multi-threaded debugging, conceptually these commands can still be 
thought of as operating on the selected thread, with the side effect 
that when one thread is resumed/suspended, all threads are 
resumed/suspended as well.

Cheers,
Pawel

Re: Non-stop multi-threaded debugging

[Vladimir Prus]

On Thursday 29 November 2007 20:36:18 Pawel Piech wrote:
> Vladimir Prus wrote:
> > On Thursday 29 November 2007 19:39:18 Pawel Piech wrote:
> >
> >   
> >>  I don't believe that I suggested adding any new command.
> >>  Instead, I suggested only changing the behavior of 
> >>  some of the existing commands to use the currently selected thread.
> >>  It was Jim's proposal that included adding new commands.
> >>  My point was that instead of adding new commands it would be cleaner
> >>  to extend the functionality of -thread-select in order to select 
> >>  a process context, i.e. a context that includes all threads. 
> >>  This way existing commands, which currently can only operate
> >>  on a global context, could now operate on a process or a thread context.        
> >>     
> >
> > Ok, we have those choices:
> >
> > 1. Make -exec-continue work on all threads, until -thread-select is used.
> > 2. Make -exec-continue work on one thread, and add another command
> > to operate on all threads.
> > 3. Make -exec-continue still operate on all threads, unless
> > and explicit option to make it operate on a thread is given.
> >
> > You've indicated that (1) and (3) are about the same in complexity for you --
> > am I right? 
> This is correct.
> > I personally prefer (3), since it does not implicitly changes
> > the meaning of existing commands.
> >
> > Surely, non-stop mode does require some changes in frontend, but the
> > fewer changes are, the better, IMO.
> >
> > - Volodya
> >   
> 
> In that case I'll try to convince you otherwise :-)
> 
> -exec-continue is not the only command that would need to be modified.  
> -exec-interrupt, would all need to take the -p parameter, and in order 
> to implement multi-process debugging, many of the commands that 
> currently operate on a global context (too many to try to list) would 
> all require an additional parameter to specify which process they are to 
> act on.  There seems to be a well established paradigm in the MI (and 
> CLI) protocol, where special commands: -thread-select and 
> -stack-frame-select change the state of the protocol so that commands 
> following these operate on the context selected by these commands.  

There are different opinions about those commands ;-) For example, I believe
that stateless protocol would be much easier.

> My  
> main point is to extend the functionality of these state-changing 
> commands in order to add the ability to select an active context, and to 
> select a context which will allow commands to operate on all the threads 
> of a process. 
> 
> IMO, the question of whether -exec-continue takes a -p argument is a 
> rather minor one.  But for sake of consistency with other -exec-* 
> commands I think it would be a mistake to add this parameter.  That's 
> because the stepping commands already do operate on the currently 
> selected thread.  While all the threads are resumed when stepping, 
> execution does not stop until the next line of code is reached by the 
> thread that was selected.  With non-stop debugging, stepping commands 
> will continue to operate on the selected thread with the difference that 
> other suspended threads will remain suspended.  

Interesting. Presently, in CLI gdb set 'scheduler mode' that controls
if step resumes all thread, or not. MI does not have a counterpart of that.
Maybe, we should start by changing -exec-step to accept a thread parameter, too,
which would mean "step only this thread, keep others suspended". Then,
-exec-continue taking thread parameter will be quite consistent with -exec-step.
I would say such change to -exec-step will be valuable in itself.

- Volodya

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Vladimir Prus wrote:
> On Thursday 29 November 2007 20:36:18 Pawel Piech wrote:
>   
>> Vladimir Prus wrote:
>>     
>>> On Thursday 29 November 2007 19:39:18 Pawel Piech wrote:
>>>
>>>   
>>>       
>>>>  I don't believe that I suggested adding any new command.
>>>>  Instead, I suggested only changing the behavior of 
>>>>  some of the existing commands to use the currently selected thread.
>>>>  It was Jim's proposal that included adding new commands.
>>>>  My point was that instead of adding new commands it would be cleaner
>>>>  to extend the functionality of -thread-select in order to select 
>>>>  a process context, i.e. a context that includes all threads. 
>>>>  This way existing commands, which currently can only operate
>>>>  on a global context, could now operate on a process or a thread context.        
>>>>     
>>>>         
>>> Ok, we have those choices:
>>>
>>> 1. Make -exec-continue work on all threads, until -thread-select is used.
>>> 2. Make -exec-continue work on one thread, and add another command
>>> to operate on all threads.
>>> 3. Make -exec-continue still operate on all threads, unless
>>> and explicit option to make it operate on a thread is given.
>>>
>>> You've indicated that (1) and (3) are about the same in complexity for you --
>>> am I right? 
>>>       
>> This is correct.
>>     
>>> I personally prefer (3), since it does not implicitly changes
>>> the meaning of existing commands.
>>>
>>> Surely, non-stop mode does require some changes in frontend, but the
>>> fewer changes are, the better, IMO.
>>>
>>> - Volodya
>>>   
>>>       
>> In that case I'll try to convince you otherwise :-)
>>
>> -exec-continue is not the only command that would need to be modified.  
>> -exec-interrupt, would all need to take the -p parameter, and in order 
>> to implement multi-process debugging, many of the commands that 
>> currently operate on a global context (too many to try to list) would 
>> all require an additional parameter to specify which process they are to 
>> act on.  There seems to be a well established paradigm in the MI (and 
>> CLI) protocol, where special commands: -thread-select and 
>> -stack-frame-select change the state of the protocol so that commands 
>> following these operate on the context selected by these commands.  
>>     
>
> There are different opinions about those commands ;-) For example, I believe
> that stateless protocol would be much easier.
>
>   
I completely agree with you, for a "machine interface" a stateless 
protocol would be much easier to use.  But GDB/MI is what it is and at 
this point trying to make it stateless would probably be counter productive.
>> My  
>> main point is to extend the functionality of these state-changing 
>> commands in order to add the ability to select an active context, and to 
>> select a context which will allow commands to operate on all the threads 
>> of a process. 
>>
>> IMO, the question of whether -exec-continue takes a -p argument is a 
>> rather minor one.  But for sake of consistency with other -exec-* 
>> commands I think it would be a mistake to add this parameter.  That's 
>> because the stepping commands already do operate on the currently 
>> selected thread.  While all the threads are resumed when stepping, 
>> execution does not stop until the next line of code is reached by the 
>> thread that was selected.  With non-stop debugging, stepping commands 
>> will continue to operate on the selected thread with the difference that 
>> other suspended threads will remain suspended.  
>>     
>
> Interesting. Presently, in CLI gdb set 'scheduler mode' that controls
> if step resumes all thread, or not. MI does not have a counterpart of that.
> Maybe, we should start by changing -exec-step to accept a thread parameter, too,
> which would mean "step only this thread, keep others suspended". Then,
> -exec-continue taking thread parameter will be quite consistent with -exec-step.
> I would say such change to -exec-step will be valuable in itself.
>
> - Volodya
>   
I'm rather confused here.  I thought that the point of non-stop 
multi-threaded debugging was to allow for this behavior.  Is "scheduler 
mode" some kind of partial implementation?  Either way though, I have no 
opinion whether it would be useful for the command line user to be able 
to specify the scheduling mode as an option to individual commands, but 
in an IDE such behavior would most likely be controlled with a toggle 
button, so a mode switch command would be sufficient.  If -exec-step 
would take a -p parameter to determine whether to resume all threads, 
then -exec-step without -p would still operate on the selected thread, 
in that it would suspend the process only when the selected thread 
reached the next line.  IHMO, this would only add to the confusion.

Cheers,
Pawel

Re: Non-stop multi-threaded debugging

[Jim Blandy]

Folks should treat this post:

  http://sourceware.org/ml/gdb/2007-11/msg00198.html

as an attempt to mention all the relevant issues --- not as a fully
worked-out design.  The plan was to hash out the details through
discussion amongst the front end authors on this list.  I don't have
experience writing clients for MI; I've only dealt with implementing
it in GDB, and even then not much.  So you people who work on
real-life front ends are in a much better position to see what would
work best.

So I encourage you, Vlad, Nick, and the rest of the MI fans to hash it
all out and come up with something everyone is satisfied with.  We
just need to make sure that the issues the analysis raises all get
addressed (or dismissed).

Re: Non-stop multi-threaded debugging

[Pawel Piech]

Hi Jim,
I'm glad to hear that you're receptive to feedback in defining the 
protocol for these new features.  I will be glad to write up my proposal 
as a starting point for discussion.  I've never participated on this 
list before or in any other GDB forum, so forgive my ignorance, but is 
this mailing list the appropriate place for this discussion?  Is there a 
bugzilla or other bug database where I could post this document?  And 
what is the best format for such design document.

Thanks
Pawel

Jim Blandy wrote:
> Folks should treat this post:
>
>   http://sourceware.org/ml/gdb/2007-11/msg00198.html
>
> as an attempt to mention all the relevant issues --- not as a fully
> worked-out design.  The plan was to hash out the details through
> discussion amongst the front end authors on this list.  I don't have
> experience writing clients for MI; I've only dealt with implementing
> it in GDB, and even then not much.  So you people who work on
> real-life front ends are in a much better position to see what would
> work best.
>
> So I encourage you, Vlad, Nick, and the rest of the MI fans to hash it
> all out and come up with something everyone is satisfied with.  We
> just need to make sure that the issues the analysis raises all get
> addressed (or dismissed).
>   

Re: Non-stop multi-threaded debugging

[Jim Blandy]

Pawel Piech <pawel.piech at windriver.com> writes:
> I'm glad to hear that you're receptive to feedback in defining the
> protocol for these new features.  I will be glad to write up my
> proposal as a starting point for discussion.

That'd be great!

> I've never participated
> on this list before or in any other GDB forum, so forgive my
> ignorance, but is this mailing list the appropriate place for this
> discussion?  Is there a bugzilla or other bug database where I could
> post this document?  And what is the best format for such design
> document.

This is the right list for general discussion like this.  If one has a
patch for discussion or review, that belongs on gdb-patches.

We haven't made a practice of putting documents in an issue tracker; I
don't think anyone has proposed it.  We just link to the archive, as I
did in the message you replied to.

The GDB documentation is written in texinfo.  When I've written draft
documentation for discussion, I've written it in texinfo, and then
posted the output from 'makeinfo --plaintext' for review and
discussion.  But if it's not intended to go into a manual, then plain
text is what we usually use.