Single stepping and threads

Single stepping and threads

[Daniel Jacobowitz]

Ulrich's message earlier reminded me of something I've been meaning to
discuss for a while.  This isn't specific to software single stepping,
but to single step in general for threaded programs.

We have a knob "set scheduler-locking".  It offers three values:

Set mode for locking scheduler during execution.
off  == no locking (threads may preempt at any time)
on   == full locking (no thread except the current thread may run)
step == scheduler locked during every single-step operation.
        In this mode, no other thread may run during a step command.
        Other threads may run while stepping over a function call
        ('next').

The default is "off".  Should it be "step" instead?  The example I used
to use whenever someone asked me about this was single stepping through
something like a barrier or mutex; if other threads don't run, you
won't advance, because no other thread will have a chance to release
the lock.  That much is true.  But it seems like a reasonable thing to
document and reference "set scheduler-locking".  And having threads
run during single stepping has surprised a lot of users who've asked
me about the current behavior.

What do you all think?

One reason I've procrastinated bringing this up is that set
scheduler-locking off, the current default, has a lot more nasty
corner cases that I've meant to look into; if step becomes the default,
I suspect more of those will linger unfixed.  But users won't encounter
them as often, which is much like fixing them :-)

A related issue is the tendency of "step" to let other threads run even
in "set scheduler-locking step".  For instance:

  - We use a breakpoint to skip the prologue of a function when we step
    into it.  This could either be implemented with a stepping range
    instead, or else we could continue to use the breakpoint but honor
    the scheduler locking mode anyway, but the current behavior is
    silly.

  - "step" acts like "next" when stepping over a function without debug
    info.  Should we honor "set scheduler-locking step" when doing
    this?

-- 
Daniel Jacobowitz
CodeSourcery

Re: Single stepping and threads

[Joel Brobecker]

> The default is "off".  Should it be "step" instead?

I am not sure. On the one hand, I personally think that the current
behavior makes it harder to debug the program. I've been fustrated
a few times in the past when my next/step command got interrupted
and I found myself in another thread -> I kept using breakpoints
to do what I needed to do.

On the other hand, this has been the default behavior for so long
that it's something you expect...  I don't know how much this will
impact the users. At AdaCore, I don't remember receiving any report
of that sort from our users, and tasking in Ada is something that's
commonly used.

The other argument against is that the debugger is supposed to be
transparent to the execution. This is not 100% true, but I think
we are very very close to it. Except maybe with certain thread
layers? Not sure, this is not my area of expertise, to say the
least. But for sure changing the setting to either "on" or "step"
will cause the debugger to affect the scheduling of the inferior.

So, to summarize, I'm somewhat in favor. I'll poll my team-mates
and see if they have any interesting ideas to share on the subject.

> One reason I've procrastinated bringing this up is that set
> scheduler-locking off, the current default, has a lot more nasty
> corner cases that I've meant to look into; if step becomes the default,
> I suspect more of those will linger unfixed.  But users won't encounter
> them as often, which is much like fixing them :-)

I agree with that. If we decide to make that change, and that hides
issues as a result, then these particular issues become less important,
and you can spend the time working on other things that you like.

> A related issue is the tendency of "step" to let other threads run even
> in "set scheduler-locking step".  For instance:
[...]
>   - "step" acts like "next" when stepping over a function without debug
>     info.  Should we honor "set scheduler-locking step" when doing
>     this?

I would say yes. A step should be a few instructions, while stepping
over a call is potentially a much larger number of instructions.
As a result, stepping over without letting the other threads go would
more likely cause a lock.

-- 
Joel

PS: My understanding is that not all systems support the running
    of an individual thread instead of the entire program. Is that
    right? Or do all systems support this feature?

Re: Single stepping and threads

[Mark Kettenis]

>  Ulrich's message earlier reminded me of something I've been meaning to
>  discuss for a while.  This isn't specific to software single stepping,
>  but to single step in general for threaded programs.
>
>  We have a knob "set scheduler-locking".  It offers three values:
>
>  Set mode for locking scheduler during execution.
>  off  == no locking (threads may preempt at any time)
>  on   == full locking (no thread except the current thread may run)
>  step == scheduler locked during every single-step operation.
>          In this mode, no other thread may run during a step command.
>          Other threads may run while stepping over a function call
>          ('next').
>
>  The default is "off".  Should it be "step" instead?  The example I used
>  to use whenever someone asked me about this was single stepping through
>  something like a barrier or mutex; if other threads don't run, you
>  won't advance, because no other thread will have a chance to release
>  the lock.  That much is true.  But it seems like a reasonable thing to
>  document and reference "set scheduler-locking".  And having threads
>  run during single stepping has surprised a lot of users who've asked
>  me about the current behavior.
>
>  What do you all think?

Are you talking about stepi or step?
For step we should be very careful, since on some platforms the dynamic
linker may play games with locks and we risk a deadlock if we don't let
the other threads run.

Re: Single stepping and threads

[Daniel Jacobowitz]

On Tue, Nov 28, 2006 at 09:59:15PM -0800, Joel Brobecker wrote:
> > A related issue is the tendency of "step" to let other threads run even
> > in "set scheduler-locking step".  For instance:
> [...]
> >   - "step" acts like "next" when stepping over a function without debug
> >     info.  Should we honor "set scheduler-locking step" when doing
> >     this?
> 
> I would say yes. A step should be a few instructions, while stepping
> over a call is potentially a much larger number of instructions.
> As a result, stepping over without letting the other threads go would
> more likely cause a lock.

I think you mean "no" then?

> PS: My understanding is that not all systems support the running
>     of an individual thread instead of the entire program. Is that
>     right? Or do all systems support this feature?

I'm really not sure.  I assume there were systems that didn't support
it when it was added; there are probably some still, but I don't know
any personally.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Single stepping and threads

[Daniel Jacobowitz]

On Wed, Nov 29, 2006 at 01:40:49PM +0100, Mark Kettenis wrote:
> Are you talking about stepi or step?
> For step we should be very careful, since on some platforms the dynamic
> linker may play games with locks and we risk a deadlock if we don't let
> the other threads run.

That's the second part of the question :-)  The current setting of
"set scheduler-locking" applies to whenever proceed() is told to step,
rather than when the user says "step".  So it affects stepi always and
step within a source line, but not step over a function.  I talked
about "stepping" over prologues and over functions without debug info;
I hadn't thought about skipping shared library trampolines, but I
can definitely see how that would be a problem.

I don't like the user interface here; we say "step" and it will
sometimes run one thread and sometimes run all threads and it's hard to
explain which.  But we may not be able to do better without adding
unacceptable risk of deadlocks :-(

-- 
Daniel Jacobowitz
CodeSourcery

Re: Single stepping and threads

[Joel Brobecker]

> > I would say yes. A step should be a few instructions, while stepping
> > over a call is potentially a much larger number of instructions.
> > As a result, stepping over without letting the other threads go would
> > more likely cause a lock.
> 
> I think you mean "no" then?

Oops, sorry, I meant "no".

One of my coworkers expressed his opinion as follow:

<<
I would find it confusing if "step" and "next" behave differently with
respect to threads, because they seem like basically the same thing.
"Next is just like step, except that it goes over calls" seems simple to
me. "Next is just like step, except that it goes over calls, and has
some subtle difference regarding threads" seems more complicated to me.

So I would suggest leaving the default as "off", or else changing it
to "on".
>>

-- 
Joel

Re: Single stepping and threads

[Robert Dewar]

Daniel Jacobowitz wrote:

> I'm really not sure.  I assume there were systems that didn't support
> it when it was added; there are probably some still, but I don't know
> any personally.

There certainly are systems that do not support this. In any case I
think it would be a very bad idea to change the default.

Re: Single stepping and threads

[Daniel Jacobowitz]

On Wed, Nov 29, 2006 at 08:38:44AM -0800, Joel Brobecker wrote:
> One of my coworkers expressed his opinion as follow:
> 
> <<
> I would find it confusing if "step" and "next" behave differently with
> respect to threads, because they seem like basically the same thing.
> "Next is just like step, except that it goes over calls" seems simple to
> me. "Next is just like step, except that it goes over calls, and has
> some subtle difference regarding threads" seems more complicated to me.
> 
> So I would suggest leaving the default as "off", or else changing it
> to "on".
> >>

Fortunately, step and next would not behave differently.  It would
actually be more like this:

  When GDB advances the program by a small amount, a single instruction
  at a time, only the current thread runs.  When GDB advances the
  program by a large amount using a breakpoint, for instance to skip
  a function or a dynamic linker trampoline, other threads are allowed
  to run.

On Thu, Nov 30, 2006 at 03:44:14AM -0500, Robert Dewar wrote:
> There certainly are systems that do not support this. In any case I
> think it would be a very bad idea to change the default.

Why?  I am looking for reasons beyond inertia.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Single stepping and threads

[Michael Snyder]

On Wed, 2006-11-29 at 00:29 -0500, Daniel Jacobowitz wrote:
> Ulrich's message earlier reminded me of something I've been meaning to
> discuss for a while.  This isn't specific to software single stepping,
> but to single step in general for threaded programs.
> 
> We have a knob "set scheduler-locking".  It offers three values:
> 
> Set mode for locking scheduler during execution.
> off  == no locking (threads may preempt at any time)
> on   == full locking (no thread except the current thread may run)
> step == scheduler locked during every single-step operation.
>         In this mode, no other thread may run during a step command.
>         Other threads may run while stepping over a function call
>         ('next').
> 
> The default is "off".  Should it be "step" instead?  The example I used
> to use whenever someone asked me about this was single stepping through
> something like a barrier or mutex; if other threads don't run, you
> won't advance, because no other thread will have a chance to release
> the lock.  That much is true.  But it seems like a reasonable thing to
> document and reference "set scheduler-locking".  And having threads
> run during single stepping has surprised a lot of users who've asked
> me about the current behavior.
> 
> What do you all think?

I implemented "set scheduler-locking", at user request.

The behavior that the users were requesting is "step".

I left the default at "off", because that was gdb's 
original behavior, and I didn't want to change it 
out from under people without notice.

It may very well be time to change the default.

> 
> One reason I've procrastinated bringing this up is that set
> scheduler-locking off, the current default, has a lot more nasty
> corner cases that I've meant to look into; if step becomes the default,
> I suspect more of those will linger unfixed.  But users won't encounter
> them as often, which is much like fixing them :-)
> 
> A related issue is the tendency of "step" to let other threads run even
> in "set scheduler-locking step".  For instance:
> 
>   - We use a breakpoint to skip the prologue of a function when we step
>     into it.  This could either be implemented with a stepping range
>     instead, or else we could continue to use the breakpoint but honor
>     the scheduler locking mode anyway, but the current behavior is
>     silly.
> 
>   - "step" acts like "next" when stepping over a function without debug
>     info.  Should we honor "set scheduler-locking step" when doing
>     this?
> 

Re: Single stepping and threads

[Michael Snyder]

On Tue, 2006-11-28 at 21:59 -0800, Joel Brobecker wrote:
> > The default is "off".  Should it be "step" instead?
> 
> I am not sure. On the one hand, I personally think that the current
> behavior makes it harder to debug the program. I've been fustrated
> a few times in the past when my next/step command got interrupted
> and I found myself in another thread -> I kept using breakpoints
> to do what I needed to do.
> 
> On the other hand, this has been the default behavior for so long
> that it's something you expect...  I don't know how much this will
> impact the users. At AdaCore, I don't remember receiving any report
> of that sort from our users, and tasking in Ada is something that's
> commonly used.
> 
> The other argument against is that the debugger is supposed to be
> transparent to the execution. This is not 100% true, but I think
> we are very very close to it. Except maybe with certain thread
> layers? Not sure, this is not my area of expertise, to say the
> least. But for sure changing the setting to either "on" or "step"
> will cause the debugger to affect the scheduling of the inferior.

That's correct.  GDB is unavoidably intrusive when it comes to
thread scheduling.  The default behavior ("off"), is the least
intrusive, and somewhat randomly, unpredictably intrusive.

"Step" would be more intrusive, and non-randomly so.  It will
cause the "current" thread to be strongly preferred, and other
threads will probably starve.

"On" is the most intrusive -- no other threads can run, and
if there are any synchronization issues, the program will 
eventually deadlock.


> 
> So, to summarize, I'm somewhat in favor. I'll poll my team-mates
> and see if they have any interesting ideas to share on the subject.
> 
> > One reason I've procrastinated bringing this up is that set
> > scheduler-locking off, the current default, has a lot more nasty
> > corner cases that I've meant to look into; if step becomes the default,
> > I suspect more of those will linger unfixed.  But users won't encounter
> > them as often, which is much like fixing them :-)
> 
> I agree with that. If we decide to make that change, and that hides
> issues as a result, then these particular issues become less important,
> and you can spend the time working on other things that you like.
> 
> > A related issue is the tendency of "step" to let other threads run even
> > in "set scheduler-locking step".  For instance:
> [...]
> >   - "step" acts like "next" when stepping over a function without debug
> >     info.  Should we honor "set scheduler-locking step" when doing
> >     this?
> 
> I would say yes. A step should be a few instructions, while stepping
> over a call is potentially a much larger number of instructions.
> As a result, stepping over without letting the other threads go would
> more likely cause a lock.
> 

Re: Single stepping and threads

[Joel Brobecker]

> That's correct.  GDB is unavoidably intrusive when it comes to
> thread scheduling.  The default behavior ("off"), is the least
> intrusive, and somewhat randomly, unpredictably intrusive.
> 
> "Step" would be more intrusive, and non-randomly so.  It will
> cause the "current" thread to be strongly preferred, and other
> threads will probably starve.
> 
> "On" is the most intrusive -- no other threads can run, and
> if there are any synchronization issues, the program will 
> eventually deadlock.

I personally am a big fan of "on", btw. Your points make me like
it even more...

-- 
Joel

Re: Single stepping and threads

[Michael Snyder]

On Wed, 2006-11-29 at 08:25 -0500, Daniel Jacobowitz wrote:
> On Tue, Nov 28, 2006 at 09:59:15PM -0800, Joel Brobecker wrote:
> > > A related issue is the tendency of "step" to let other threads run even
> > > in "set scheduler-locking step".  For instance:
> > [...]
> > >   - "step" acts like "next" when stepping over a function without debug
> > >     info.  Should we honor "set scheduler-locking step" when doing
> > >     this?
> > 
> > I would say yes. A step should be a few instructions, while stepping
> > over a call is potentially a much larger number of instructions.
> > As a result, stepping over without letting the other threads go would
> > more likely cause a lock.
> 
> I think you mean "no" then?
> 
> > PS: My understanding is that not all systems support the running
> >     of an individual thread instead of the entire program. Is that
> >     right? Or do all systems support this feature?
> 
> I'm really not sure. 

Not all systems support the "set scheduler-locking" command.
There is a target-vector element that tells gdb whether the
command is implemented or not.

If it isn't implemented, failure will be "clean".

>  I assume there were systems that didn't support
> it when it was added; there are probably some still, but I don't know
> any personally.

grep for tc_schedlock, which is only mentioned in linux-nat.c, 
procfs.c, and remote.c.  I believe remote.c probes the target
to find out if it can support schedlock.

Re: Single stepping and threads

[Michael Snyder]

On Wed, 2006-11-29 at 08:38 -0800, Joel Brobecker wrote:
> > > I would say yes. A step should be a few instructions, while stepping
> > > over a call is potentially a much larger number of instructions.
> > > As a result, stepping over without letting the other threads go would
> > > more likely cause a lock.
> > 
> > I think you mean "no" then?
> 
> Oops, sorry, I meant "no".
> 
> One of my coworkers expressed his opinion as follow:
> 
> <<
> I would find it confusing if "step" and "next" behave differently with
> respect to threads, because they seem like basically the same thing.
> "Next is just like step, except that it goes over calls" seems simple to
> me. "Next is just like step, except that it goes over calls, and has
> some subtle difference regarding threads" seems more complicated to me.
> 
> So I would suggest leaving the default as "off", or else changing it
> to "on".

Default on would be a disaster -- most threaded programs would
not behave even remotely the same under the debugger as they would
solo.

In fact, many would deadlock almost immediately.

Re: Single stepping and threads

[Michael Snyder]

On Wed, 2006-11-29 at 08:36 -0500, Daniel Jacobowitz wrote:
> On Wed, Nov 29, 2006 at 01:40:49PM +0100, Mark Kettenis wrote:
> > Are you talking about stepi or step?
> > For step we should be very careful, since on some platforms the dynamic
> > linker may play games with locks and we risk a deadlock if we don't let
> > the other threads run.
> 
> That's the second part of the question :-)  The current setting of
> "set scheduler-locking" applies to whenever proceed() is told to step,
> rather than when the user says "step".  So it affects stepi always and
> step within a source line, but not step over a function. 

That was intentional -- a compromise between more stringent locking, 
and the tendency to deadlock.

The idea was, "a function call is too long to keep the scheduler
locked, unles you insist on locking it down completely (schedlock on)".


>  I talked
> about "stepping" over prologues and over functions without debug info;
> I hadn't thought about skipping shared library trampolines, but I
> can definitely see how that would be a problem.
> 
> I don't like the user interface here; we say "step" and it will
> sometimes run one thread and sometimes run all threads and it's hard to
> explain which.  But we may not be able to do better without adding
> unacceptable risk of deadlocks :-(
> 

Re: Single stepping and threads

[Joel Brobecker]

> Default on would be a disaster -- most threaded programs would
> not behave even remotely the same under the debugger as they would
> solo.
> 
> In fact, many would deadlock almost immediately.

Something just occured to me that was clear but maybe isn't. Is the
scheduling affected when you do a "continue"? I assumed that, if you
do a "run" or "continue", the actual scheduling policy is irrelevant,
and all threads are resumed.

-- 
Joel

Re: Single stepping and threads

[Daniel Jacobowitz]

On Thu, Nov 30, 2006 at 03:55:15PM -0800, Joel Brobecker wrote:
> > Default on would be a disaster -- most threaded programs would
> > not behave even remotely the same under the debugger as they would
> > solo.
> > 
> > In fact, many would deadlock almost immediately.
> 
> Something just occured to me that was clear but maybe isn't. Is the
> scheduling affected when you do a "continue"? I assumed that, if you
> do a "run" or "continue", the actual scheduling policy is irrelevant,
> and all threads are resumed.

No, the options are basically never, PTRACE_SINGLESTEP only, or
PTRACE_SINGLESTEP and PTRACE_CONT.  With "on", no other thread ever
runs.  It's only useful in very specific cases.

An option different to any of the ones we've discussed so far, but that
I think you're expecting from the current "on", would be one that only
applies to the step and next commands.  I still think "step" is more
useful a default than we have now, though.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Single stepping and threads

[Michael Snyder]

On Thu, 2006-11-30 at 15:55 -0800, Joel Brobecker wrote:
> > Default on would be a disaster -- most threaded programs would
> > not behave even remotely the same under the debugger as they would
> > solo.
> > 
> > In fact, many would deadlock almost immediately.
> 
> Something just occured to me that was clear but maybe isn't. Is the
> scheduling affected when you do a "continue"? I assumed that, if you
> do a "run" or "continue", the actual scheduling policy is irrelevant,
> and all threads are resumed.

"Schedlock step" does not affect the continue behavior, 
but "schedlock on" does.  With "on", no other thread can
execute, period.

Re: Single stepping and threads

[Rob Quill]

On 30/11/06, Michael Snyder <Michael.Snyder@palmsource.com> wrote:
> On Wed, 2006-11-29 at 08:38 -0800, Joel Brobecker wrote:
> > > > I would say yes. A step should be a few instructions, while stepping
> > > > over a call is potentially a much larger number of instructions.
> > > > As a result, stepping over without letting the other threads go would
> > > > more likely cause a lock.
> > >
> > > I think you mean "no" then?
> >
> > Oops, sorry, I meant "no".
> >
> > One of my coworkers expressed his opinion as follow:
> >
> > <<
> > I would find it confusing if "step" and "next" behave differently with
> > respect to threads, because they seem like basically the same thing.
> > "Next is just like step, except that it goes over calls" seems simple to
> > me. "Next is just like step, except that it goes over calls, and has
> > some subtle difference regarding threads" seems more complicated to me.
> >
> > So I would suggest leaving the default as "off", or else changing it
> > to "on".
>
> Default on would be a disaster -- most threaded programs would
> not behave even remotely the same under the debugger as they would
> solo.
>
> In fact, many would deadlock almost immediately.

I have a question regarding this. In concurrent programming (as we
were tuaght it), the principle was that the interleaving of
instructions from threads was random. So, if "on" were the default,
and a few steps were done in GDB, in fact, as many as it took to
deadlock the program, surely it is possible (although, however
unlikely) that when the program is run without GDB that the
interleaving is the same as that forced by GDB, and the code would
deadlock. Thus making the code bad, rather than the debugger.

What I'm trying to say is that it was my understanding that when doing
concurent programming the interleaving was random and that for the
program to be "corrent" it should not deadlock under any possible
interleaving.

I fail to see how stopping all threads and just going forward with one
should stop "correct" code from executiong properly.

Rob
>
>
>

Re: Single stepping and threads

[Daniel Jacobowitz]

On Sat, Dec 02, 2006 at 04:27:12PM +0000, Rob Quill wrote:
> On 30/11/06, Michael Snyder <Michael.Snyder@palmsource.com> wrote:
> >Default on would be a disaster -- most threaded programs would
> >not behave even remotely the same under the debugger as they would
> >solo.
> >
> >In fact, many would deadlock almost immediately.
> 
> I have a question regarding this. In concurrent programming (as we
> were tuaght it), the principle was that the interleaving of
> instructions from threads was random. So, if "on" were the default,
> and a few steps were done in GDB, in fact, as many as it took to
> deadlock the program, surely it is possible (although, however
> unlikely) that when the program is run without GDB that the
> interleaving is the same as that forced by GDB, and the code would
> deadlock. Thus making the code bad, rather than the debugger.
> 
> What I'm trying to say is that it was my understanding that when doing
> concurent programming the interleaving was random and that for the
> program to be "corrent" it should not deadlock under any possible
> interleaving.
> 
> I fail to see how stopping all threads and just going forward with one
> should stop "correct" code from executiong properly.

That's not what "on" does - it would permanently starve all other
threads.

There's also the point that what is correct is not always useful; GDB
tries to interfere as little as practical.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Single stepping and threads

[Joel Brobecker]

> I fail to see how stopping all threads and just going forward with one
> should stop "correct" code from executiong properly.

Consider a rendez-vous, for instance. I'm not sure if a "rendez-vous"
is a universal term or not, so I'll briefly explain what I mean:
"rendez-vous" is used in Ada speak to represent a situation where
one task is blocked waiting for another task to call one of its
entries; neither task is allowed to run until both tasks reach the
entry.

Back to our question: If one of the tasks is not running, it cannot
reach its entry, and thus the other task gets stuck on that entry call.

-- 
Joel

Re: Single stepping and threads

[Michael Snyder]

On Sat, 2006-12-02 at 16:27 +0000, Rob Quill wrote:
> On 30/11/06, Michael Snyder <Michael.Snyder@palmsource.com> wrote:
> > On Wed, 2006-11-29 at 08:38 -0800, Joel Brobecker wrote:
> > > > > I would say yes. A step should be a few instructions, while stepping
> > > > > over a call is potentially a much larger number of instructions.
> > > > > As a result, stepping over without letting the other threads go would
> > > > > more likely cause a lock.
> > > >
> > > > I think you mean "no" then?
> > >
> > > Oops, sorry, I meant "no".
> > >
> > > One of my coworkers expressed his opinion as follow:
> > >
> > > <<
> > > I would find it confusing if "step" and "next" behave differently with
> > > respect to threads, because they seem like basically the same thing.
> > > "Next is just like step, except that it goes over calls" seems simple to
> > > me. "Next is just like step, except that it goes over calls, and has
> > > some subtle difference regarding threads" seems more complicated to me.
> > >
> > > So I would suggest leaving the default as "off", or else changing it
> > > to "on".
> >
> > Default on would be a disaster -- most threaded programs would
> > not behave even remotely the same under the debugger as they would
> > solo.
> >
> > In fact, many would deadlock almost immediately.
> 
> I have a question regarding this. In concurrent programming (as we
> were tuaght it), the principle was that the interleaving of
> instructions from threads was random.

It's not really random, it's "unpredictable" -- within the 
constraints of the model.  If you violate the model, say, by
having a direct pipeline to the kernel scheduler, you could
predict task switches perfectly.

Any given scheduler has an algorithm for deciding when to 
switch threads.  Some have a choice of algorithms, and many
can be "tuned".  Most or all are affected by outside events, 
eg. programs blocking on a device.

The first point to be made here is that gdb's participation in
the system *changes* the system -- so that the quasi-random
task switches will happen at different times and in potentially
different orders than otherwise.


>  So, if "on" were the default,
> and a few steps were done in GDB, in fact, as many as it took to
> deadlock the program, surely it is possible (although, however
> unlikely) that when the program is run without GDB that the
> interleaving is the same as that forced by GDB, and the code would
> deadlock. Thus making the code bad, rather than the debugger.

No.  Say you have a producer thread and a consumer thread.
You decide to debug the consumer thread, in isolation (ie. 
the producer thread cannot run.

You come to a point where the consumer thread blocks, waiting
for the producer thread to produce something.

Normally, this is not deadlock.  The consumer will sleep, and 
the producer will wake up and produce something.  But the producer
now CANNOT run, therefore we are deadlocked.