Debugging X with GDB

Debugging X with GDB

[Bernie Innocenti]

Cc'ing the gdb and xorg lists, in case someone knows better.

On Tue, 2010-01-05 at 22:21 -0500, Richard Stallman wrote:
> The version of GDB installed is 6.8, so I guess that is rather old.
> It is important for MIPS gNewSense to have a version of GDB
> which is well maintained for the MIPS.

I asked on #xorg-devel about our issue with ^C not working in gdb.

So, we seems to have encountered a long-standing problem, not related to
MIPS or a specific version of GDB:


<bernie> is it still to be expected that gdb can't stop the X server with ^C ?
<bernie> I'm debugging on a mips platform... it may very well be a platform bug
<bernie> ajax, airlied: maybe one of you knows?
<airlied> bernie: I'm not sure its ever worked for me, I generally kill -INT `pidof Xorg`
<whot> it works if you attach to a running server, not if you start through gdb
<bernie> whot: indeed, I just discovered it
<bernie> I wonder if it's a bug in gdb
<bernie> airlied: thanks for the suggestion, I thought it was just me :)
<airlied> I think X might change TTY state and piss gdb off
<bernie> airlied: oh, good point (although I'm debugging over ssh, it would be a bug for X to attempt changing that tty rather than the actual console)
<bernie> xf86DrvMsg(pScrn->scrnIndex, from, "Using %sware Cursor\n",
<bernie> lol
<nwnk> ^c'ing a running server is expected to work, regardless of how you attached gdb.
<nwnk> if it doesn't it's a bug in gdb or in the kernel's signal delivery code
<bernie> nwnk: I have "NoTrapSignals", but it doesn't help
<bernie> nwnk: I suspect a bug in gdb...
<nwnk> wouldn't be the first
<bernie> nwnk: as a matter of fact, X is resilient to ^C even when running standalone
<whot> really? that'd be news to me
<jcristau> ^c in gdb doesn't send sigint to the traced process though, it sends sigint to gdb
<bernie> whot: only when it's sort of crashed :-)
<bernie> whot: or maybe it's because of the NoTrapSignals
<nwnk> jcristau: right; gdb should then handle it according to whatever you set for 'signal SIGINT', which defaults to 'stop'
<nwnk> excuse me, 'handle SIGINT'
<nwnk> but what often seems to happen is the signal just gets ignored by both


-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/

Re: Debugging X with GDB

[Daniel Jacobowitz]

On Sun, Jan 10, 2010 at 07:42:16PM -0500, Bernie Innocenti wrote:
> <airlied> I think X might change TTY state and piss gdb off

I'd imagine this is it, but it's impossible to tell without debugging
the TTY state.

When you hit ^C, that doesn't necessarily cause anything to happen.
The TTY subsystem sees ^C, and may or may not generate a SIGINT to the
foreground process group depending on the tty settings; try checking
using stty -F, for instance.  Then the foreground process group is X,
not GDB; if the SIGINT is blocked or ignored, it won't be delivered,
so GDB can't intercept it at delivery.

Using ^C is not always reliable.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Debugging X with GDB

[Jonathan Morton]

On Mon, 2010-01-11 at 09:01 -0500, Daniel Jacobowitz wrote:
> On Sun, Jan 10, 2010 at 07:42:16PM -0500, Bernie Innocenti wrote:
> > <airlied> I think X might change TTY state and piss gdb off
> 
> I'd imagine this is it, but it's impossible to tell without debugging
> the TTY state.
> 
> When you hit ^C, that doesn't necessarily cause anything to happen.
> The TTY subsystem sees ^C, and may or may not generate a SIGINT to the
> foreground process group depending on the tty settings; try checking
> using stty -F, for instance.  Then the foreground process group is X,
> not GDB; if the SIGINT is blocked or ignored, it won't be delivered,
> so GDB can't intercept it at delivery.
> 
> Using ^C is not always reliable.

A good trick might be to run X+gdb via SSH or a serial console.  This is
then immune to VT switching semantics.

If X screws up sufficiently well to leave the graphics hardware in an
unusable state, or traps in gdb while you're watching it and is thus
unable to release the hardware to textmode, using SSH or serial console
also allows you to continue debugging.

-- 
------
From: Jonathan Morton
      jonathan.morton@movial.com

Re: Debugging X with GDB

[Adam Jackson]

[-- Attachment #1: Type: text/plain, Size: 1115 bytes --]

On Mon, 2010-01-11 at 16:26 +0200, Jonathan Morton wrote:
> On Mon, 2010-01-11 at 09:01 -0500, Daniel Jacobowitz wrote:
> > On Sun, Jan 10, 2010 at 07:42:16PM -0500, Bernie Innocenti wrote:
> > > <airlied> I think X might change TTY state and piss gdb off
> > 
> > I'd imagine this is it, but it's impossible to tell without debugging
> > the TTY state.
> > 
> > When you hit ^C, that doesn't necessarily cause anything to happen.
> > The TTY subsystem sees ^C, and may or may not generate a SIGINT to the
> > foreground process group depending on the tty settings; try checking
> > using stty -F, for instance.  Then the foreground process group is X,
> > not GDB; if the SIGINT is blocked or ignored, it won't be delivered,
> > so GDB can't intercept it at delivery.
> > 
> > Using ^C is not always reliable.
> 
> A good trick might be to run X+gdb via SSH or a serial console.  This is
> then immune to VT switching semantics.

This is the normal way people gdb X, yes.  If you tried to attach gdb to
the X server that's hosting the xterm that's running the gdb, you'd
deadlock.

- ajax

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Debugging X with GDB

[Richard Stallman]

    > A good trick might be to run X+gdb via SSH or a serial console.

That is what we did.  We were running GDB and X on another machine
through ssh.

[SOLVED] Re: Debugging X with GDB

[Bernie Innocenti]

On Mon, 2010-01-11 at 16:26 +0200, Jonathan Morton wrote:

> A good trick might be to run X+gdb via SSH or a serial console.  This
> is then immune to VT switching semantics.

This is how I'm currently debugging, but X unconditionally cripples the
console even if it's a pty.

...

Wait, I've found a workaround! There's a specific option:

  -keeptty         don't detach controlling tty (for debugging only)


That's cool, but I guess we should document it very prominently in the
wiki:

 http://www.x.org/wiki/Development/Documentation/ServerDebugging

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/

Re: Debugging X with GDB

[Richard Stallman]

    That would be indeed useful, but it seems that setting breakpoints on
    symbols that will become available in the future is also working
    correctly.  The difference is that now I'm debugging a newer version of
    the X server that doesn't confuse the debugger by re-implementing a
    custom ELF loader.

That is good, but GDB knew to recognize that the driver file was being
loaded even with old X server.  Given that it could recognize the
file, it could also offer a command to specify "stop the program after
loading file FOO".  That will be useful when strange things are
happening.

Re: Debugging X with GDB

[Bernie Innocenti]

On Tue, 2010-01-12 at 18:19 -0500, Richard Stallman wrote:
> That is good, but GDB knew to recognize that the driver file was being
> loaded even with old X server.  Given that it could recognize the
> file, it could also offer a command to specify "stop the program after
> loading file FOO".  That will be useful when strange things are
> happening.

I thought we could easily do that by just setting a breakpoint on
dlopen(), but it didn't seem to work for the X server, perhaps because
the old, hand-crafted dynamic loader did not call dlopen() at all.
Perhaps mmap() would have worked or, as a last resort, open().

gdb supports setting conditional expressions on breakpoints, but I'm not
sure they can be used on function arguments and if basic string
manipulation is supported.

Ideally, we'd want to have the moral equivalent of

  break dlopen if strstr(filename, "siliconmotion_drv")

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/

Re: Debugging X with GDB

[Tom Tromey]

>>>>> "Bernie" == Bernie Innocenti <bernie@codewiz.org> writes:

Bernie> On Tue, 2010-01-12 at 18:19 -0500, Richard Stallman wrote:
>> That is good, but GDB knew to recognize that the driver file was being
>> loaded even with old X server.  Given that it could recognize the
>> file, it could also offer a command to specify "stop the program after
>> loading file FOO".  That will be useful when strange things are
>> happening.

Bernie> I thought we could easily do that by just setting a breakpoint on
Bernie> dlopen(), but it didn't seem to work for the X server, perhaps because
Bernie> the old, hand-crafted dynamic loader did not call dlopen() at all.
Bernie> Perhaps mmap() would have worked or, as a last resort, open().

You could try `set stop-on-solib-events on'.

This isn't as handy as a breakpoint because you can't make it conditional.
IMO, both this and `handle' would be better expressed as `catch'
subcommands (thus allowing conditions and attached commands
automatically) -- but AFAIK nobody is actively working on this.

Bernie> gdb supports setting conditional expressions on breakpoints, but I'm not
Bernie> sure they can be used on function arguments and if basic string
Bernie> manipulation is supported.

Yes to both.

Tom