Towards better x86 system debugging support

Towards better x86 system debugging support

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3521 bytes --]

Hi,

as many of you may know, there is a gdb server in QEMU that allows to
debug kernels, boot loaders and other low-level stuff without real
target hardware. I'm e.g. using it heavily for analyzing Linux kernel
issues on x86 targets (thanks to KVM, you can even debug weird SMP races
in NMI-using kernel debuggers...).

Unfortunately, the x86 support is incomplete in so far that neither the
gdb remote protocol nor the gdb backend are aware of most special
registers x86 system-level software uses. This comes with several drawbacks:

 o Current code bit width (16, 32 or 64) is unknown to the debugger,
   so correct disassembling is not automatically possible
 o Real mode cannot be detected, which would include setting 16 bit
   disassembly mode and calculating segment bases appropriately
 o Manually setting the architecture (set arch i8086/i386/i386:x86-64)
   influences the register set layout of the remote protocol, preventing
   straightforward switches from 16-bit bootloader/BIOS code to 64-bit
   kernel code (to give just one example)
 o Only flat memory models are supported and debugging becomes very
   hairy when some segment uses a non-zero base address - note that this
   also prevents support for TLS variable lookup (which is GS or
   FS-based)

As a first step toward enhanced x86 support, I think there is a need for
an extended register set in the remote protocol. The following registers
should be added:

 o GDTR, LDTR, IDTR, TR (visible part, ie. selector value)
 o CR0..4
 o DR0..7
 o selected MSRs, at least
    - IA32_EFER (64-bit mode detection)
    - IA32_FS_Base (TLS)
    - IA32_GS_Base (TLS)
    - IA32_KernelGSbase (TLS)
 o Shadow states of segment registers, GDTR, LDTR, IDTR and TR
   (relevant for virtual targets where the VM often has access to these
   hidden states, helpful when debugging targets that modify in-use
   descriptor table entries)

If anyone thinks that there should be more registers or MSRs included,
please extend this list!

The question for me is how to extend the protocol precisely. I guess we
need some new qSupported feature. But should we then, if both sides
agreed on it, switch to a completely new register set or rather exchange
those additional registers separately, ie. via some new packet?

A related issue that was already the source of a lot of pain is the
register width. For 16 and 32 bit mode, gdb uses 32 bit registers in the
remote protocol. But we have a fairly different set for 64 bit mode. Now
I wonder if we should go for a full 64 bit set unconditionally with this
extension (even if the target only supports 32 bit), avoiding all
protocol problems related to switching from 64 bit to lower modes (*).
The only minor issue I see is suboptimal performance with 32 bit targets
over real serial lines - if that still matters...

Looking forward to comments, suggestions, (gdb) insights!

Jan

(*) QEMU recently decided to stick with 64 bit layout even if the x86-64
target is running in 16 or 32 bit mode. Before that the remote protocol
used to be switched between 32 and 64 bit dynamically, depending on the
current target mode. That solved many issues, but not all (manual 'set
arch' was required, and gdb became confused in a few cases). We are now
discussing again on qemu-devel how to deal with 16/32 bit system-level
debugging in 64 bit emulation environment: either try to improve gdb
quickly or reintroduce the old workaround, at least temporarily.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: Towards better x86 system debugging support

[Daniel Jacobowitz]

On Sun, Jan 04, 2009 at 02:34:00PM +0100, Jan Kiszka wrote:
> The question for me is how to extend the protocol precisely. I guess we
> need some new qSupported feature. But should we then, if both sides
> agreed on it, switch to a completely new register set or rather exchange
> those additional registers separately, ie. via some new packet?

No new feature required.  Take a look at the description of
target-described registers in the current manual; we'd just need
a naming convention for the x86 control registers of interest to GDB.
That solves your other issue too about width.

-- 
Daniel Jacobowitz
CodeSourcery

GDB MI and actual type of a pointer or reference

[Elmenthaler, Jens]

Hi,

When using gdb on the command line, the print command is able to inspect the actual type of pointers and references, instead of just the declared type (after sending "set print object on").

Is something similar available when using MI?

Jens.

Re: Towards better x86 system debugging support

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1239 bytes --]

Daniel Jacobowitz wrote:
> On Sun, Jan 04, 2009 at 02:34:00PM +0100, Jan Kiszka wrote:
>> The question for me is how to extend the protocol precisely. I guess we
>> need some new qSupported feature. But should we then, if both sides
>> agreed on it, switch to a completely new register set or rather exchange
>> those additional registers separately, ie. via some new packet?
> 
> No new feature required.  Take a look at the description of
> target-described registers in the current manual; we'd just need
> a naming convention for the x86 control registers of interest to GDB.
> That solves your other issue too about width.

Ah, of course, once again forgot about this.

So another convention would be that a target capable of up to 32 bit
mode would report its registers as 32 bit and a 64 bit target as 64 bit
- and they would transfer this width _independent_ of the current mode.
That leads me to the questions:

 o Roughly, what code changes are required to exchange some i386.xml or
   x86-64.xml?

 o If gdb accepted such a static XML description from some x86 target,
   would it already stick with the register layout even when setting the
   arch manually (or later automatically)?

Thanks,
Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: Towards better x86 system debugging support

[Mark Kettenis]

> Date: Sun, 04 Jan 2009 14:34:00 +0100
> From: Jan Kiszka <jan.kiszka@web.de>
> 
> Hi,
> 
> Unfortunately, the x86 support is incomplete in so far that neither the
> gdb remote protocol nor the gdb backend are aware of most special
> registers x86 system-level software uses. This comes with several drawbacks:

Yes there are some limitations.  What's there is mostly adequeate for
debugging traditional C kernel code.  And I for myself have always
considered implementing the bits needed for more low-level code an
effort too big to be worth it (how much 16-bit code is there in the
Linux kernel?).  But if you think otherwise, feel free to contribute
the appropriate code.

>  o Current code bit width (16, 32 or 64) is unknown to the debugger,
>    so correct disassembling is not automatically possible

It's tricky to figure this out when running on real hardware, but I
suppose it would be trivial for an emulator like QEMU to tell GDB what
mode it's in.  The idea has always been that GDB would become
multi-arch and that every frame would have an architecture associated
with it.  Perhaps that's the way to go.

>  o Real mode cannot be detected, which would include setting 16 bit
>    disassembly mode and calculating segment bases appropriately

Sorry, what are you trying to say here?

>  o Manually setting the architecture (set arch i8086/i386/i386:x86-64)
>    influences the register set layout of the remote protocol, preventing
>    straightforward switches from 16-bit bootloader/BIOS code to 64-bit
>    kernel code (to give just one example)

Even worse, it changes the layout of GDB's internal register cache.
And I'm not even sure it is feasable to unify the i386 and amd64
layouts in a meaningful way.

>  o Only flat memory models are supported and debugging becomes very
>    hairy when some segment uses a non-zero base address - note that this
>    also prevents support for TLS variable lookup (which is GS or
>    FS-based)

This problem is not unique to the i386/amd64 targets.  There are other
architectures that have more than one address space.  Unfortunately, I
believe nobody has really implemented a satisfactory solution to
implement them properly in GDB.

Not that the TLS problem has been solved already (in a satisfactory
way I think).  I'm not sure making GDB aware of %gs and %fs would
really help here.  Systems use those registers in different ways, and
they don't really help when you want to look at variables in a
different thread than the one you're running isn't it?

> As a first step toward enhanced x86 support, I think there is a need for
> an extended register set in the remote protocol. The following registers
> should be added:

Sorry, but I think you're starting at the wrong end here.  You should
probably first add support for additional registers to GDB's internal
register cache.  It doen't make sense to worry about the remote
protocol until that's done.

>  o GDTR, LDTR, IDTR, TR (visible part, ie. selector value)
>  o CR0..4
>  o DR0..7
>  o selected MSRs, at least
>     - IA32_EFER (64-bit mode detection)
>     - IA32_FS_Base (TLS)
>     - IA32_GS_Base (TLS)
>     - IA32_KernelGSbase (TLS)
>  o Shadow states of segment registers, GDTR, LDTR, IDTR and TR
>    (relevant for virtual targets where the VM often has access to these
>    hidden states, helpful when debugging targets that modify in-use
>    descriptor table entries)
> 
> If anyone thinks that there should be more registers or MSRs included,
> please extend this list!

I don't think it is a good idea to start adding MSRs to GDB's internal
register cache.  There are zillions of them.  It is probably a better
idea to implement them as a seperate address space.

One thing that's probably a good idea before starting hacking on all
this is to fully decouple the layout of the internal register cache
from the remote protocol such that when you change things there, you
don't have to worry about breaking existing remote stubs.

Re: Towards better x86 system debugging support

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 7036 bytes --]

Mark Kettenis wrote:
>> Date: Sun, 04 Jan 2009 14:34:00 +0100
>> From: Jan Kiszka <jan.kiszka@web.de>
>>
>> Hi,
>>
>> Unfortunately, the x86 support is incomplete in so far that neither the
>> gdb remote protocol nor the gdb backend are aware of most special
>> registers x86 system-level software uses. This comes with several drawbacks:
> 
> Yes there are some limitations.  What's there is mostly adequeate for
> debugging traditional C kernel code.  And I for myself have always
> considered implementing the bits needed for more low-level code an
> effort too big to be worth it (how much 16-bit code is there in the
> Linux kernel?).  But if you think otherwise, feel free to contribute
> the appropriate code.

Virtualization changed this picture a bit, specifically as some VMs like
QEMU allow guest debugging. There are a lot of legacy OSes around, some
of them use non-flat memory models, and some of them (at least ours) now
run inside a VM. Then, as long as x86 CPUs start in real mode, you may
want to debug boot loaders. And finally you may get the idea to debug
the VM itself while it runs legacy code (=> nested VMs - the next
logical step might be to teach VT-x/AMD-V awareness to gdb...).

> 
>>  o Current code bit width (16, 32 or 64) is unknown to the debugger,
>>    so correct disassembling is not automatically possible
> 
> It's tricky to figure this out when running on real hardware, but I
> suppose it would be trivial for an emulator like QEMU to tell GDB what
> mode it's in.

On real hw you need support by the target OS. Think of kernel debuggers
(kgdb...) - they typically run in privileged context along their OS and
have access to most information (except for the shadow segment register
states).

>  The idea has always been that GDB would become
> multi-arch and that every frame would have an architecture associated
> with it.  Perhaps that's the way to go.

Yes, would be nice. But I'm primarily looking for an iterative path, and
this feature sounds a bit like it could come later on that way.

> 
>>  o Real mode cannot be detected, which would include setting 16 bit
>>    disassembly mode and calculating segment bases appropriately
> 
> Sorry, what are you trying to say here?

gdb assumes protected mode so far. There is some old patch, I'm using it
in some more lengthy real mode sessions, which adds "set i386 real-mode
on" to enable at least proper code address translation + you have to
"set arch i8086". If we could detect CR0.PE=0, we would be able to
switch such things automatically (given non-flat addressing support as
well, of course).

> 
>>  o Manually setting the architecture (set arch i8086/i386/i386:x86-64)
>>    influences the register set layout of the remote protocol, preventing
>>    straightforward switches from 16-bit bootloader/BIOS code to 64-bit
>>    kernel code (to give just one example)
> 
> Even worse, it changes the layout of GDB's internal register cache.
> And I'm not even sure it is feasable to unify the i386 and amd64
> layouts in a meaningful way.

/me wonders how other archs with "legacy" modes handle this. Is x86
special here? And if so, why? Historically?

> 
>>  o Only flat memory models are supported and debugging becomes very
>>    hairy when some segment uses a non-zero base address - note that this
>>    also prevents support for TLS variable lookup (which is GS or
>>    FS-based)
> 
> This problem is not unique to the i386/amd64 targets.  There are other
> architectures that have more than one address space.  Unfortunately, I
> believe nobody has really implemented a satisfactory solution to
> implement them properly in GDB.

For x86, a colleague implemented an extended address resolution via
<segment>:<offset>. Works in practice but we likely need to
rebase/rework this once the register set extension is done.

> 
> Not that the TLS problem has been solved already (in a satisfactory
> way I think).  I'm not sure making GDB aware of %gs and %fs would
> really help here.  Systems use those registers in different ways, and
> they don't really help when you want to look at variables in a
> different thread than the one you're running isn't it?

I was thinking of system level debugging here where you don't have OS
support. But you are right, teaching the TLS (or per-CPU) lookup
algorithms to gdb might not be a good idea. Maybe some helper gdb
scripts would be better (given they can do things like "x $fs:0").

> 
>> As a first step toward enhanced x86 support, I think there is a need for
>> an extended register set in the remote protocol. The following registers
>> should be added:
> 
> Sorry, but I think you're starting at the wrong end here.  You should
> probably first add support for additional registers to GDB's internal
> register cache.  It doen't make sense to worry about the remote
> protocol until that's done.

Yes and no. Due to the fact that several projects are affected (at least
gdb and qemu, maybe also kgdb) and that we already have temporary
patches for many use cases, I'm looking for a smooth, iterative
realization path. Just having extended registers won't help as much as
already have a source (e.g. QEMU's gdbstub) for retrieving them from
real targets.

On the other hand, I now see that an earlier step is teaching gdb to use
non-static target register descriptions that would come via xml and that
this way the remote protocol is already defined.

> 
>>  o GDTR, LDTR, IDTR, TR (visible part, ie. selector value)
>>  o CR0..4
>>  o DR0..7
>>  o selected MSRs, at least
>>     - IA32_EFER (64-bit mode detection)
>>     - IA32_FS_Base (TLS)
>>     - IA32_GS_Base (TLS)
>>     - IA32_KernelGSbase (TLS)
>>  o Shadow states of segment registers, GDTR, LDTR, IDTR and TR
>>    (relevant for virtual targets where the VM often has access to these
>>    hidden states, helpful when debugging targets that modify in-use
>>    descriptor table entries)
>>
>> If anyone thinks that there should be more registers or MSRs included,
>> please extend this list!
> 
> I don't think it is a good idea to start adding MSRs to GDB's internal
> register cache.  There are zillions of them.  It is probably a better
> idea to implement them as a seperate address space.

Sounds reasonable - you wouldn't have to decide which ones to include or
to add later on. But how would I access this address space, specifically
remotely?

BTW, this approach would also be interesting to add the PIO address
space to gdb (we are currently doing this via maintenance packets that
issue QEMU monitor commands to access I/O ports...).

> 
> One thing that's probably a good idea before starting hacking on all
> this is to fully decouple the layout of the internal register cache
> from the remote protocol such that when you change things there, you
> don't have to worry about breaking existing remote stubs.

Mmm, makes sense. Will dig into this.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: Towards better x86 system debugging support

[Daniel Jacobowitz]

On Mon, Jan 05, 2009 at 10:36:07AM +0100, Jan Kiszka wrote:
>  o Roughly, what code changes are required to exchange some i386.xml or
>    x86-64.xml?

- Verification of the supplied registers in gdbarch init.
- Changes to hardcoded register numbers to always access pseudo
registers relative to gdbarch_num_regs, which will no longer be
constant.

If you agree with Mark that we should add the MSRs as a separate
address space rather than into the register cache it's a bit
trickier.  I have relevant patches somewhere...

>  o If gdb accepted such a static XML description from some x86 target,
>    would it already stick with the register layout even when setting the
>    arch manually (or later automatically)?

I don't remember whether set arch overrides just the arch, or the
entire target description.  But probably the former.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Towards better x86 system debugging support

[Doug Evans]

On Sun, Jan 4, 2009 at 5:34 AM, Jan Kiszka <jan.kiszka@web.de> wrote:
> Hi,
>
> as many of you may know, there is a gdb server in QEMU that allows to
> debug kernels, boot loaders and other low-level stuff without real
> target hardware. I'm e.g. using it heavily for analyzing Linux kernel
> issues on x86 targets (thanks to KVM, you can even debug weird SMP races
> in NMI-using kernel debuggers...).
>
> Unfortunately, the x86 support is incomplete in so far that neither the
> gdb remote protocol nor the gdb backend are aware of most special
> registers x86 system-level software uses. This comes with several drawbacks:
>
>  o Current code bit width (16, 32 or 64) is unknown to the debugger,
>   so correct disassembling is not automatically possible
>  o Real mode cannot be detected, which would include setting 16 bit
>   disassembly mode and calculating segment bases appropriately
>  o Manually setting the architecture (set arch i8086/i386/i386:x86-64)
>   influences the register set layout of the remote protocol, preventing
>   straightforward switches from 16-bit bootloader/BIOS code to 64-bit
>   kernel code (to give just one example)
>  o Only flat memory models are supported and debugging becomes very
>   hairy when some segment uses a non-zero base address - note that this
>   also prevents support for TLS variable lookup (which is GS or
>   FS-based)
>
> As a first step toward enhanced x86 support, I think there is a need for
> an extended register set in the remote protocol. The following registers
> should be added:
>
>  o GDTR, LDTR, IDTR, TR (visible part, ie. selector value)
>  o CR0..4
>  o DR0..7
>  o selected MSRs, at least
>    - IA32_EFER (64-bit mode detection)
>    - IA32_FS_Base (TLS)
>    - IA32_GS_Base (TLS)
>    - IA32_KernelGSbase (TLS)
>  o Shadow states of segment registers, GDTR, LDTR, IDTR and TR
>   (relevant for virtual targets where the VM often has access to these
>   hidden states, helpful when debugging targets that modify in-use
>   descriptor table entries)
>
> If anyone thinks that there should be more registers or MSRs included,
> please extend this list!

I know of one linux program that uses modify_ldt() and changes all
segment registers, so I would add cs_base,ds_base,es_base,ss_base.
Some folks would presumably find *_limit, *_flags useful too.

Re: Towards better x86 system debugging support

[Samuel Bronson]

Jan Kiszka <jan.kiszka <at> web.de> writes:
> Mark Kettenis wrote:
> >> From: Jan Kiszka <jan.kiszka <at> web.de>

> >> Unfortunately, the x86 support is incomplete in so far that neither the
> >> gdb remote protocol nor the gdb backend are aware of most special
> >> registers x86 system-level software uses. This comes with several drawbacks:

> >>  o Current code bit width (16, 32 or 64) is unknown to the debugger,
> >>    so correct disassembling is not automatically possible

> >>  o Real mode cannot be detected, which would include setting 16 bit
> >>    disassembly mode and calculating segment bases appropriately

> >>  o Only flat memory models are supported and debugging becomes very
> >>    hairy when some segment uses a non-zero base address - note that this
> >>    also prevents support for TLS variable lookup (which is GS or
> >>    FS-based)

> >> As a first step toward enhanced x86 support, I think there is a need for
> >> an extended register set in the remote protocol. The following registers
> >> should be added:

> >>  o GDTR, LDTR, IDTR, TR (visible part, ie. selector value)
> >>  o CR0..4
> >>  o DR0..7
> >>  o selected MSRs, at least
> >>     - IA32_EFER (64-bit mode detection)
> >>     - IA32_FS_Base (TLS)
> >>     - IA32_GS_Base (TLS)
> >>     - IA32_KernelGSbase (TLS)
> >>  o Shadow states of segment registers, GDTR, LDTR, IDTR and TR
> >>    (relevant for virtual targets where the VM often has access to these
> >>    hidden states, helpful when debugging targets that modify in-use
> >>    descriptor table entries)

These things have also been bothering me lately, along with GDB's poor handling
of interrupt handlers, namely:

o the "finish" command doesn't work in an interrupt-handler frame

o GDB (presumably) does not notice when a frame's (interrupt) return CS:EIP
  is at a different privilege level from that frame's own CS:EIP, which it
  would need to do in order to correctly unwind the calling frame's SS:ESP,
  which are stored on the stack in this situation rather than inferred
  based on the present frame

Have you made any progress on any of this, Jan?

Re: Towards better x86 system debugging support

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]

Samuel Bronson wrote:
> Jan Kiszka <jan.kiszka <at> web.de> writes:
>> Mark Kettenis wrote:
>>>> From: Jan Kiszka <jan.kiszka <at> web.de>
> 
>>>> Unfortunately, the x86 support is incomplete in so far that neither the
>>>> gdb remote protocol nor the gdb backend are aware of most special
>>>> registers x86 system-level software uses. This comes with several drawbacks:
> 
>>>>  o Current code bit width (16, 32 or 64) is unknown to the debugger,
>>>>    so correct disassembling is not automatically possible
> 
>>>>  o Real mode cannot be detected, which would include setting 16 bit
>>>>    disassembly mode and calculating segment bases appropriately
> 
>>>>  o Only flat memory models are supported and debugging becomes very
>>>>    hairy when some segment uses a non-zero base address - note that this
>>>>    also prevents support for TLS variable lookup (which is GS or
>>>>    FS-based)
> 
>>>> As a first step toward enhanced x86 support, I think there is a need for
>>>> an extended register set in the remote protocol. The following registers
>>>> should be added:
> 
>>>>  o GDTR, LDTR, IDTR, TR (visible part, ie. selector value)
>>>>  o CR0..4
>>>>  o DR0..7
>>>>  o selected MSRs, at least
>>>>     - IA32_EFER (64-bit mode detection)
>>>>     - IA32_FS_Base (TLS)
>>>>     - IA32_GS_Base (TLS)
>>>>     - IA32_KernelGSbase (TLS)
>>>>  o Shadow states of segment registers, GDTR, LDTR, IDTR and TR
>>>>    (relevant for virtual targets where the VM often has access to these
>>>>    hidden states, helpful when debugging targets that modify in-use
>>>>    descriptor table entries)
> 
> These things have also been bothering me lately, along with GDB's poor handling
> of interrupt handlers, namely:
> 
> o the "finish" command doesn't work in an interrupt-handler frame
> 
> o GDB (presumably) does not notice when a frame's (interrupt) return CS:EIP
>   is at a different privilege level from that frame's own CS:EIP, which it
>   would need to do in order to correctly unwind the calling frame's SS:ESP,
>   which are stored on the stack in this situation rather than inferred
>   based on the present frame
> 
> Have you made any progress on any of this, Jan?

Unfortunately not. I was hoping my todo list would shrink a bit or some
customer explicitly ask for a clean integration, but both did not happen
yet. But it's not forgotten, it's just "slightly" delayed.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]