Post mortem debugging for Windows CE

Post mortem debugging for Windows CE

[Danny Backx]

Hi,

I wrote a program, see below, that can be used either on a development
host, or on a Windows CE target, to decode stuff from a dump file as
generated by CE.

I don't think gdb currently has this capability (to inspect postmortem
dump files) for debugging CE applications.

Is this something I should port into the gdb source tree and submit, or
is it not worth it ? If yes, please advise where this belongs.

Sources are in the SVN of the cegcc project
(https://cegcc.svn.sourceforge.net/svnroot/cegcc/trunk/cegcc/tools/cedump) .

	Danny

dannypc: {1273} rsh
ipaq /temp/cedump /windows/system/dumpfiles/Ce042709-01/Ce042709-01.kdmp
Number of streams 10
stream 0, type 8001 (System info), size 72
System info :
        arch 5, 1 processors
        Proc type 2577 level 4 rev 1
        Proc family 4 Platform 3
        Locale id 2067  813
        OEM {hp iPAQ 210}
stream 1, type 8002 (Exception), size 160
Exception : code C0000005, flags 0, address 1117C
Parameter 0 : [1]
Parameter 1 : [26000000]
stream 2, type 8003 (Module), size 336
  Element 0 :
        Name : NK.EXE
        Base Ptr : 0x80001000
  Element 1 :
        Name : null2.exe
        Base Ptr : 0x26010000
  Element 2 :
        Name : coredll.dll.0409.mui
        Base Ptr : 0x7FFE0000
  Element 3 :
        Name : coredll.dll
        Base Ptr : 0x03F4E000
stream 3, type 8004 (Process), size 208
  Element 0 :
        ProcSlot# : 0
        Name : NK.EXE
        VMBase : 0xC2000000
        BasePtr : 0x80001000
        pProcess : 0x8034D260
  Element 1 :
        ProcSlot# : 18
        Name : null2.exe
        VMBase : 0x26000000
        BasePtr : 0x00010000
        pProcess : 0x8034E340
stream 4, type 8005 (Thread), size 416
  Element 0 :
        pThread : 0x8172B640
        RunState : Awak,Rung
        hThread : 0x22CE3EAE
        LastError : 0x00000000
        PC : 0x0001117C
        StkRetAddr : 0x00000000
stream 5, type 8006 (Thread Context), size 48
  Element 0 :
        ThreadID : 0x22ce3eae
stream 6, type 8007 (Thread Callstack), size 24
Thread Call Stack List : 1 entries
  Process 22E4A1C6 thread 22CE3EAE, 1 frames
    Frame 0 : RetAddr 0x2601117C FP 0x2611FE04
stream 7, type 800a (Bucket parameters), size 64
stream 8, type 8008 (Memory virtual list), size 56
stream 9, type 8009 (Memory physical list), size 8

-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Pedro Alves]

On Tuesday 28 April 2009 15:45:01, Danny Backx wrote:
> Hi,
> 
> I wrote a program, see below, that can be used either on a development
> host, or on a Windows CE target, to decode stuff from a dump file as
> generated by CE.
> 
> I don't think gdb currently has this capability (to inspect postmortem
> dump files) for debugging CE applications.

These are minidump files, right?  The same infrastructure could
be used for minidumps for desktop Windows apps?  I've never used
a minidump, but, IIRC, these are a form of bare-essential core
dumps, without much of memory contents?  I'm not exactly sure what
kinds of info a minidump contains -- I'd expect at least the register
contexts of all threads in the process, and the list of loaded
dlls, along with their load addresses.  Is it possible to recontruct
a call stack from these?  Do they include stack memory?

> Is this something I should port into the gdb source tree and submit, or
> is it not worth it ? If yes, please advise where this belongs. 

Can you clarify what you mean exactly by porting to the gdb source
tree?  If you mean integrating it as a standalone app as is currently,
then, this would better go into binutils, as a minidump
dumper.  You'd want to make it build independantly if any CE headers.

If you mean adding gdb support to load minidump files, I can think
of several ways to skin that cat:

1) write a minidump -> elf core dump translator.  Base the core file
notes on the cygwin elf core format, for e.g..  See e.g,
src/bfd/elf.c:elfcore_grok_win32pstatus, and cygwin's dumper utility
sources for inspiration.  Probably no bfd changes are required
this way.  Teach src/gdb/arm-wince-tdep.c how to extract loaded
dll list from the core's .module sections.  See
src/gdb/i386-cygwin-tdep.c:windows_core_xfer_shared_libraries.  We could
probably reuse most of this.

2) teach bfd proper about minidumps, and export the register
set info with fake .reg sections, similarly to how we do for elf
core dumps.  If you export the same set of sections GDB expects from
elf cores (e.g., .reg/XXX sections for thread register sets, .module sections
for loaded dlls.), then the GDB tweaks are the same as
for #1 above.  E.g, teach src/gdb/arm-wince-tdep.c how to extract the loaded
dll list from the core's .module sections.

3) Teach GDB about minidumps, by adding a new minidump target_ops,
similar in vein to src/gdb/corelow.c.  GDB doesn't currenly like
having more than on core file target_ops, though.

4) forget about adding support for minidumps, or post-processing them
to elf; and instead port cygwin's dumper.exe utility to CE, making
it spit real elf full-contents core dumps.  This assumes you have
memory space on the device for this, of course.

-- 
Pedro Alves

Re: Post mortem debugging for Windows CE

[Danny Backx]

On Tue, 2009-04-28 at 17:02 +0100, Pedro Alves wrote:
> On Tuesday 28 April 2009 15:45:01, Danny Backx wrote:
> > Hi,
> > 
> > I wrote a program, see below, that can be used either on a development
> > host, or on a Windows CE target, to decode stuff from a dump file as
> > generated by CE.
> > 
> > I don't think gdb currently has this capability (to inspect postmortem
> > dump files) for debugging CE applications.
> 
> These are minidump files, right?  The same infrastructure could
> be used for minidumps for desktop Windows apps?  I've never used
> a minidump, but, IIRC, these are a form of bare-essential core
> dumps, without much of memory contents?  I'm not exactly sure what
> kinds of info a minidump contains -- I'd expect at least the register
> contexts of all threads in the process, and the list of loaded
> dlls, along with their load addresses.  Is it possible to recontruct
> a call stack from these?  Do they include stack memory?

Yes, minidumps are one of the terms Microsoft uses. "Dr. Watson" is
another, although that usually refers to the tool to read those dumps.

All of the info I have on this comes from MSDN, you can look around near
http://msdn.microsoft.com/en-us/library/ms939593.aspx .

I searched for "minidump format vista" on MSDN and found
http://support.microsoft.com/kb/315263 which describes that desktop
Windows uses the same mechanism.

One of the comments in the first link I provided is that the dump file
format is the same across architectures.

There are several "streams" in such a dump file, for CE they're named
        ceStreamSystemInfo
        ceStreamException
        ceStreamModuleList
        ceStreamProcessList
        ceStreamThreadList
        ceStreamThreadContextList
        ceStreamThreadCallStackList
        ceStreamMemoryVirtualList
        ceStreamMemoryPhysicalList
        ceStreamBucketParameters

I've not decoded them all yet. The example I sent in the earlier mail
shows output after the "stream %d, .." line if I decode that stream
already.

> > Is this something I should port into the gdb source tree and submit, or
> > is it not worth it ? If yes, please advise where this belongs. 
> 
> Can you clarify what you mean exactly by porting to the gdb source
> tree?  If you mean integrating it as a standalone app as is currently,
> then, this would better go into binutils, as a minidump
> dumper.  You'd want to make it build independantly if any CE headers.

Good questions.

The last one is simple :
- right now the source depends on only one header file (except from the
  Linux includes on my PC). This one header file has been created by
  reading MSDN material only. Look around at the first link above and
  you'll find it all.
- so : no dependencies you don't want in the gdb or bfd sources

Your other question is basically what I wanted to ask about. My idea was
your option #2 below.

But before I even start thinking about doing that, I'd like to know
whether this is a sensible thing to put my time in, and pointers on
where to start.

	Danny

> If you mean adding gdb support to load minidump files, I can think
> of several ways to skin that cat:
> 
> 1) write a minidump -> elf core dump translator.  Base the core file
> notes on the cygwin elf core format, for e.g..  See e.g,
> src/bfd/elf.c:elfcore_grok_win32pstatus, and cygwin's dumper utility
> sources for inspiration.  Probably no bfd changes are required
> this way.  Teach src/gdb/arm-wince-tdep.c how to extract loaded
> dll list from the core's .module sections.  See
> src/gdb/i386-cygwin-tdep.c:windows_core_xfer_shared_libraries.  We could
> probably reuse most of this.
> 
> 2) teach bfd proper about minidumps, and export the register
> set info with fake .reg sections, similarly to how we do for elf
> core dumps.  If you export the same set of sections GDB expects from
> elf cores (e.g., .reg/XXX sections for thread register sets, .module sections
> for loaded dlls.), then the GDB tweaks are the same as
> for #1 above.  E.g, teach src/gdb/arm-wince-tdep.c how to extract the loaded
> dll list from the core's .module sections.
> 
> 3) Teach GDB about minidumps, by adding a new minidump target_ops,
> similar in vein to src/gdb/corelow.c.  GDB doesn't currenly like
> having more than on core file target_ops, though.
> 
> 4) forget about adding support for minidumps, or post-processing them
> to elf; and instead port cygwin's dumper.exe utility to CE, making
> it spit real elf full-contents core dumps.  This assumes you have
> memory space on the device for this, of course.
> 
-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Danny Backx]

Here's more info, I've decoded more of the dump.

I think the three chunks of memory included (see stream 8) are :
#0 : 80 bytes from the program text at the point of the crash
#1 : 380 bytes near the end of address space (I have no clue)
#2 : some amount of memory from the frame pointer (see stream 6)

I've run this report on two crash dumps just now, here are the last
lines of both sets of output. Note that #2 differs.
Roadmap (the first dump) is a real application (GPS stuff).
The second dump is from a trivial test app called "callstack".

===========
stream 6, type 8007 (Thread Callstack), size 24
Thread Call Stack List : 1 entries
  Process A0D26B72 thread A17A19A2, 1 frames
    Frame 0 : RetAddr 0x20039CE4 FP 0x201EF654
stream 7, type 800a (Bucket parameters), size 64
  Bucket parameters :
        AppName [wroadmap.exe]
        OwnerName [wroadmap.exe]
        Module [wroadmap.exe]
stream 8, type 8008 (Memory virtual list), size 56
  Memory list 3 entries
    Entry 0 start 0x20039CA4 len 80 Rva 2C70
    Entry 1 start 0xFFFFC800 len 380 Rva 2CF0
    Entry 2 start 0x201EC000 len 4000 Rva 3070
stream 9, type 8009 (Memory physical list), size 8
  Memory list 0 entries
==========
stream 6, type 8007 (Thread Callstack), size 24
Thread Call Stack List : 1 entries
  Process 21623686 thread 236B284E, 1 frames
    Frame 0 : RetAddr 0x26011178 FP 0x2611FB0C
stream 7, type 800a (Bucket parameters), size 64
  Bucket parameters :
        AppName [callstack]
        OwnerName [callstack]
        Module [callstack]
stream 8, type 8008 (Memory virtual list), size 56
  Memory list 3 entries
    Entry 0 start 0x26011138 len 80 Rva 18D0
    Entry 1 start 0xFFFFC800 len 380 Rva 1950
    Entry 2 start 0x2611F000 len 1000 Rva 1CD0
stream 9, type 8009 (Memory physical list), size 8
  Memory list 0 entries
==========

So I think all the bits and pieces for gdb to do a backtrace might
actually be there...

Maybe memory dump #1 is a register dump :-)

	Danny

On Tue, 2009-04-28 at 19:48 +0200, Danny Backx wrote:
> On Tue, 2009-04-28 at 17:02 +0100, Pedro Alves wrote:
> > On Tuesday 28 April 2009 15:45:01, Danny Backx wrote:
> > > Hi,
> > > 
> > > I wrote a program, see below, that can be used either on a development
> > > host, or on a Windows CE target, to decode stuff from a dump file as
> > > generated by CE.
> > > 
> > > I don't think gdb currently has this capability (to inspect postmortem
> > > dump files) for debugging CE applications.
> > 
> > These are minidump files, right?  The same infrastructure could
> > be used for minidumps for desktop Windows apps?  I've never used
> > a minidump, but, IIRC, these are a form of bare-essential core
> > dumps, without much of memory contents?  I'm not exactly sure what
> > kinds of info a minidump contains -- I'd expect at least the register
> > contexts of all threads in the process, and the list of loaded
> > dlls, along with their load addresses.  Is it possible to recontruct
> > a call stack from these?  Do they include stack memory?
> 
> Yes, minidumps are one of the terms Microsoft uses. "Dr. Watson" is
> another, although that usually refers to the tool to read those dumps.
> 
> All of the info I have on this comes from MSDN, you can look around near
> http://msdn.microsoft.com/en-us/library/ms939593.aspx .
> 
> I searched for "minidump format vista" on MSDN and found
> http://support.microsoft.com/kb/315263 which describes that desktop
> Windows uses the same mechanism.
> 
> One of the comments in the first link I provided is that the dump file
> format is the same across architectures.
> 
> There are several "streams" in such a dump file, for CE they're named
>         ceStreamSystemInfo
>         ceStreamException
>         ceStreamModuleList
>         ceStreamProcessList
>         ceStreamThreadList
>         ceStreamThreadContextList
>         ceStreamThreadCallStackList
>         ceStreamMemoryVirtualList
>         ceStreamMemoryPhysicalList
>         ceStreamBucketParameters
> 
> I've not decoded them all yet. The example I sent in the earlier mail
> shows output after the "stream %d, .." line if I decode that stream
> already.
> 
> > > Is this something I should port into the gdb source tree and submit, or
> > > is it not worth it ? If yes, please advise where this belongs. 
> > 
> > Can you clarify what you mean exactly by porting to the gdb source
> > tree?  If you mean integrating it as a standalone app as is currently,
> > then, this would better go into binutils, as a minidump
> > dumper.  You'd want to make it build independantly if any CE headers.
> 
> Good questions.
> 
> The last one is simple :
> - right now the source depends on only one header file (except from the
>   Linux includes on my PC). This one header file has been created by
>   reading MSDN material only. Look around at the first link above and
>   you'll find it all.
> - so : no dependencies you don't want in the gdb or bfd sources
> 
> Your other question is basically what I wanted to ask about. My idea was
> your option #2 below.
> 
> But before I even start thinking about doing that, I'd like to know
> whether this is a sensible thing to put my time in, and pointers on
> where to start.
> 
> 	Danny
> 
> > If you mean adding gdb support to load minidump files, I can think
> > of several ways to skin that cat:
> > 
> > 1) write a minidump -> elf core dump translator.  Base the core file
> > notes on the cygwin elf core format, for e.g..  See e.g,
> > src/bfd/elf.c:elfcore_grok_win32pstatus, and cygwin's dumper utility
> > sources for inspiration.  Probably no bfd changes are required
> > this way.  Teach src/gdb/arm-wince-tdep.c how to extract loaded
> > dll list from the core's .module sections.  See
> > src/gdb/i386-cygwin-tdep.c:windows_core_xfer_shared_libraries.  We could
> > probably reuse most of this.
> > 
> > 2) teach bfd proper about minidumps, and export the register
> > set info with fake .reg sections, similarly to how we do for elf
> > core dumps.  If you export the same set of sections GDB expects from
> > elf cores (e.g., .reg/XXX sections for thread register sets, .module sections
> > for loaded dlls.), then the GDB tweaks are the same as
> > for #1 above.  E.g, teach src/gdb/arm-wince-tdep.c how to extract the loaded
> > dll list from the core's .module sections.
> > 
> > 3) Teach GDB about minidumps, by adding a new minidump target_ops,
> > similar in vein to src/gdb/corelow.c.  GDB doesn't currenly like
> > having more than on core file target_ops, though.
> > 
> > 4) forget about adding support for minidumps, or post-processing them
> > to elf; and instead port cygwin's dumper.exe utility to CE, making
> > it spit real elf full-contents core dumps.  This assumes you have
> > memory space on the device for this, of course.
> > 
-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Eli Zaretskii]

> From: Danny Backx <danny.backx@scarlet.be>
> Cc: gdb-patches@sourceware.org
> Date: Tue, 28 Apr 2009 21:17:34 +0200
> 
> Here's more info, I've decoded more of the dump.
> 
> I think the three chunks of memory included (see stream 8) are :
> #0 : 80 bytes from the program text at the point of the crash
> #1 : 380 bytes near the end of address space (I have no clue)
> #2 : some amount of memory from the frame pointer (see stream 6)

Thanks.

I know very little about Windows minidump files, but unless this
feature is going to allow GDB to debug the dump, as we do with core
files, the feature does not belong to GDB but to Binutils.  At least
IMHO.

Re: Post mortem debugging for Windows CE

[Danny Backx]

http://www.windowsfordevices.com/articles/AT7568439504.html shows what
the Microsoft tools do based on this file.

On Wed, 2009-04-29 at 10:10 +0300, Eli Zaretskii wrote:
> > From: Danny Backx <danny.backx@scarlet.be>
> > Cc: gdb-patches@sourceware.org
> > Date: Tue, 28 Apr 2009 21:17:34 +0200
> > 
> > Here's more info, I've decoded more of the dump.
> > 
> > I think the three chunks of memory included (see stream 8) are :
> > #0 : 80 bytes from the program text at the point of the crash
> > #1 : 380 bytes near the end of address space (I have no clue)
> > #2 : some amount of memory from the frame pointer (see stream 6)
> 
> Thanks.
> 
> I know very little about Windows minidump files, but unless this
> feature is going to allow GDB to debug the dump, as we do with core
> files, the feature does not belong to GDB but to Binutils.  At least
> IMHO.
> 
-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Eli Zaretskii]

> From: Danny Backx <danny.backx@scarlet.be>
> Cc: pedro@codesourcery.com, gdb-patches@sourceware.org
> Date: Wed, 29 Apr 2009 09:18:55 +0200
> 
> http://www.windowsfordevices.com/articles/AT7568439504.html shows what
> the Microsoft tools do based on this file.

Thanks, I knew most of that (although not in the context of CE).  My
question was: are we going to provide such functionality as part of
GDB?  If that's what we intend to do, then I think adding this would
be a welcome GDB feature.

Re: Post mortem debugging for Windows CE

[Danny Backx]

On Tue, 2009-04-28 at 17:02 +0100, Pedro Alves wrote:
> On Tuesday 28 April 2009 15:45:01, Danny Backx wrote:
> > I don't think gdb currently has this capability (to inspect postmortem
> > dump files) for debugging CE applications.
[..]
> If you mean adding gdb support to load minidump files, I can think
> of several ways to skin that cat:
[..]
> 2) teach bfd proper about minidumps, and export the register
> set info with fake .reg sections, similarly to how we do for elf
> core dumps.  If you export the same set of sections GDB expects from
> elf cores (e.g., .reg/XXX sections for thread register sets, .module sections
> for loaded dlls.), then the GDB tweaks are the same as
> for #1 above.  E.g, teach src/gdb/arm-wince-tdep.c how to extract the loaded
> dll list from the core's .module sections.
> 
> 3) Teach GDB about minidumps, by adding a new minidump target_ops,
> similar in vein to src/gdb/corelow.c.  GDB doesn't currenly like
> having more than on core file target_ops, though.

I was wondering what is the difference between 2 and 3, and which one is
preferred ?

	Danny
-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Pedro Alves]

Hi Danny,

On Thursday 30 April 2009 08:24:41, Danny Backx wrote:
> On Tue, 2009-04-28 at 17:02 +0100, Pedro Alves wrote:
> > On Tuesday 28 April 2009 15:45:01, Danny Backx wrote:
> > > I don't think gdb currently has this capability (to inspect postmortem
> > > dump files) for debugging CE applications.
> [..]
> > If you mean adding gdb support to load minidump files, I can think
> > of several ways to skin that cat:
> [..]
> > 2) teach bfd proper about minidumps, and export the register
> > set info with fake .reg sections, similarly to how we do for elf
> > core dumps.  If you export the same set of sections GDB expects from
> > elf cores (e.g., .reg/XXX sections for thread register sets, .module sections
> > for loaded dlls.), then the GDB tweaks are the same as
> > for #1 above.  E.g, teach src/gdb/arm-wince-tdep.c how to extract the loaded
> > dll list from the core's .module sections.
> > 
> > 3) Teach GDB about minidumps, by adding a new minidump target_ops,
> > similar in vein to src/gdb/corelow.c.  GDB doesn't currenly like
> > having more than on core file target_ops, though.
> 
> I was wondering what is the difference between 2 and 3, and which one is
> preferred ?

3. meant putting the minidump file parsing/loading code in GDB
itself.  The core of GDB is abstracted out from a "target" by
the target_ops interface.  This is the interface responsible for
implementing the "read memory", "read registers", "gimme me
shared library list", etc. callbacks.  src/gdb/corelow.c is 
that interface implemented for reading core files.  If you
take a look there, you'll notice that that file doesn't load
and parsing a core file itself.  It uses bfd to do so.  This means
that GDB doesn't really care much if the core file is an
elf core dump, an a.out core dump or something else.  The file
*format* itself is abstracted out on the bfd side.  As long
as the bfd interface is the same, new core file formats
will work without much fuss.

I wasn't certain how much of a core dump a minidump is when I
wrote that; I'm now mostly convinced that 2. is the way
to go.

2. meant putting the minidump file parsing code in bfd.  On
GDB side, things become mostly transparent.  This is the
prefered approach.  See all the *-core.c files in bfd for
starters, e.g. cisco-core.c or ptrace-core.c looks small
enough to base on.  See also bfd/elf.c, and look for bfd_core,
and look at all the elfcore_grok_* functions to get a
feel, specifically, you'll be interested in copying
elfcore_grok_win32pstatus's NOTE_INFO_MODULE handling.
I've never done this myself, but it looks to me you'll
need to follow suit and add a new minidump bfd target, in
a new bfd/minidump-core.c file.

-- 
Pedro Alves

Re: Post mortem debugging for Windows CE

[Danny Backx]

On Thu, 2009-04-30 at 10:58 +0100, Pedro Alves wrote:
> On Thursday 30 April 2009 08:24:41, Danny Backx wrote:
> > On Tue, 2009-04-28 at 17:02 +0100, Pedro Alves wrote:
> > > On Tuesday 28 April 2009 15:45:01, Danny Backx wrote:
> > > > I don't think gdb currently has this capability (to inspect postmortem
> > > > dump files) for debugging CE applications.
> > [..]
> > > If you mean adding gdb support to load minidump files, I can think
> > > of several ways to skin that cat:
> > [..]
> > > 2) teach bfd proper about minidumps, and export the register
> > > set info with fake .reg sections, similarly to how we do for elf
> > > core dumps.  If you export the same set of sections GDB expects from
> > > elf cores (e.g., .reg/XXX sections for thread register sets, .module sections
> > > for loaded dlls.), then the GDB tweaks are the same as
> > > for #1 above.  E.g, teach src/gdb/arm-wince-tdep.c how to extract the loaded
> > > dll list from the core's .module sections.
> > > 
> > > 3) Teach GDB about minidumps, by adding a new minidump target_ops,
> > > similar in vein to src/gdb/corelow.c.  GDB doesn't currenly like
> > > having more than on core file target_ops, though.
> > 
> > I was wondering what is the difference between 2 and 3, and which one is
> > preferred ?
[..]
> I wasn't certain how much of a core dump a minidump is when I
> wrote that; I'm now mostly convinced that 2. is the way
> to go.
> 
> 2. meant putting the minidump file parsing code in bfd.  On
> GDB side, things become mostly transparent.  This is the
> prefered approach.  See all the *-core.c files in bfd for
> starters, e.g. cisco-core.c or ptrace-core.c looks small
> enough to base on.  See also bfd/elf.c, and look for bfd_core,
> and look at all the elfcore_grok_* functions to get a
> feel, specifically, you'll be interested in copying
> elfcore_grok_win32pstatus's NOTE_INFO_MODULE handling.
> I've never done this myself, but it looks to me you'll
> need to follow suit and add a new minidump bfd target, in
> a new bfd/minidump-core.c file.

I've done some of that, but I'm stuck, I have questions.

Basically I think I don't find the documentation that says what (from my
new minidump-core.c module) is used to implement things like the
backtrace command.

Here is what it says now :
dannypc: {519} gdb/gdb examples/Ce042809-01.exe
examples/Ce042809-01.kdmp
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=arm-mingw32ce"...
minidump:Exception Address 0x00011178

warning: "/home/danny/src/gdb/gdb/arm/gdb/examples/Ce042809-01.kdmp": no
core file handler recognizes format, using default

warning: core file may not match specified executable file.
Core was generated by `callstack'.
minidump_core_file_failing_signal -> C0000005
Program terminated with signal 11, Segmentation fault.
Can't fetch registers from this type of core file
Can't fetch registers from this type of core file
#0  0x00000000 in ?? ()
(gdb)  bt
#0  0x00000000 in ?? ()
Can't fetch registers from this type of core file
Can't fetch registers from this type of core file
(gdb) q

(Two of the output lines are debugging code.)

I see code that creates ".reg", ".crash", ".data" sections.
Where can I find documentation for that ? Are these the things I need to
do ?

Thanks,

	Danny

-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Pedro Alves]

On Friday 01 May 2009 15:35:13, Danny Backx wrote:

> I've done some of that, but I'm stuck, I have questions.
> 
> Basically I think I don't find the documentation that says what (from my
> new minidump-core.c module) is used to implement things like the
> backtrace command.

GDB needs registers and the thread's stack memory, so it can
unwind the stack, and needs to be tought about the register layout
in those ".reg" sections, as that layout differs between machines,
ABIs, and core file formats.

> warning: "/home/danny/src/gdb/gdb/arm/gdb/examples/Ce042809-01.kdmp": no
> core file handler recognizes format, using default

You need to fix this.  Look at i386-cygwin-tdep.c, and do something
similar in arm-wince-tdep.c.  You'll need to register a new sniffer
in _initialize_arm_wince_tdep (probably for bfd_target_unknown_flavour
or for a new minidump flavour you've added?), the sniffer callback can be
the current `arm_wince_osabi_sniffer' function, but you'll need to
tweak it to return GDB_OSABI_WINCE for your bfd.

Then, in arm_wince_init_abi, you need to register a
callback with set_gdbarch_regset_from_core_section that
extracts the registers from .reg sections.

> I see code that creates ".reg", ".crash", ".data" sections.

> Where can I find documentation for that ? 

Docs, who needs stinking docs ?!!!1 :-)  I don't know where
these things are documented, I just look at src/gdb/corelow.c
to see what GDB expects.

> Are these the things I need to do ?

Yes.  In your case, it will be one '.reg/TID' section per thread.  The
thread that was current at the time of the crash gets a literal ".reg"
section, that lives at the same offset as its corresponding ".reg/TID"
section.  Best look at the sections of a core file on your linux box
to see what I mean (objdump -h core).  You can borrow bits 
from bfd/elf.c:_bfd_elfcore_make_pseudosection for this.

".crash" isn't used by GDB, it seems specific to cisco cores.  It doesn't
matter the name of the section that contains data (only the
section flags count), but ".data" is fine.

I (re-)suggest to also add a ".module" section, containing the
list of loaded dlls, and their load addresses, and then to
copy the bits from i386-cygwin-tdep.c to arm-wince-tdep.c that
parse that section --- start with this bit:

  set_gdbarch_core_xfer_shared_libraries
    (gdbarch, windows_core_xfer_shared_libraries);

It looks to me that this ends sharing significant chunks of code
between arm-wince-tdep.c and i386-cygwin-tdep.c, so we
could move those bits of shared code to windows-tdep.c.

-- 
Pedro Alves

Re: Post mortem debugging for Windows CE

[Danny Backx]

It looks like I'm getting close to something that works. Small session
below. Cleanup and more testing are certainly required.

I'm about to look into your suggestion about .module sections.

How can I observe the effect of that ? How can I test ?

	Danny

dannypc: {923} gdb/gdb examples/callstack.exe examples/Ce042809-01.kdmp
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=arm-mingw32ce"...
Yow Thread 236B284E Ret Addr 26011178
Yow Register RVA 0x00001760
Yow md 0xa1f58b0 sz 48 nread 48 entries 3
Yow cmp(callstack,callstack)

warning: exec file is newer than core file.
Core was generated by `callstack'.
Yow minidump_core_file_failing_signal -> C0000005
Program terminated with signal 11, Segmentation fault.
[New process 594225230]
Yow arm_minidump_regset_from_core_section
Yow arm_minidump_supply_gregset
#0  0x00011178 in fibo (n=2) at callstack.c:11
11                      return *i;
(gdb) bt
#0  0x00011178 in fibo (n=2) at callstack.c:11
#1  0x000111ac in fibo (n=3) at callstack.c:15
#2  0x000111ac in fibo (n=4) at callstack.c:15
#3  0x000111ac in fibo (n=5) at callstack.c:15
#4  0x000111ac in fibo (n=6) at callstack.c:15
#5  0x000111ac in fibo (n=7) at callstack.c:15
#6  0x000111ac in fibo (n=8) at callstack.c:15
#7  0x000111ac in fibo (n=9) at callstack.c:15
#8  0x000111ac in fibo (n=10) at callstack.c:15
#9  0x000111ac in fibo (n=11) at callstack.c:15
#10 0x000111ac in fibo (n=12) at callstack.c:15
#11 0x000111ac in fibo (n=13) at callstack.c:15
#12 0x000111ac in fibo (n=14) at callstack.c:15
#13 0x000111ac in fibo (n=15) at callstack.c:15
#14 0x000111ac in fibo (n=16) at callstack.c:15
#15 0x000111ac in fibo (n=17) at callstack.c:15
#16 0x000111ac in fibo (n=18) at callstack.c:15
#17 0x000111ac in fibo (n=19) at callstack.c:15
#18 0x000111ac in fibo (n=20) at callstack.c:15
#19 0x00011204 in WinMain (a=0x21623686, b=0x0, c=0x2611fed8, d=5) at
callstack.c:22
#20 0x00011134 in WinMainCRTStartup (hInst=0x21623686, hPrevInst=0x0,
lpCmdLine=0x2611fed8, 
    nCmdShow=5)
at /home/danny/src/cegcc/svn.sf.net/cegcc/trunk/cegcc/src/mingw/crt3.c:62
(gdb) up
#1  0x000111ac in fibo (n=3) at callstack.c:15
15              return fibo(n-1) + fibo(n-2);
(gdb) 
#2  0x000111ac in fibo (n=4) at callstack.c:15
15              return fibo(n-1) + fibo(n-2);
(gdb) p n
$1 = 4
(gdb) up
#3  0x000111ac in fibo (n=5) at callstack.c:15
15              return fibo(n-1) + fibo(n-2);
(gdb) up
#4  0x000111ac in fibo (n=6) at callstack.c:15
15              return fibo(n-1) + fibo(n-2);
(gdb) p n
$2 = 6
(gdb) info registers
r0             0x2      2
r1             0x0      0
r2             0x2611fed8       638713560
r3             0x0      0
r4             0x21623686       560084614
r5             0x0      0
r6             0x2611fed8       638713560
r7             0x5      5
r8             0x1ffc9e0        33540576
r9             0x2611fed8       638713560
r10            0x21623686       560084614
r11            0x2611fbd0       638712784
r12            0x2611fb34       638712628
sp             0x2611fbac       0x2611fbac
lr             0x111ac  70060
pc             0x111ac  0x111ac <fibo+96>
fps            0x0      0
cpsr           0x0      0
(gdb) q
dannypc: {924} 

On Fri, 2009-05-01 at 16:37 +0100, Pedro Alves wrote:
> On Friday 01 May 2009 15:35:13, Danny Backx wrote:
> 
> > I've done some of that, but I'm stuck, I have questions.
> > 
> > Basically I think I don't find the documentation that says what (from my
> > new minidump-core.c module) is used to implement things like the
> > backtrace command.
> 
> GDB needs registers and the thread's stack memory, so it can
> unwind the stack, and needs to be tought about the register layout
> in those ".reg" sections, as that layout differs between machines,
> ABIs, and core file formats.
> 
> > warning: "/home/danny/src/gdb/gdb/arm/gdb/examples/Ce042809-01.kdmp": no
> > core file handler recognizes format, using default
> 
> You need to fix this.  Look at i386-cygwin-tdep.c, and do something
> similar in arm-wince-tdep.c.  You'll need to register a new sniffer
> in _initialize_arm_wince_tdep (probably for bfd_target_unknown_flavour
> or for a new minidump flavour you've added?), the sniffer callback can be
> the current `arm_wince_osabi_sniffer' function, but you'll need to
> tweak it to return GDB_OSABI_WINCE for your bfd.
> 
> Then, in arm_wince_init_abi, you need to register a
> callback with set_gdbarch_regset_from_core_section that
> extracts the registers from .reg sections.
> 
> > I see code that creates ".reg", ".crash", ".data" sections.
> 
> > Where can I find documentation for that ? 
> 
> Docs, who needs stinking docs ?!!!1 :-)  I don't know where
> these things are documented, I just look at src/gdb/corelow.c
> to see what GDB expects.
> 
> > Are these the things I need to do ?
> 
> Yes.  In your case, it will be one '.reg/TID' section per thread.  The
> thread that was current at the time of the crash gets a literal ".reg"
> section, that lives at the same offset as its corresponding ".reg/TID"
> section.  Best look at the sections of a core file on your linux box
> to see what I mean (objdump -h core).  You can borrow bits 
> from bfd/elf.c:_bfd_elfcore_make_pseudosection for this.
> 
> ".crash" isn't used by GDB, it seems specific to cisco cores.  It doesn't
> matter the name of the section that contains data (only the
> section flags count), but ".data" is fine.
> 
> I (re-)suggest to also add a ".module" section, containing the
> list of loaded dlls, and their load addresses, and then to
> copy the bits from i386-cygwin-tdep.c to arm-wince-tdep.c that
> parse that section --- start with this bit:
> 
>   set_gdbarch_core_xfer_shared_libraries
>     (gdbarch, windows_core_xfer_shared_libraries);
> 
> It looks to me that this ends sharing significant chunks of code
> between arm-wince-tdep.c and i386-cygwin-tdep.c, so we
> could move those bits of shared code to windows-tdep.c.
> 
-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Pedro Alves]

On Tuesday 05 May 2009 14:30:51, Danny Backx wrote:

> It looks like I'm getting close to something that works. Small session
> below. Cleanup and more testing are certainly required.

This is looking very good!

> I'm about to look into your suggestion about .module sections.
> 
> How can I observe the effect of that ? How can I test ?

The .modules sections would contain the list of loaded modules
at the time of the crash.  So, 'info sharedlibrary' should list the same
list of dlls as the live inferior had, and their load addresses should
be the same.

You could tweak your test app to move the crashing code into a mini
dll you'd write.  Use 'set sysroot' or 'set solib-search-path' command to
point GDB at the host copy of the dll, and load the minidump
into GDB.  GDB should be able to show you the crash site source code,
backtracing from the function in  the dll should work, and
'info sharedlibrary' should report correct data, of course.

-- 
Pedro Alves

Re: Post mortem debugging for Windows CE

[Danny Backx]

After a good deal of actually using gdb to debug itself (as I did a lot
in these days), here is where I am at :

dannypc: {1013} gdb/gdb examples/callstack.exe examples/Ce042809-01.kdmp
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=arm-mingw32ce"...

warning: exec file is newer than core file.
Error while mapping shared library sections:
coredll.dll.0409.mui: No such file or directory.
Error while mapping shared library sections:
coredll.dll: No such file or directory.
Symbol file not found for coredll.dll.0409.mui
Symbol file not found for coredll.dll
Core was generated by `callstack'.
Program terminated with signal 11, Segmentation fault.
[New process 594225230]
#0  0x00011178 in fibo (n=2) at callstack.c:11
11                      return *i;
(gdb) info sharedlibrary 
From        To          Syms Read   Shared Object Library
                        No          coredll.dll.0409.mui
                        No          coredll.dll
(gdb) 

It doesn't print from- and to-addresses in this example. I believe that
this is because solib_map_sections can't read the DLL files (they're
target DLLs from Windows CE, I don't have them on my host). So it fails
in :
  scratch_chan = solib_open (filename, &scratch_pathname);

  if (scratch_chan < 0)
    {
      perror_with_name (filename);
    }

Am I right in concluding that this is the way it should work ?

Does anyone have tricks to produce some fake DLL file for this type of
scenario ?

	Danny


On Tue, 2009-05-05 at 17:53 +0100, Pedro Alves wrote:
> On Tuesday 05 May 2009 14:30:51, Danny Backx wrote:
> 
> > It looks like I'm getting close to something that works. Small session
> > below. Cleanup and more testing are certainly required.
> 
> This is looking very good!
> 
> > I'm about to look into your suggestion about .module sections.
> > 
> > How can I observe the effect of that ? How can I test ?
> 
> The .modules sections would contain the list of loaded modules
> at the time of the crash.  So, 'info sharedlibrary' should list the same
> list of dlls as the live inferior had, and their load addresses should
> be the same.
> 
> You could tweak your test app to move the crashing code into a mini
> dll you'd write.  Use 'set sysroot' or 'set solib-search-path' command to
> point GDB at the host copy of the dll, and load the minidump
> into GDB.  GDB should be able to show you the crash site source code,
> backtracing from the function in  the dll should work, and
> 'info sharedlibrary' should report correct data, of course.
> 
-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Pedro Alves]

On Wednesday 06 May 2009 15:12:10, Danny Backx wrote:
> After a good deal of actually using gdb to debug itself (as I did a lot
> in these days), 

Welcome to the club.  :-)

> here is where I am at : 
> 
> dannypc: {1013} gdb/gdb examples/callstack.exe examples/Ce042809-01.kdmp
> GNU gdb 6.8

 ^^^ you'll need to move to CVS gdb at some point.  GDB's been changing
a lot since 6.8, but, I don't think you'll be much affected.

> coredll.dll.0409.mui: No such file or directory.
> Error while mapping shared library sections:
> coredll.dll: No such file or directory.
> Symbol file not found for coredll.dll.0409.mui
> Symbol file not found for coredll.dll
> Core was generated by `callstack'.
> Program terminated with signal 11, Segmentation fault.
> [New process 594225230]
> #0  0x00011178 in fibo (n=2) at callstack.c:11
> 11                      return *i;
> (gdb) info sharedlibrary 
> >From        To          Syms Read   Shared Object Library
>                         No          coredll.dll.0409.mui
>                         No          coredll.dll
> (gdb) 
> 

> It doesn't print from- and to-addresses in this example. I believe that
> this is because solib_map_sections can't read the DLL files (they're
> target DLLs from Windows CE, I don't have them on my host). So it fails
> in :
>   scratch_chan = solib_open (filename, &scratch_pathname);
> 
>   if (scratch_chan < 0)
>     {
>       perror_with_name (filename);
>     }
> 
> Am I right in concluding that this is the way it should work ?

Looking very good.  I had a vague memory that WinCE gdbserver reports
absolute dlls paths when doing live debugging.  Does it not?  What
does "info sharedlibrary" say when doing live debugging?

When paths are reported as relative (coredll.dll vs
/Windows/coredll.dll), then pointing gdb at a local host copy
of dlls with the 'set sysroot' command doesn't help.  You'll
need 'set solib-search-path'.

 (gdb) help set sysroot
 Set an alternate system root.
 The system root is used to load absolute shared library symbol files.
 For other (relative) files, you can add directories using
 `set solib-search-path'.

 (gdb) help set solib-search-path
 Set the search path for loading non-absolute shared library symbol files.
 This takes precedence over the environment variables PATH and LD_LIBRARY_PATH.

[ coredll.dll can't be (easilly) extracted from ROM, but, you're not
expected to debug coredll.dll anyway. ]

> Does anyone have tricks to produce some fake DLL file for this type of
> scenario ?

Completely untested, but something as simple as this?

bad.c:
int crashme (int *foofoo)
{
  return *foofoo;
}

arm-mingw32ce-gcc bad.c -o bad.dll -O0 -g3 -shared

main.c:
int main()
{
   crashme (NULL);
}

arm-mingw32ce-gcc main.c -o main.exe -O0 -g3 -L. -lbad

?

BTW, does "info threads" show the correct list of threads?

-- 
Pedro Alves

Re: Post mortem debugging for Windows CE

[Danny Backx]

On Wed, 2009-05-06 at 15:35 +0100, Pedro Alves wrote:
> On Wednesday 06 May 2009 15:12:10, Danny Backx wrote:
> > It doesn't print from- and to-addresses in this example. I believe
> that
> > this is because solib_map_sections can't read the DLL files (they're
> > target DLLs from Windows CE, I don't have them on my host). So it
> fails
> > in :
> > scratch_chan = solib_open (filename, &scratch_pathname);
> > 
> > if (scratch_chan < 0)
> > {
> > perror_with_name (filename);
> > }
> > 
> > Am I right in concluding that this is the way it should work ?
>
> Looking very good. I had a vague memory that WinCE gdbserver reports
> absolute dlls paths when doing live debugging. Does it not? What
> does "info sharedlibrary" say when doing live debugging?

I've done a couple of tests. (See below, lengthy.) The bottom line :
- I should have used solib-search-path the first time you told me, not
  after the second time :-)
- without setting solib-search-path, the output for both live and post-
  mortem (minidump) sessions is a list of DLLs without address range.
- with setting solib-search-path, the output of both live and postmortem
  sessions shows addresses for those DLLs found

> BTW, does "info threads" show the correct list of threads?

Not yet. The place where I found the register contents until now was not
'thread-safe' - there's only one set of registers there.

The only additional info I found points to memory regions that the minidump
doesn't contain a copy of.

Further poking on MSDN reveals a table that explains the contents of "context
dumps" (limited in size) vs. "system dumps" and "complete dumps".
See http://msdn.microsoft.com/en-us/library/ms939598.aspx .

Also http://msdn.microsoft.com/en-us/library/ms939607.aspx describes
registry settings to influence dump handling. Can't find a way to change
the type of dump though.

	Danny


dannypc: {1017} gdb/gdb ~/tmp/arm/test/pedro/main.exe 
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=arm-mingw32ce"...
(gdb) target remote ipaq:9999
Remote debugging using ipaq:9999
[New Thread 659080034]
Yow #### solib_map_sections
Error while mapping shared library sections:
bad.dll: No such file or directory.
Yow #### solib_map_sections
Error while mapping shared library sections:
coredll.dll.0409.mui: No such file or directory.
Yow #### solib_map_sections
Error while mapping shared library sections:
coredll.dll: No such file or directory.
Symbol file not found for bad.dll
Symbol file not found for coredll.dll.0409.mui
Symbol file not found for coredll.dll
WinMainCRTStartup (hInst=0x2564721e, hPrevInst=0x0,
lpCmdLine=0x2012fed8, nCmdShow=5)

at /home/danny/src/cegcc/svn.sf.net/cegcc/trunk/cegcc/src/mingw/crt3.c:35
35      {
(gdb) l main
1       #include <stdio.h>
2       int main()
3       {
4                  crashme (NULL);
5       }
(gdb) break main
Breakpoint 1 at 0x11060: file main.c, line 3.
(gdb) c
Continuing.

Breakpoint 1, main () at main.c:3
3       {
(gdb) info sharedlibrary 
From        To          Syms Read   Shared Object Library
                        No          bad.dll
                        No          coredll.dll.0409.mui
                        No          coredll.dll
(gdb) c
Continuing.
Data Abort: Thread=85eec650 Proc=8034e070 'main.exe'
AKY=00008001 PC=01451098(bad.dll+0x00001098) RA=0001106c(main.exe
+0x0000106c) BVA=20000000 FSR=00000007

Program received signal SIGSEGV, Segmentation fault.
0x01451098 in ?? ()
(gdb) q
The program is running.  Exit anyway? (y or n) y
dannypc: {1018} gdb/gdb ~/tmp/arm/test/pedro/main.exe
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=arm-mingw32ce"...

(gdb) set solib-search-path /home/danny/tmp/arm/test/pedro
(gdb) break main
Breakpoint 1 at 0x11060: file main.c, line 3.
(gdb) c
The program is not being run.
(gdb) target remote ipaq:9999
Remote debugging using ipaq:9999
[New Thread 1154232258]
Yow #### solib_map_sections
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow #### solib_map_sections
Error while mapping shared library sections:
coredll.dll.0409.mui: No such file or directory.
Yow #### solib_map_sections
Error while mapping shared library sections:
coredll.dll: No such file or directory.
Reading symbols from /home/danny/tmp/arm/test/pedro/bad.dll...done.
Loaded symbols for /home/danny/tmp/arm/test/pedro/bad.dll
Symbol file not found for coredll.dll.0409.mui
Symbol file not found for coredll.dll
WinMainCRTStartup (hInst=0x45da2d4a, hPrevInst=0x0,
lpCmdLine=0x2012fed8, nCmdShow=5)

at /home/danny/src/cegcc/svn.sf.net/cegcc/trunk/cegcc/src/mingw/crt3.c:35
35      {
(gdb) break main
Note: breakpoint 1 also set at pc 0x11060.
Breakpoint 2 at 0x11060: file main.c, line 3.
(gdb) c
Continuing.

Breakpoint 1, main () at main.c:3
3       {
(gdb) info sharedlibrary 
From        To          Syms Read   Shared Object Library
0x01451000  0x01456044
Yes         /home/danny/tmp/arm/test/pedro/bad.dll
                        No          coredll.dll.0409.mui
                        No          coredll.dll
(gdb) q
The program is running.  Exit anyway? (y or n) y
dannypc: {1019} rsh ipaq:/temp/main
ipaq:/temp/main: Unknown host
dannypc: {1020} rsh ipaq /temp/main
dannypc: {1021} rsh ipaq:/temp/ls /windows/system/dumpfiles/\*
ipaq:/temp/ls: Unknown host
dannypc: {1022} rsh ipaq /temp/ls /windows/system/dumpfiles/\*
ls /windows/system/dumpfiles/* :
0 [Ce041409-01] dir
1 [Ce042709-01] dir
2 [CabFiles] dir
3 [Ce042809-01] dir
4 [Ce042909-01] dir
5 [Ce042909-02] dir
6 [Ce050209-01] dir
7 [Ce050409-01] dir
8 [Ce050409-02] dir
9 [Ce050609-01] dir
dannypc: {1023} rsh
ipaq /temp/copydump /windows/system/dumpfiles/Ce050609-01/Ce050609-01k.dmp
dannypc: {1024} rcp ipaq:/temp/Ce050609-01.kdmp .
rcp: /temp/Ce050609-01.kdmp: No such file or directory
dannypc: {1025} rsh
ipaq /temp/copydump /windows/system/dumpfiles/Ce050609-01/Ce050609-01.kdmp
dannypc: {1026} rcp ipaq:/temp/Ce050609-01.kdmp .
dannypc: {1027} gdb/gdb ~/tmp/arm/test/pedro/main.exe Ce050609-01.kdmp 
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=arm-mingw32ce"...
Yow module NK.EXE base 80001000 sz 0006C000
Yow pass ptr 0x881d970 -> module NK.EXE
Yow module main base 1E010000 sz 00012000
Yow pass ptr 0x881d9a0 -> module main
Yow module bad.dll base 01450000 sz 00010000
Yow pass ptr 0x880f040 -> module bad.dll
Yow module coredll.dll.0409.mui base 7FFE0000 sz 00015000
Yow pass ptr 0x881e0a0 -> module coredll.dll.0409.mui
Yow module coredll.dll base 03F4E000 sz 00096000
Yow pass ptr 0x881d8c8 -> module coredll.dll
Yow ##### win32_core_xfer_shared_libraries
Yow module name (NK.EXE) base_addr 0x80001000
Yow module name (main) base_addr 0x1e010000
Yow module name (bad.dll) base_addr 0x1450000
Yow ##### win32_xfer_shared_library (bad.dll) load 0x1450000
Yow module name (coredll.dll.0409.mui) base_addr 0x7ffe0000
Yow ##### win32_xfer_shared_library (coredll.dll.0409.mui) load
0x7ffe0000
Yow module name (coredll.dll) base_addr 0x3f4e000
Yow ##### win32_xfer_shared_library (coredll.dll) load 0x3f4e000
Yow ##### win32_core_xfer_shared_libraries
Yow module name (NK.EXE) base_addr 0x80001000
Yow module name (main) base_addr 0x1e010000
Yow module name (bad.dll) base_addr 0x1450000
Yow ##### win32_xfer_shared_library (bad.dll) load 0x1450000
Yow module name (coredll.dll.0409.mui) base_addr 0x7ffe0000
Yow ##### win32_xfer_shared_library (coredll.dll.0409.mui) load
0x7ffe0000
Yow module name (coredll.dll) base_addr 0x3f4e000
Yow ##### win32_xfer_shared_library (coredll.dll) load 0x3f4e000
Yow #### solib_map_sections
Error while mapping shared library sections:
bad.dll: No such file or directory.
Yow #### solib_map_sections
Error while mapping shared library sections:
coredll.dll.0409.mui: No such file or directory.
Yow #### solib_map_sections
Error while mapping shared library sections:
coredll.dll: No such file or directory.
Symbol file not found for bad.dll
Symbol file not found for coredll.dll.0409.mui
Symbol file not found for coredll.dll
Core was generated by `main'.
Yow minidump_core_file_failing_signal -> C0000005
Program terminated with signal 11, Segmentation fault.
[New process 1182062390]
Yow arm_minidump_regset_from_core_section
Yow arm_minidump_supply_gregset
#0  0x01451098 in ?? ()
(gdb) set solib-search-path /home/danny/tmp/arm/test/pedro
Yow ##### win32_core_xfer_shared_libraries
Yow module name (NK.EXE) base_addr 0x80001000
Yow module name (main) base_addr 0x1e010000
Yow module name (bad.dll) base_addr 0x1450000
Yow ##### win32_xfer_shared_library (bad.dll) load 0x1450000
Yow module name (coredll.dll.0409.mui) base_addr 0x7ffe0000
Yow ##### win32_xfer_shared_library (coredll.dll.0409.mui) load
0x7ffe0000
Yow module name (coredll.dll) base_addr 0x3f4e000
Yow ##### win32_xfer_shared_library (coredll.dll) load 0x3f4e000
Yow ##### win32_core_xfer_shared_libraries
Yow module name (NK.EXE) base_addr 0x80001000
Yow module name (main) base_addr 0x1e010000
Yow module name (bad.dll) base_addr 0x1450000
Yow ##### win32_xfer_shared_library (bad.dll) load 0x1450000
Yow module name (coredll.dll.0409.mui) base_addr 0x7ffe0000
Yow ##### win32_xfer_shared_library (coredll.dll.0409.mui) load
0x7ffe0000
Yow module name (coredll.dll) base_addr 0x3f4e000
Yow ##### win32_xfer_shared_library (coredll.dll) load 0x3f4e000
Yow #### solib_map_sections
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow ### solib_target_relocate_section_addresses
Yow #### solib_map_sections
Error while mapping shared library sections:
coredll.dll.0409.mui: No such file or directory.
Yow #### solib_map_sections
Error while mapping shared library sections:
coredll.dll: No such file or directory.
Reading symbols from /home/danny/tmp/arm/test/pedro/bad.dll...done.
Loaded symbols for /home/danny/tmp/arm/test/pedro/bad.dll
Symbol file not found for coredll.dll.0409.mui
Symbol file not found for coredll.dll
(gdb) info sharedlibrary 
Yow ##### win32_core_xfer_shared_libraries
Yow module name (NK.EXE) base_addr 0x80001000
Yow module name (main) base_addr 0x1e010000
Yow module name (bad.dll) base_addr 0x1450000
Yow ##### win32_xfer_shared_library (bad.dll) load 0x1450000
Yow module name (coredll.dll.0409.mui) base_addr 0x7ffe0000
Yow ##### win32_xfer_shared_library (coredll.dll.0409.mui) load
0x7ffe0000
Yow module name (coredll.dll) base_addr 0x3f4e000
Yow ##### win32_xfer_shared_library (coredll.dll) load 0x3f4e000
Yow ##### win32_core_xfer_shared_libraries
Yow module name (NK.EXE) base_addr 0x80001000
Yow module name (main) base_addr 0x1e010000
Yow module name (bad.dll) base_addr 0x1450000
Yow ##### win32_xfer_shared_library (bad.dll) load 0x1450000
Yow module name (coredll.dll.0409.mui) base_addr 0x7ffe0000
Yow ##### win32_xfer_shared_library (coredll.dll.0409.mui) load
0x7ffe0000
Yow module name (coredll.dll) base_addr 0x3f4e000
Yow ##### win32_xfer_shared_library (coredll.dll) load 0x3f4e000
From        To          Syms Read   Shared Object Library
0x01451000  0x01456044
Yes         /home/danny/tmp/arm/test/pedro/bad.dll
                        No          coredll.dll.0409.mui
                        No          coredll.dll
(gdb) q
dannypc: {1028} 



-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Pedro Alves]

On Wednesday 06 May 2009 21:04:16, Danny Backx wrote:
> I've done a couple of tests. (See below, lengthy.) The bottom line :
> - I should have used solib-search-path the first time you told me, not
>   after the second time :-)
> - without setting solib-search-path, the output for both live and post-
>   mortem (minidump) sessions is a list of DLLs without address range.
> - with setting solib-search-path, the output of both live and postmortem
>   sessions shows addresses for those DLLs found

Right, that is because GDB only shows that info if it finds
symbols for the shared library:

 From        To          Syms Read   Shared Object Library
                         No          bad.dll

                         ^^

vs:

 From        To          Syms Read   Shared Object Library
 0x01451000  0x01456044  Yes         /home/danny/tmp/arm/test/pedro/bad.dll
                         ^^^

On Wednesday 06 May 2009 21:04:16, Danny Backx wrote:
> > BTW, does "info threads" show the correct list of threads?
> 
> Not yet. The place where I found the register contents until now was not
> 'thread-safe' - there's only one set of registers there.
> 
> The only additional info I found points to memory regions that the minidump
> doesn't contain a copy of.
> 
> Further poking on MSDN reveals a table that explains the contents of "context
> dumps" (limited in size) vs. "system dumps" and "complete dumps".
> See http://msdn.microsoft.com/en-us/library/ms939598.aspx .

Ah, I see.  If the data ain't there, there's nothing we can do
about it.

> Also http://msdn.microsoft.com/en-us/library/ms939607.aspx describes
> registry settings to influence dump handling. Can't find a way to change
> the type of dump though.

I think that if you clean this up and post it, it will be nice
addition already.  You'll need the copyright assignment issues sorted
first before we can accept the code though, for binutils (bfd)
and gdb.

A few suggestions:

 - Please forgive me if you know this already.
   Your minidump bfd code should work on all hosts, 64-bit, 32-bit,
   big endian or little endian.  This means that code like:

    typedef struct FooStructureMSDefinesToBePresentInTheFile
    {
       DWORD foo;
       char b;
       int c;
    } FooStructureMSDefinesToBePresentInTheFile;

    FooStructureMSDefinesToBePresentInTheFile bar;

    read (fildes, &bar, sizeof (bar));

   ... is unacceptable.  You need to use the bfd_get_32 and friends
   to parse the data on the file.

 - Be sure to follow the coding standards:

    http://www.gnu.org/prep/standards/html_node/Formatting.html

    Standard nit-picks:

     space between function name and '('.
     double space after periods in comments.
     tabs are prefered to spaces.

 - Don't forget a ChangeLog entry.

-- 
Pedro Alves

Re: Post mortem debugging for Windows CE

[Danny Backx]

On Wed, 2009-05-06 at 21:48 +0100, Pedro Alves wrote:
> A few suggestions:
> 
>  - Please forgive me if you know this already.
>    Your minidump bfd code should work on all hosts, 64-bit, 32-bit,
>    big endian or little endian.  This means that code like:
[..]
>    ... is unacceptable.  You need to use the bfd_get_32 and friends
>    to parse the data on the file.

I'll start doing all that. I didn't know about avoiding structs so
this'll be some work.

Should they be avoided completely ? I wrote code like the snippet below.
Should the sizeof (CEDUMP_THREAD_CALL_STACK_LIST) be replaced ? Should
the struct definitions be gone completely, or still be there in
comment ?

Also this'll work for CE based minidumps now, not the ones from desktop
Windows. This may be as simple as getting the code to handle not only
the 
        ceStreamThreadCallStackList = 0x8007
but also the desktop value. But life is usually not that simple.

	Danny



  rva = minidump_core_locate_stream(abfd, ceStreamThreadCallStackList);
  if (bfd_seek (abfd, (file_ptr) rva, SEEK_SET) != 0)
    return;
  nread = bfd_bread (&tcsl, (bfd_size_type) sizeof
(CEDUMP_THREAD_CALL_STACK_LIST), abfd);
  if (nread != sizeof (CEDUMP_THREAD_CALL_STACK_LIST))
    {
      if (bfd_get_error () != bfd_error_system_call)
        bfd_set_error (bfd_error_wrong_format);
      return;
    }

  /*
   * Need to read all the CEDUMP_THREAD_CALL_STACK entries,
   * they're just after the CEDUMP_THREAD_CALL_STACK_LIST.
   * The CEDUMP_THREAD_CALL_STACK_FRAME fields are in potentially other
places though.
   * So allocate space for enough CEDUMP_THREAD_CALL_STACK_LIST entries.
   */
  sz = tcsl.NumberOfEntries * sizeof(CEDUMP_THREAD_CALL_STACK);
  tcs = bfd_malloc(sz);
  nread = bfd_bread (tcs, (bfd_size_type) sz, abfd);
  if (nread != sz)
    {
      if (bfd_get_error () != bfd_error_system_call)
        bfd_set_error (bfd_error_wrong_format);
      return;
    }




-- 
Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info

Re: Post mortem debugging for Windows CE

[Pedro Alves]

On Thursday 07 May 2009 02:21:04, Danny Backx wrote:
> On Wed, 2009-05-06 at 21:48 +0100, Pedro Alves wrote:
> > A few suggestions:
> > 
> >  - Please forgive me if you know this already.
> >    Your minidump bfd code should work on all hosts, 64-bit, 32-bit,
> >    big endian or little endian.  This means that code like:
> [..]
> >    ... is unacceptable.  You need to use the bfd_get_32 and friends
> >    to parse the data on the file.
> 
> I'll start doing all that. I didn't know about avoiding structs so
> this'll be some work.
> 
> Should they be avoided completely ? I wrote code like the snippet below.
> Should the sizeof (CEDUMP_THREAD_CALL_STACK_LIST) be replaced ? Should
> the struct definitions be gone completely, or still be there in
> comment ?

Take a look at src/include/coff/external.h and
src/include/coff/internal.h, or grep for Internal and External
in src/include/elf/, or grep for "swap" in bfd.  The external variants
are the ones that are read out of file.  The internal variants
are the ones used internally.  Presumably, you won't have that much
code, so you may chose to always use "external" types, and extract
the fields with bfd_get_foo.

> Also this'll work for CE based minidumps now, not the ones from desktop
> Windows. This may be as simple as getting the code to handle not only
> the 
>         ceStreamThreadCallStackList = 0x8007
> but also the desktop value. But life is usually not that simple.

If this is not useful to you, then you don't have to implement it.
I'm sure someone else having an interest in desktop Windows will
take of adding such functionality.

>   rva = minidump_core_locate_stream(abfd, ceStreamThreadCallStackList);
>   if (bfd_seek (abfd, (file_ptr) rva, SEEK_SET) != 0)
>     return;
>   nread = bfd_bread (&tcsl, (bfd_size_type) sizeof
> (CEDUMP_THREAD_CALL_STACK_LIST), abfd);
>   if (nread != sizeof (CEDUMP_THREAD_CALL_STACK_LIST))
>     {
>       if (bfd_get_error () != bfd_error_system_call)
>         bfd_set_error (bfd_error_wrong_format);
>       return;
>     }
> 
>   /*
>    * Need to read all the CEDUMP_THREAD_CALL_STACK entries,
>    * they're just after the CEDUMP_THREAD_CALL_STACK_LIST.
>    * The CEDUMP_THREAD_CALL_STACK_FRAME fields are in potentially other
> places though.
>    * So allocate space for enough CEDUMP_THREAD_CALL_STACK_LIST entries.
>    */
>   sz = tcsl.NumberOfEntries * sizeof(CEDUMP_THREAD_CALL_STACK);

Right, this is exactly what you should *not* do.  That
'tcsl.NumberOfEntries' field will return garbage if e.g., the
host machine running gdb is big endian, while the minidump data
is supposedly stored in little endian (ARM).  Same for hidden
padding the compiler may insert in the structure, influencing
the tcsl fields offsets, and the size of the structure in the
view of the host.

-- 
Pedro Alves

Re: Post mortem debugging for Windows CE

[Danny Backx]

[-- Attachment #1: Type: text/plain, Size: 3048 bytes --]

Here is a first cut of my work. Please comment.

	Danny

On Sun, 2009-05-10 at 20:21 +0100, Pedro Alves wrote:
> On Thursday 07 May 2009 02:21:04, Danny Backx wrote:
> > On Wed, 2009-05-06 at 21:48 +0100, Pedro Alves wrote:
> > > A few suggestions:
> > > 
> > >  - Please forgive me if you know this already.
> > >    Your minidump bfd code should work on all hosts, 64-bit, 32-bit,
> > >    big endian or little endian.  This means that code like:
> > [..]
> > >    ... is unacceptable.  You need to use the bfd_get_32 and friends
> > >    to parse the data on the file.
> > 
> > I'll start doing all that. I didn't know about avoiding structs so
> > this'll be some work.
> > 
> > Should they be avoided completely ? I wrote code like the snippet below.
> > Should the sizeof (CEDUMP_THREAD_CALL_STACK_LIST) be replaced ? Should
> > the struct definitions be gone completely, or still be there in
> > comment ?
> 
> Take a look at src/include/coff/external.h and
> src/include/coff/internal.h, or grep for Internal and External
> in src/include/elf/, or grep for "swap" in bfd.  The external variants
> are the ones that are read out of file.  The internal variants
> are the ones used internally.  Presumably, you won't have that much
> code, so you may chose to always use "external" types, and extract
> the fields with bfd_get_foo.
> 
> > Also this'll work for CE based minidumps now, not the ones from desktop
> > Windows. This may be as simple as getting the code to handle not only
> > the 
> >         ceStreamThreadCallStackList = 0x8007
> > but also the desktop value. But life is usually not that simple.
> 
> If this is not useful to you, then you don't have to implement it.
> I'm sure someone else having an interest in desktop Windows will
> take of adding such functionality.
> 
> >   rva = minidump_core_locate_stream(abfd, ceStreamThreadCallStackList);
> >   if (bfd_seek (abfd, (file_ptr) rva, SEEK_SET) != 0)
> >     return;
> >   nread = bfd_bread (&tcsl, (bfd_size_type) sizeof
> > (CEDUMP_THREAD_CALL_STACK_LIST), abfd);
> >   if (nread != sizeof (CEDUMP_THREAD_CALL_STACK_LIST))
> >     {
> >       if (bfd_get_error () != bfd_error_system_call)
> >         bfd_set_error (bfd_error_wrong_format);
> >       return;
> >     }
> > 
> >   /*
> >    * Need to read all the CEDUMP_THREAD_CALL_STACK entries,
> >    * they're just after the CEDUMP_THREAD_CALL_STACK_LIST.
> >    * The CEDUMP_THREAD_CALL_STACK_FRAME fields are in potentially other
> > places though.
> >    * So allocate space for enough CEDUMP_THREAD_CALL_STACK_LIST entries.
> >    */
> >   sz = tcsl.NumberOfEntries * sizeof(CEDUMP_THREAD_CALL_STACK);
> 
> Right, this is exactly what you should *not* do.  That
> 'tcsl.NumberOfEntries' field will return garbage if e.g., the
> host machine running gdb is big endian, while the minidump data
> is supposedly stored in little endian (ARM).  Same for hidden
> padding the compiler may insert in the structure, influencing
> the tcsl fields offsets, and the size of the structure in the
> view of the host.
> 

[-- Attachment #2: diffs-v1.gz --]
[-- Type: application/x-gzip, Size: 14649 bytes --]

Re: Post mortem debugging for Windows CE

[Danny Backx]

I had not included material for interested parties to actually do a
little test with.

I won't put such executables on the mailing list, instead I put it up on
my site :
 http://danny.backx.info/download/gdb/gdb-minidump-example.tar.gz

The file is 11446 bytes long, and contains :
pavilion: {185} tar tvfz /tmp/gdb-minidump-example.tar.gz
-rw-rw-r-- danny/danny     371 2009-04-28 19:37 examples/callstack.c
-rw-rw-r-- danny/danny   24586 2009-04-30 15:07 examples/callstack.exe
-rw------- danny/danny   11480 2009-04-30 15:01
examples/Ce042809-01.kdmp

This produces output such as :

pavilion: {186} gdb/gdb examples/callstack.exe
examples/Ce042809-01.kdmp 
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=arm-mingw32ce"...

warning: exec file is newer than core file.
Error while mapping shared library sections:
coredll.dll.0409.mui: No such file or directory.
Error while mapping shared library sections:
coredll.dll: No such file or directory.
Symbol file not found for coredll.dll.0409.mui
Symbol file not found for coredll.dll
Core was generated by `callstack'.
Program terminated with signal 11, Segmentation fault.
[New process 594225230]
#0  0x00011178 in fibo (n=2) at callstack.c:11
11                      return *i;
(gdb) where
#0  0x00011178 in fibo (n=2) at callstack.c:11
#1  0x000111ac in fibo (n=3) at callstack.c:15
#2  0x000111ac in fibo (n=4) at callstack.c:15
#3  0x000111ac in fibo (n=5) at callstack.c:15
#4  0x000111ac in fibo (n=6) at callstack.c:15
#5  0x000111ac in fibo (n=7) at callstack.c:15
#6  0x000111ac in fibo (n=8) at callstack.c:15
#7  0x000111ac in fibo (n=9) at callstack.c:15
#8  0x000111ac in fibo (n=10) at callstack.c:15
#9  0x000111ac in fibo (n=11) at callstack.c:15
#10 0x000111ac in fibo (n=12) at callstack.c:15
#11 0x000111ac in fibo (n=13) at callstack.c:15
#12 0x000111ac in fibo (n=14) at callstack.c:15
#13 0x000111ac in fibo (n=15) at callstack.c:15
#14 0x000111ac in fibo (n=16) at callstack.c:15
#15 0x000111ac in fibo (n=17) at callstack.c:15
#16 0x000111ac in fibo (n=18) at callstack.c:15
#17 0x000111ac in fibo (n=19) at callstack.c:15
#18 0x000111ac in fibo (n=20) at callstack.c:15
#19 0x00011204 in WinMain (a=0x21623686, b=0x0, c=0x2611fed8, d=5) at
callstack.c:22
#20 0x00011134 in WinMainCRTStartup (hInst=0x21623686, hPrevInst=0x0,
lpCmdLine=0x2611fed8, 
    nCmdShow=5)
at /home/danny/src/cegcc/svn.sf.net/cegcc/trunk/cegcc/src/mingw/crt3.c:62


Hope this helps.

	Danny

On Mon, 2009-05-11 at 16:03 +0200, Danny Backx wrote:
> Here is a first cut of my work. Please comment.
> 
> 	Danny
> 
> On Sun, 2009-05-10 at 20:21 +0100, Pedro Alves wrote:
> > On Thursday 07 May 2009 02:21:04, Danny Backx wrote:
> > > On Wed, 2009-05-06 at 21:48 +0100, Pedro Alves wrote:
> > > > A few suggestions:
> > > > 
> > > >  - Please forgive me if you know this already.
> > > >    Your minidump bfd code should work on all hosts, 64-bit, 32-bit,
> > > >    big endian or little endian.  This means that code like:
> > > [..]
> > > >    ... is unacceptable.  You need to use the bfd_get_32 and friends
> > > >    to parse the data on the file.
> > > 
> > > I'll start doing all that. I didn't know about avoiding structs so
> > > this'll be some work.
> > > 
> > > Should they be avoided completely ? I wrote code like the snippet below.
> > > Should the sizeof (CEDUMP_THREAD_CALL_STACK_LIST) be replaced ? Should
> > > the struct definitions be gone completely, or still be there in
> > > comment ?
> > 
> > Take a look at src/include/coff/external.h and
> > src/include/coff/internal.h, or grep for Internal and External
> > in src/include/elf/, or grep for "swap" in bfd.  The external variants
> > are the ones that are read out of file.  The internal variants
> > are the ones used internally.  Presumably, you won't have that much
> > code, so you may chose to always use "external" types, and extract
> > the fields with bfd_get_foo.
> > 
> > > Also this'll work for CE based minidumps now, not the ones from desktop
> > > Windows. This may be as simple as getting the code to handle not only
> > > the 
> > >         ceStreamThreadCallStackList = 0x8007
> > > but also the desktop value. But life is usually not that simple.
> > 
> > If this is not useful to you, then you don't have to implement it.
> > I'm sure someone else having an interest in desktop Windows will
> > take of adding such functionality.
> > 
> > >   rva = minidump_core_locate_stream(abfd, ceStreamThreadCallStackList);
> > >   if (bfd_seek (abfd, (file_ptr) rva, SEEK_SET) != 0)
> > >     return;
> > >   nread = bfd_bread (&tcsl, (bfd_size_type) sizeof
> > > (CEDUMP_THREAD_CALL_STACK_LIST), abfd);
> > >   if (nread != sizeof (CEDUMP_THREAD_CALL_STACK_LIST))
> > >     {
> > >       if (bfd_get_error () != bfd_error_system_call)
> > >         bfd_set_error (bfd_error_wrong_format);
> > >       return;
> > >     }
> > > 
> > >   /*
> > >    * Need to read all the CEDUMP_THREAD_CALL_STACK entries,
> > >    * they're just after the CEDUMP_THREAD_CALL_STACK_LIST.
> > >    * The CEDUMP_THREAD_CALL_STACK_FRAME fields are in potentially other
> > > places though.
> > >    * So allocate space for enough CEDUMP_THREAD_CALL_STACK_LIST entries.
> > >    */
> > >   sz = tcsl.NumberOfEntries * sizeof(CEDUMP_THREAD_CALL_STACK);
> > 
> > Right, this is exactly what you should *not* do.  That
> > 'tcsl.NumberOfEntries' field will return garbage if e.g., the
> > host machine running gdb is big endian, while the minidump data
> > is supposedly stored in little endian (ARM).  Same for hidden
> > padding the compiler may insert in the structure, influencing
> > the tcsl fields offsets, and the size of the structure in the
> > view of the host.
> >