Custom core file

Custom core file

[Nikolay Martyanov]

Hello guys!

I have a self-written bare-metal hypervisor for x86 arch and I'd like to 
perform postmortem debugging of it's core (not VM, hypervisor itself!).
So the idea is to save physical memory state and later use GDB to 
interpret it.

As I understand, GDB can interpret only `core files`. But I didn't found 
a lot of info about this file format.
Some sources say it should be a specific ELF file with several mandatory 
".note" sections.
Others say `core file` can be in any format and adaptation should be 
done on BFD-level of GDB.

I looked through the code in corelow.c. Now my undestanding is somthing 
like:

1. Save data section of my hypervisor
2. Save current stack state of my hypervisor
3. Save current registers state (MSR?)
4. Define "sys/user.h"-like header with layout description of above 
mentioned structures
5. Using this header, implement function analog to 
`trad_unix_core_file_p()` in file analog to `trad-core.c` (It translates 
my `struct user` to `struct trad_core_struct`)
6. Implement functions like `supply_*regset()` in file like 
`i386-myhyper-nat.c`
7. ??? (magic I didn't realized yet)
8. Profit!

Note, steps 1-3 say nothing about ELF-format. It embarrasses me...

So my question is: am I looking in a right direction? Can you point me 
in to the issues in the plan (or even my understanding)?
Or may be there are any other ways to feed a raw physical memory to GDB?

Thanks,
Nikolay

Re: Custom core file

[Duane Ellis]

> I have a self-written bare-metal hypervisor for x86 arch and I'd like to perform postmortem debugging of it's core (not VM, hypervisor itself!).
> So the idea is to save physical memory state and later use GDB to interpret it.

[snip]

> Or may be there are any other ways to feed a raw physical memory to GDB?


here’s a method

GDB has a built in Python interpreter.

GDB has commands such as “dump” and “restore” that allow you to copy a binary file into memory.

Thus - you can use a bit of python to extract your custom core dump
For example you might use Python to unpack your custom core file into various temp binary files
Then in the same python script execute dump/restore commands to load these into memory

http://www.delorie.com/gnu/docs/gdb/gdb_69.html

See:   restore FILENAME binary …..

Re: Custom core file

[Jan Kratochvil]

On Wed, 28 Sep 2016 15:31:12 +0200, Nikolay Martyanov wrote:
> I have a self-written bare-metal hypervisor for x86 arch and I'd like to
> perform postmortem debugging of it's core (not VM, hypervisor itself!).
> So the idea is to save physical memory state and later use GDB to interpret
> it.

I still do not understand what is the goal.

"not VM, hypervisor itself!" would say that running "/usr/bin/gcore "+getpid()
on the hypervisor process would do the job.

But then "the idea is to save physical memory state" would suggest me you want
to dump the guest VM - like what guest kdump does or what is in guest Linux
kernel /proc/kcore (where it is without the physical memory).


Jan

Re: Custom core file

[Nikolay Martyanov]

On 09/28/2016 05:58 PM, Jan Kratochvil wrote:
> On Wed, 28 Sep 2016 15:31:12 +0200, Nikolay Martyanov wrote:
>> I have a self-written bare-metal hypervisor for x86 arch and I'd like to
>> perform postmortem debugging of it's core (not VM, hypervisor itself!).
>> So the idea is to save physical memory state and later use GDB to interpret
>> it.
> I still do not understand what is the goal.
>
> "not VM, hypervisor itself!" would say that running "/usr/bin/gcore "+getpid()
> on the hypervisor process would do the job.
Bare-metal hypervisor doesn't have a representation in process tree of 
any guest VM. I do not have running OS where I can do `/usr/bin/gcore`. 
It's Xen alike hypervisor =)
> But then "the idea is to save physical memory state" would suggest me you want
> to dump the guest VM - like what guest kdump does or what is in guest Linux
> kernel /proc/kcore (where it is without the physical memory).
>
>
> Jan
Idea is very similar to `kdump`, you are right. But instead of Linux 
kernel I have a hypervisor running on a bare-metal. So, in some point of 
view, I try to implement my own kdump. And it will dump not Linux kernel 
mem, but mem of hypervisor.

Re: Custom core file

[Jan Kratochvil]

On Wed, 28 Sep 2016 17:18:35 +0200, Nikolay Martyanov wrote:
> Idea is very similar to `kdump`, you are right. But instead of Linux kernel
> I have a hypervisor running on a bare-metal. So, in some point of view, I
> try to implement my own kdump. And it will dump not Linux kernel mem, but
> mem of hypervisor.

Ah, sorry I forgot you wrote that "bare-metal" there.

OK, so core files are not standardized the way executables+shlibs are by the
standards of ELF-generic and ELF-x86_64 or ELF-x86 addons. You can either
invent your own format (based on ELF or not) or - what I suggest - is to just
mimic the Linux kernel process format.

In fact /proc/kcore and kdump also create a core file looking as a userland
process to make it easier for consumers to read it, despite the content is
unrelated to userland processes.

I am not aware of official Linux core file documentation, it is best to read
sources in its producers (Linux kernel, gdb gcore) + consumers (many)


Jan

Re: Custom core file

[Nikolay Martyanov]

On 09/28/2016 06:32 PM, Jan Kratochvil wrote:
> On Wed, 28 Sep 2016 17:18:35 +0200, Nikolay Martyanov wrote:
>> Idea is very similar to `kdump`, you are right. But instead of Linux kernel
>> I have a hypervisor running on a bare-metal. So, in some point of view, I
>> try to implement my own kdump. And it will dump not Linux kernel mem, but
>> mem of hypervisor.
>
> OK, so core files are not standardized the way executables+shlibs are by the
> standards of ELF-generic and ELF-x86_64 or ELF-x86 addons. You can either
> invent your own format (based on ELF or not) or - what I suggest - is to just
> mimic the Linux kernel process format.
>
>
Not sure I can mimic Linux format exactly. At least I have another set 
of registers in use (VMX of Intel x86 arch are involved). May be it's 
critical, may be not...
Any way, Linux code was the first thing I looked at. So my way is "mimic 
as close as possible, adapt as few as possible".
Thanks for confirmation that it is one of possible ways to solve my 
problem =)

And in case I can't mimic it for 100%, I have to make several 
adaptations in GDB. The details were listed in my first message.
Still hoping someone can comment on them

Thanks,
Nikolay

Re: Custom core file

[Nikolay Martyanov]

<http://www.securitylab.ru/>Thanks for response!

 > Then in the same python script execute dump/restore commands to load 
these into memory

I have tried to use "dump/restore" approach, but there is an issue.
When you use a restore command, you should already be in a context of 
debugging process.

I tried it like this:
1. Run GDB
2. Load symbols with  `file core.so`
3. Try to shove memory with `restore raw_mem.dump`
After that I have message: `You can't do that without a process to debug.`

So, I guess, dump/restore approach works only in two cases:
a) debugging live process you have attached to (it's not my case, as I 
perform postmortem debug)
b) already has loaded core file, which provides a context via saved CPU 
state - and it is the way I chose to follow.

Am I wrong in my guesses?

Thanks,
Nikolay

RE: Custom core file

[duane]

>> So, I guess, dump/restore approach works only in two cases:
>> a) debugging live process you have attached to (it's not my case, as I 
>> perform postmortem debug)
>> b) already has loaded core file, which provides a context via saved CPU 
>> state - and it is the way I chose to follow.

For step (B) - first load a dummy (tiny) standard core file - that gives
you a debug context, think of it as a shim.

Then proceed with adding your symbols and using dump/restore to load
your image.

Re: Custom core file

[Nikolay Martyanov]

On 09/28/2016 06:31 PM, duane@duaneellis.com wrote:
>>> So, I guess, dump/restore approach works only in two cases:
>>> a) debugging live process you have attached to (it's not my case, as I
>>> perform postmortem debug)
>>> b) already has loaded core file, which provides a context via saved CPU
>>> state - and it is the way I chose to follow.
> For step (B) - first load a dummy (tiny) standard core file - that gives
> you a debug context, think of it as a shim.
Is it a common case for GDB? Can this "fake" context bring me some 
erroneous debug info?
I had such an idea, but due to doubt I mentioned above, decided not even 
to try. Now I will =)

Thanks,
Nikolay
<http://www.securitylab.ru/>

Re: Custom core file

[Nikolay Martyanov]

On 09/28/2016 06:31 PM, duane@duaneellis.com wrote:
>>> So, I guess, dump/restore approach works only in two cases:
>>> a) debugging live process you have attached to (it's not my case, as I
>>> perform postmortem debug)
>>> b) already has loaded core file, which provides a context via saved CPU
>>> state - and it is the way I chose to follow.
> For step (B) - first load a dummy (tiny) standard core file - that gives
> you a debug context, think of it as a shim.
>
> Then proceed with adding your symbols and using dump/restore to load
> your image.
>
>
At first I tried to restore memory in a case of fake core file 
(according to your advice). Looks like it's impossible, cause it still 
asks for a process. `Restore` doesn't work with core file debugging, I 
guess.

But now I have one other idea =)