System call support in reversible debugging

System call support in reversible debugging

[Sean Chen]

Hi,

Can gdb record system calls without checkpoints or snapshots
currently? If not, is it possible to record the instructions and
reverse the inferior?

When I recorded fclose(), I got the following message.
The next instruction is syscall munmap.  It will free the memory addr
= 0xb7fe0000 len = 4096.  It will make record target get error.  Do
you want to stop the program?([y] or n)

Thanks in advance.

-- 
Best Regards,
Sean Chen

Re: System call support in reversible debugging

[Hui Zhu]

This is a old query.  For now, this munmap just can let prec ignore
some memory change record entry.

You can answer 'no' then inferior will keep exec.  Most of thing will
be find.  :)

Maybe we need update this query looks not so scary and  do some other update.

Thanks,
Hui

On Fri, Nov 27, 2009 at 14:31, Sean Chen <sean.chen1234@gmail.com> wrote:
> Hi,
>
> Can gdb record system calls without checkpoints or snapshots
> currently? If not, is it possible to record the instructions and
> reverse the inferior?
>
> When I recorded fclose(), I got the following message.
> The next instruction is syscall munmap.  It will free the memory addr
> = 0xb7fe0000 len = 4096.  It will make record target get error.  Do
> you want to stop the program?([y] or n)
>
> Thanks in advance.
>
> --
> Best Regards,
> Sean Chen
>

Re: System call support in reversible debugging

[Sean Chen]

On Fri, Nov 27, 2009 at 4:04 PM, Hui Zhu <teawater@gmail.com> wrote:
> This is a old query.  For now, this munmap just can let prec ignore
> some memory change record entry.
>
> You can answer 'no' then inferior will keep exec.  Most of thing will
> be find.  :)
>
> Maybe we need update this query looks not so scary and  do some other update.
>
> Thanks,
> Hui
>
> On Fri, Nov 27, 2009 at 14:31, Sean Chen <sean.chen1234@gmail.com> wrote:
>> Hi,
>>
>> Can gdb record system calls without checkpoints or snapshots
>> currently? If not, is it possible to record the instructions and
>> reverse the inferior?
>>
>> When I recorded fclose(), I got the following message.
>> The next instruction is syscall munmap.  It will free the memory addr
>> = 0xb7fe0000 len = 4096.  It will make record target get error.  Do
>> you want to stop the program?([y] or n)
>>
>> Thanks in advance.
>>
>> --
>> Best Regards,
>> Sean Chen
>>
>

Thanks for your reply.

Does it mean that we have to ignore all memory changes in all system
calls or kernel space code?

-- 
Best Regards,
Sean Chen

Re: System call support in reversible debugging

[Hui Zhu]

No.  munmap is not memory change.  It release memory.

Hui

On Fri, Nov 27, 2009 at 23:11, Sean Chen <sean.chen1234@gmail.com> wrote:
> On Fri, Nov 27, 2009 at 4:04 PM, Hui Zhu <teawater@gmail.com> wrote:
>> This is a old query.  For now, this munmap just can let prec ignore
>> some memory change record entry.
>>
>> You can answer 'no' then inferior will keep exec.  Most of thing will
>> be find.  :)
>>
>> Maybe we need update this query looks not so scary and  do some other update.
>>
>> Thanks,
>> Hui
>>
>> On Fri, Nov 27, 2009 at 14:31, Sean Chen <sean.chen1234@gmail.com> wrote:
>>> Hi,
>>>
>>> Can gdb record system calls without checkpoints or snapshots
>>> currently? If not, is it possible to record the instructions and
>>> reverse the inferior?
>>>
>>> When I recorded fclose(), I got the following message.
>>> The next instruction is syscall munmap.  It will free the memory addr
>>> = 0xb7fe0000 len = 4096.  It will make record target get error.  Do
>>> you want to stop the program?([y] or n)
>>>
>>> Thanks in advance.
>>>
>>> --
>>> Best Regards,
>>> Sean Chen
>>>
>>
>
> Thanks for your reply.
>
> Does it mean that we have to ignore all memory changes in all system
> calls or kernel space code?
>
> --
> Best Regards,
> Sean Chen
>

Re: System call support in reversible debugging

[Sean Chen]

On Fri, Nov 27, 2009 at 11:16 PM, Hui Zhu <teawater@gmail.com> wrote:
> No.  munmap is not memory change.  It release memory.
>
> Hui
>
> On Fri, Nov 27, 2009 at 23:11, Sean Chen <sean.chen1234@gmail.com> wrote:
>> On Fri, Nov 27, 2009 at 4:04 PM, Hui Zhu <teawater@gmail.com> wrote:
>>> This is a old query.  For now, this munmap just can let prec ignore
>>> some memory change record entry.
>>>
>>> You can answer 'no' then inferior will keep exec.  Most of thing will
>>> be find.  :)
>>>
>>> Maybe we need update this query looks not so scary and  do some other update.
>>>
>>> Thanks,
>>> Hui
>>>
>>> On Fri, Nov 27, 2009 at 14:31, Sean Chen <sean.chen1234@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> Can gdb record system calls without checkpoints or snapshots
>>>> currently? If not, is it possible to record the instructions and
>>>> reverse the inferior?
>>>>
>>>> When I recorded fclose(), I got the following message.
>>>> The next instruction is syscall munmap.  It will free the memory addr
>>>> = 0xb7fe0000 len = 4096.  It will make record target get error.  Do
>>>> you want to stop the program?([y] or n)
>>>>
>>>> Thanks in advance.
>>>>
>>>> --
>>>> Best Regards,
>>>> Sean Chen
>>>>
>>>
>>
>> Thanks for your reply.
>>
>> Does it mean that we have to ignore all memory changes in all system
>> calls or kernel space code?
>>
>> --
>> Best Regards,
>> Sean Chen
>>
>

Thanks.

Well. You mentioned that this munmap just can let prec ignore some
memory change record entry. Could you please show me what kind of
memory change is ignored?

Can gdb record the system call?

-- 
Best Regards,
Sean Chen

Re: System call support in reversible debugging

[Michael Snyder]

Sean Chen wrote:

> Well. You mentioned that this munmap just can let prec ignore some
> memory change record entry. Could you please show me what kind of
> memory change is ignored?
> 
> Can gdb record the system call?

These are two separate questions.  I think the one you started with
is can gdb record a system call, and the answer is "yes".


The issue with mmap has a lot of history, and rather than try to
explain it, I urge you to look up the threads which have "mmap"
or "sbrk" in the title) and read them.

Re: System call support in reversible debugging

[Sean Chen]

On Sat, Nov 28, 2009 at 2:07 AM, Michael Snyder <msnyder@vmware.com> wrote:
> These are two separate questions.  I think the one you started with
> is can gdb record a system call, and the answer is "yes".
>
>
> The issue with mmap has a lot of history, and rather than try to
> explain it, I urge you to look up the threads which have "mmap"
> or "sbrk" in the title) and read them.
>
>

Thanks for the explanation. That is very kind of you.

I am confused about the first question. How does gdb record the system
call instructions? You know, they are in the kernel space? It seems
that I must have made a mistake somewhere. Please help to clarify.

Thanks in advance.

-- 
Best Regards,
Sean Chen

Re: System call support in reversible debugging

[Michael Snyder]

Sean Chen wrote:
> On Sat, Nov 28, 2009 at 2:07 AM, Michael Snyder <msnyder@vmware.com> wrote:
>> These are two separate questions.  I think the one you started with
>> is can gdb record a system call, and the answer is "yes".
>>
>>
>> The issue with mmap has a lot of history, and rather than try to
>> explain it, I urge you to look up the threads which have "mmap"
>> or "sbrk" in the title) and read them.
>>
>>
> 
> Thanks for the explanation. That is very kind of you.
> 
> I am confused about the first question. How does gdb record the system
> call instructions? You know, they are in the kernel space? It seems
> that I must have made a mistake somewhere. Please help to clarify.
> 
> Thanks in advance.

Sean,

I wish I understood this better -- maybe Hui will explain it more.

As I understand it, each system call is recorded as if it were a
single instruction.  Instead of 'tracing' into the system code,
we know the specific side effects for each system call, and for
instance if the syscall will write to a buffer we take a snapshot
of that buffer first.

Michael

Re: System call support in reversible debugging

[Hui Zhu]

Cool! I think your explain is very clear. Thanks.  :)

Hui

On Sun, Nov 29, 2009 at 01:40, Michael Snyder <msnyder@vmware.com> wrote:
> Sean Chen wrote:
>>
>> On Sat, Nov 28, 2009 at 2:07 AM, Michael Snyder <msnyder@vmware.com>
>> wrote:
>>>
>>> These are two separate questions.  I think the one you started with
>>> is can gdb record a system call, and the answer is "yes".
>>>
>>>
>>> The issue with mmap has a lot of history, and rather than try to
>>> explain it, I urge you to look up the threads which have "mmap"
>>> or "sbrk" in the title) and read them.
>>>
>>>
>>
>> Thanks for the explanation. That is very kind of you.
>>
>> I am confused about the first question. How does gdb record the system
>> call instructions? You know, they are in the kernel space? It seems
>> that I must have made a mistake somewhere. Please help to clarify.
>>
>> Thanks in advance.
>
> Sean,
>
> I wish I understood this better -- maybe Hui will explain it more.
>
> As I understand it, each system call is recorded as if it were a
> single instruction.  Instead of 'tracing' into the system code,
> we know the specific side effects for each system call, and for
> instance if the syscall will write to a buffer we take a snapshot
> of that buffer first.
>
> Michael
>
>

Re: System call support in reversible debugging

[Sean Chen]

On Mon, Nov 30, 2009 at 8:27 PM, Hui Zhu <teawater@gmail.com> wrote:
> Cool! I think your explain is very clear. Thanks.  :)
>
> Hui
>
> On Sun, Nov 29, 2009 at 01:40, Michael Snyder <msnyder@vmware.com> wrote:
>> Sean Chen wrote:
>>>
>>> On Sat, Nov 28, 2009 at 2:07 AM, Michael Snyder <msnyder@vmware.com>
>>> wrote:
>>>>
>>>> These are two separate questions.  I think the one you started with
>>>> is can gdb record a system call, and the answer is "yes".
>>>>
>>>>
>>>> The issue with mmap has a lot of history, and rather than try to
>>>> explain it, I urge you to look up the threads which have "mmap"
>>>> or "sbrk" in the title) and read them.
>>>>
>>>>
>>>
>>> Thanks for the explanation. That is very kind of you.
>>>
>>> I am confused about the first question. How does gdb record the system
>>> call instructions? You know, they are in the kernel space? It seems
>>> that I must have made a mistake somewhere. Please help to clarify.
>>>
>>> Thanks in advance.
>>
>> Sean,
>>
>> I wish I understood this better -- maybe Hui will explain it more.
>>
>> As I understand it, each system call is recorded as if it were a
>> single instruction.  Instead of 'tracing' into the system code,
>> we know the specific side effects for each system call, and for
>> instance if the syscall will write to a buffer we take a snapshot
>> of that buffer first.
>>
>> Michael
>>
>>
>

Hi Michael and Hui,

I am sorry for my late response.

Thanks for your explanation. So we can’t treat the system calls as a
black box and have to understand the detailed implementation of each
system call. I think we need to understand every lines of the code in
the system calls carefully enough, and care about the difference of
the Linux kernel since the code of system calls might change
frequently. Do we have any good ways to do it?

Thanks.

-- 
Best Regards,
Sean Chen

RE: System call support in reversible debugging

[Jakob Engblom]

 
> Hi Michael and Hui,
> 
> I am sorry for my late response.
> 
> Thanks for your explanation. So we can’t treat the system calls as a
> black box and have to understand the detailed implementation of each
> system call. I think we need to understand every lines of the code in
> the system calls carefully enough, and care about the difference of
> the Linux kernel since the code of system calls might change
> frequently. Do we have any good ways to do it?

To really do this right, you should use a full-system simulator that lets you
debug OS and user code at the same time, as it is attacking the system at the
hardware/software interface level. 

As long as you attack a program as a user-level problem only, you cannot debug
into the kernel, as the process record facilities are built on kernel features
that only apply to user-level programs. As I understand it. 

Both VmWare and Virtutech offer solutions for this that you can drive over
gdb-serial, in reverse when needed. 

Best regards,

/jakob

_______________________________________________________

Jakob Engblom, PhD, Technical Marketing Manager

Virtutech                   Direct: +46 8 690 07 47   
Drottningholmsvägen 22      Mobile: +46 709 242 646  
11243 Stockholm             Web:    www.virtutech.com 
Sweden
________________________________________________________
  

Re: System call support in reversible debugging

[Greg Law]

Ok, I can't resist jumping in :)

Jakob Engblom wrote:
>  
>> Hi Michael and Hui,
>>
>> I am sorry for my late response.
>>
>> Thanks for your explanation. So we canÂ’t treat the system calls as a
>> black box and have to understand the detailed implementation of each
>> system call. I think we need to understand every lines of the code in
>> the system calls carefully enough, and care about the difference of
>> the Linux kernel since the code of system calls might change
>> frequently. Do we have any good ways to do it?
> 
> To really do this right, you should use a full-system simulator that lets you
> debug OS and user code at the same time, as it is attacking the system at the
> hardware/software interface level. 

It all depends what you want to do.

If you want to debug kernel code, then absolutely you need a full system 
approach, such as Simics or VMware offers.  Similarly if you want to 
debug the whole host.  But if you're debugging just a process (i.e. the 
classic use-case of gdb), you may not want to wind back the state of the 
entire (virtual) machine.  In which case, something like UndoDB or prec 
is more appropriate.

I don't claim either approach is superior.  It's a bit like native 
debugging versus remote debugging.  Which one makes most sense all 
depends on what it is you're trying to debug.

Greg
-- 
Greg Law, Undo Software                       http://undo-software.com/

RE: System call support in reversible debugging

[Jakob Engblom]

> >> Hi Michael and Hui,
> >>
> >> I am sorry for my late response.
> >>
> >> Thanks for your explanation. So we can't treat the system calls as a
> >> black box and have to understand the detailed implementation of each
> >> system call. I think we need to understand every lines of the code in
> >> the system calls carefully enough, and care about the difference of
> >> the Linux kernel since the code of system calls might change
> >> frequently. Do we have any good ways to do it?
> >
> > To really do this right, you should use a full-system simulator that lets
you
> > debug OS and user code at the same time, as it is attacking the system at
the
> > hardware/software interface level.
> 
> It all depends what you want to do.
> 
> If you want to debug kernel code, then absolutely you need a full system
> approach, such as Simics or VMware offers.  Similarly if you want to
> debug the whole host.  But if you're debugging just a process (i.e. the
> classic use-case of gdb), you may not want to wind back the state of the
> entire (virtual) machine.  In which case, something like UndoDB or prec
> is more appropriate.
> 
> I don't claim either approach is superior.  It's a bit like native
> debugging versus remote debugging.  Which one makes most sense all
> depends on what it is you're trying to debug.

Couldn't agree more.  The full-system approach is a bit more gnarly, but it
gives you more insight.  It depends on the problem.

/jakob

Re: System call support in reversible debugging

[Sean Chen]

On Thu, Dec 3, 2009 at 1:16 AM, Jakob Engblom <jakob@virtutech.com> wrote:
>> >> Hi Michael and Hui,
>> >>
>> >> I am sorry for my late response.
>> >>
>> >> Thanks for your explanation. So we can't treat the system calls as a
>> >> black box and have to understand the detailed implementation of each
>> >> system call. I think we need to understand every lines of the code in
>> >> the system calls carefully enough, and care about the difference of
>> >> the Linux kernel since the code of system calls might change
>> >> frequently. Do we have any good ways to do it?
>> >
>> > To really do this right, you should use a full-system simulator that lets
> you
>> > debug OS and user code at the same time, as it is attacking the system at
> the
>> > hardware/software interface level.
>>
>> It all depends what you want to do.
>>
>> If you want to debug kernel code, then absolutely you need a full system
>> approach, such as Simics or VMware offers.  Similarly if you want to
>> debug the whole host.  But if you're debugging just a process (i.e. the
>> classic use-case of gdb), you may not want to wind back the state of the
>> entire (virtual) machine.  In which case, something like UndoDB or prec
>> is more appropriate.
>>
>> I don't claim either approach is superior.  It's a bit like native
>> debugging versus remote debugging.  Which one makes most sense all
>> depends on what it is you're trying to debug.
>
> Couldn't agree more.  The full-system approach is a bit more gnarly, but it
> gives you more insight.  It depends on the problem.
>
> /jakob
>
>

Hi Jakob and Greg,

You are both right. These two solutions focus on different problems.
If you want to debug a user space process and don’t care what’s going
on in the system call at all, Michael’s solution is OK. If you want to
look more closely into the kernel, a full-system simulator is
absolutely the best one.

-- 
Best Regards,
Sean Chen

Re: System call support in reversible debugging

[Sean Chen]

On Tue, Dec 1, 2009 at 4:34 AM, Michael Snyder <msnyder@vmware.com> wrote:
> Not a black box -- but not a detailed implementation either.
> Each system call has a specification -- that's all we need to
> understand.  The specification tells what user-visible external
> side effects can be expected (eg. this buffer will be filled).
> Changes that are internal to the kernel do not concern us.

Thanks. I understand your concern. Does Linux kernel provide such an
official specification? If so, could you throw me a hint on where to
get it? Since we have all of the side effects, recording the system
calls are feasible and trustable.

> Sean, this stuff is already implemented and working.
> Why don't you have a look at the existing code in
> linux-record.c?  It sounds like you're more than qualified
> to understand it.  If you find something that's not working
> correctly, please let us know!  ;-)

I am reading the source and the archive of the mailing list. Do you
have other systematic documents on this technique? Life becomes better
with your help. Thanks. :)

-- 
Best Regards,
Sean Chen

RE: System call support in reversible debugging

[Jakob Engblom]

> > Not a black box -- but not a detailed implementation either.
> > Each system call has a specification -- that's all we need to
> > understand.  The specification tells what user-visible external
> > side effects can be expected (eg. this buffer will be filled).
> > Changes that are internal to the kernel do not concern us.
> 
> Thanks. I understand your concern. Does Linux kernel provide such an
> official specification? If so, could you throw me a hint on where to
> get it? Since we have all of the side effects, recording the system
> calls are feasible and trustable.

I guess these have to be the header files specifying the function call
prototypes? 

Or is something else also done for precord?

/jakob

Re: System call support in reversible debugging

[Sean Chen]

On Thu, Dec 3, 2009 at 5:00 PM, Jakob Engblom <jakob@virtutech.com> wrote:
>> > Not a black box -- but not a detailed implementation either.
>> > Each system call has a specification -- that's all we need to
>> > understand.  The specification tells what user-visible external
>> > side effects can be expected (eg. this buffer will be filled).
>> > Changes that are internal to the kernel do not concern us.
>>
>> Thanks. I understand your concern. Does Linux kernel provide such an
>> official specification? If so, could you throw me a hint on where to
>> get it? Since we have all of the side effects, recording the system
>> calls are feasible and trustable.
>
> I guess these have to be the header files specifying the function call
> prototypes?
>
> Or is something else also done for precord?
>
> /jakob
>
>

I searched sys_open in the whole Linux kernel, and I didn’t find such
information. Thanks all the same.

-- 
Best Regards,
Sean Chen

Re: System call support in reversible debugging

[Michael Snyder]

Sean Chen wrote:
> On Tue, Dec 1, 2009 at 4:34 AM, Michael Snyder <msnyder@vmware.com> wrote:
>> Not a black box -- but not a detailed implementation either.
>> Each system call has a specification -- that's all we need to
>> understand.  The specification tells what user-visible external
>> side effects can be expected (eg. this buffer will be filled).
>> Changes that are internal to the kernel do not concern us.
> 
> Thanks. I understand your concern. Does Linux kernel provide such an
> official specification? If so, could you throw me a hint on where to
> get it? Since we have all of the side effects, recording the system
> calls are feasible and trustable.
> 
>> Sean, this stuff is already implemented and working.
>> Why don't you have a look at the existing code in
>> linux-record.c?  It sounds like you're more than qualified
>> to understand it.  If you find something that's not working
>> correctly, please let us know!  ;-)
> 
> I am reading the source and the archive of the mailing list. Do you
> have other systematic documents on this technique? Life becomes better
> with your help. Thanks. :)

Teawater implemented them, I am only reverse-engineering
them, like you.  Man pages are helpful.   ;-)

Re: System call support in reversible debugging

[Sean Chen]

On Fri, Dec 4, 2009 at 12:57 AM, Michael Snyder <msnyder@vmware.com> wrote:
> Sean Chen wrote:
>>
>> On Tue, Dec 1, 2009 at 4:34 AM, Michael Snyder <msnyder@vmware.com> wrote:
>>>
>>> Not a black box -- but not a detailed implementation either.
>>> Each system call has a specification -- that's all we need to
>>> understand.  The specification tells what user-visible external
>>> side effects can be expected (eg. this buffer will be filled).
>>> Changes that are internal to the kernel do not concern us.
>>
>> Thanks. I understand your concern. Does Linux kernel provide such an
>> official specification? If so, could you throw me a hint on where to
>> get it? Since we have all of the side effects, recording the system
>> calls are feasible and trustable.
>>
>>> Sean, this stuff is already implemented and working.
>>> Why don't you have a look at the existing code in
>>> linux-record.c?  It sounds like you're more than qualified
>>> to understand it.  If you find something that's not working
>>> correctly, please let us know!  ;-)
>>
>> I am reading the source and the archive of the mailing list. Do you
>> have other systematic documents on this technique? Life becomes better
>> with your help. Thanks. :)
>
> Teawater implemented them, I am only reverse-engineering
> them, like you.  Man pages are helpful.   ;-)
>
>

Man pages just tell how to use it. It won’t tell why use it. 

Hui,

Could you help with this? Thanks in advance.

-- 
Best Regards,
Sean Chen