Re: ReBranch - a record-replay debugging tool

Re: ReBranch - a record-replay debugging tool

[Nan Wang]

What I mean "control-flow only debugging" is:

Sometimes user only use GDB's control-flow functions, such as 'c', 'b',
'n', 's' ... to watch how the program get to the bug. He or she doesn't
care the variable name, the memory and some data-flow information.

ReBranch demands "control-flow only debugging" because it only records
every branch instruction. In current implementation (the modified
version of gdbserver), the replayer still need to create a process and
use ptrace to control it. When data-flow have error (caused by data-race
in multi threading situation), the ptraced process will generate
segfault for every instructions, which slows down the performance.

ReBranch have a GUI replayer -- ReBranchK -- which is a simple
control-flow-only debugging tool. ReBranchK doesn't really create the
process and debug it. It 'executes' the program virtually by reads the
log and shows corresponding source code. It implements 's', 'b' and 'c'
command. However, when writing ReBranchK, I found that, without stack
information, many useful control-flow command such as 'n' and 'bt' are
hard to be implemented. Therefore, I hope someone help me to put this
"control-flow only debugging" function into gdbserver.
> Can you clarify what do you mean by "control-flow only debugging"?
>
> (Note: I haven't had the time yet to read your document on ReBranch,
> so I don't really know how it works or why would you need gdbserver
> for replay)
>

Re: ReBranch - a record-replay debugging tool

[paawan oza]

Hi,

Is it something like you do an instrumentation in object code....mostly at all 
control flows and system calls.
and record some things.
so indirectly you do not record every instruction, but you need to modify object 
code by binary instrumentation.

but what I fail to understand is; what all do you record ?


Regards,
Oza.



----- Original Message ----
From: Nan Wang <pi3orama@gmail.com>
To: Pedro Alves <pedro@codesourcery.com>; gdb@sourceware.org
Sent: Thu, June 9, 2011 10:09:09 PM
Subject: Re: ReBranch - a record-replay debugging tool

What I mean "control-flow only debugging" is:

Sometimes user only use GDB's control-flow functions, such as 'c', 'b',
'n', 's' ... to watch how the program get to the bug. He or she doesn't
care the variable name, the memory and some data-flow information.

ReBranch demands "control-flow only debugging" because it only records
every branch instruction. In current implementation (the modified
version of gdbserver), the replayer still need to create a process and
use ptrace to control it. When data-flow have error (caused by data-race
in multi threading situation), the ptraced process will generate
segfault for every instructions, which slows down the performance.

ReBranch have a GUI replayer -- ReBranchK -- which is a simple
control-flow-only debugging tool. ReBranchK doesn't really create the
process and debug it. It 'executes' the program virtually by reads the
log and shows corresponding source code. It implements 's', 'b' and 'c'
command. However, when writing ReBranchK, I found that, without stack
information, many useful control-flow command such as 'n' and 'bt' are
hard to be implemented. Therefore, I hope someone help me to put this
"control-flow only debugging" function into gdbserver.
> Can you clarify what do you mean by "control-flow only debugging"?
>
> (Note: I haven't had the time yet to read your document on ReBranch,
> so I don't really know how it works or why would you need gdbserver
> for replay)
>

Re: Re: ReBranch - a record-replay debugging tool

[pi3orama]

Hi,
ReBranch instruments all indirect branch instruction (such as jnz, jmp
*0x12345, call *%eax..., and syscall), records their branch targets.
For system calls, ReBranch also record their results (for write(),
ReBranch records its return value; for read(), ReBranch records its
return value as well as the data it retrieves). It also records the
result of 'rdtsc'.

All instrumentation is done dynamically at runtime -- no recompilation
or relinking is required.

 On , paawan oza <paawan1982@yahoo.com> wrote:
> Hi,
>
>
>
> Is it something like you do an instrumentation in object code....mostly at
> all
>
> control flows and system calls.
>
> and record some things.
>
> so indirectly you do not record every instruction, but you need to modify
> object
>
> code by binary instrumentation.
>
>
>
> but what I fail to understand is; what all do you record ?
>
>
>
>
>
> Regards,
>
> Oza.
>
>
>
>
>
>
>
> ----- Original Message ----
>
> From: Nan Wang pi3orama@gmail.com>
>
> To: Pedro Alves pedro@codesourcery.com>; gdb@sourceware.org
>
> Sent: Thu, June 9, 2011 10:09:09 PM
>
> Subject: Re: ReBranch - a record-replay debugging tool
>
>
>
> What I mean "control-flow only debugging" is:
>
>
>
> Sometimes user only use GDB's control-flow functions, such as 'c', 'b',
>
> 'n', 's' ... to watch how the program get to the bug. He or she doesn't
>
> care the variable name, the memory and some data-flow information.
>
>
>
> ReBranch demands "control-flow only debugging" because it only records
>
> every branch instruction. In current implementation (the modified
>
> version of gdbserver), the replayer still need to create a process and
>
> use ptrace to control it. When data-flow have error (caused by data-race
>
> in multi threading situation), the ptraced process will generate
>
> segfault for every instructions, which slows down the performance.
>
>
>
> ReBranch have a GUI replayer -- ReBranchK -- which is a simple
>
> control-flow-only debugging tool. ReBranchK doesn't really create the
>
> process and debug it. It 'executes' the program virtually by reads the
>
> log and shows corresponding source code. It implements 's', 'b' and 'c'
>
> command. However, when writing ReBranchK, I found that, without stack
>
> information, many useful control-flow command such as 'n' and 'bt' are
>
> hard to be implemented. Therefore, I hope someone help me to put this
>
> "control-flow only debugging" function into gdbserver.
>
> > Can you clarify what do you mean by "control-flow only debugging"?
>
> >
>
> > (Note: I haven't had the time yet to read your document on ReBranch,
>
> > so I don't really know how it works or why would you need gdbserver
>
> > for replay)
>
> >
>
>

Re: Re: ReBranch - a record-replay debugging tool

[paawan oza]

data-flow info is not recorded because assumption is: many of the 
non-deterministic bugs are control-flow level.

so, if you do not have/or minial data-flow info, and if you only set link/branch 
registers and instruction pointers, 

then while replaying, do you think that program may not behave as in previous 
run because of loss of data flow info. 

especially it may not take some of the branches which it previously took.

because in you pdf example

switch (X)   /* here branch is recorded but not X where X might have been 
changed before it, and when you reply it may take different branch */

let me know if my thinking is ok in this sense.

Regards,
Oza.



----- Original Message ----
From: pi3orama <pi3orama@gmail.com>
To: paawan oza <paawan1982@yahoo.com>; Nan Wang <pi3orama@gmail.com>
Cc: gdb@sourceware.org
Sent: Fri, June 10, 2011 7:25:09 AM
Subject: Re: Re: ReBranch - a record-replay debugging tool

Hi,
ReBranch instruments all indirect branch instruction (such as jnz, jmp
*0x12345, call *%eax..., and syscall), records their branch targets.
For system calls, ReBranch also record their results (for write(),
ReBranch records its return value; for read(), ReBranch records its
return value as well as the data it retrieves). It also records the
result of 'rdtsc'.

All instrumentation is done dynamically at runtime -- no recompilation
or relinking is required.

On , paawan oza <paawan1982@yahoo.com> wrote:
> Hi,
>
>
>
> Is it something like you do an instrumentation in object code....mostly at
> all
>
> control flows and system calls.
>
> and record some things.
>
> so indirectly you do not record every instruction, but you need to modify
> object
>
> code by binary instrumentation.
>
>
>
> but what I fail to understand is; what all do you record ?
>
>
>
>
>
> Regards,
>
> Oza.
>
>
>
>
>
>
>
> ----- Original Message ----
>
> From: Nan Wang pi3orama@gmail.com>
>
> To: Pedro Alves pedro@codesourcery.com>; gdb@sourceware.org
>
> Sent: Thu, June 9, 2011 10:09:09 PM
>
> Subject: Re: ReBranch - a record-replay debugging tool
>
>
>
> What I mean "control-flow only debugging" is:
>
>
>
> Sometimes user only use GDB's control-flow functions, such as 'c', 'b',
>
> 'n', 's' ... to watch how the program get to the bug. He or she doesn't
>
> care the variable name, the memory and some data-flow information.
>
>
>
> ReBranch demands "control-flow only debugging" because it only records
>
> every branch instruction. In current implementation (the modified
>
> version of gdbserver), the replayer still need to create a process and
>
> use ptrace to control it. When data-flow have error (caused by data-race
>
> in multi threading situation), the ptraced process will generate
>
> segfault for every instructions, which slows down the performance.
>
>
>
> ReBranch have a GUI replayer -- ReBranchK -- which is a simple
>
> control-flow-only debugging tool. ReBranchK doesn't really create the
>
> process and debug it. It 'executes' the program virtually by reads the
>
> log and shows corresponding source code. It implements 's', 'b' and 'c'
>
> command. However, when writing ReBranchK, I found that, without stack
>
> information, many useful control-flow command such as 'n' and 'bt' are
>
> hard to be implemented. Therefore, I hope someone help me to put this
>
> "control-flow only debugging" function into gdbserver.
>
> > Can you clarify what do you mean by "control-flow only debugging"?
>
> >
>
> > (Note: I haven't had the time yet to read your document on ReBranch,
>
> > so I don't really know how it works or why would you need gdbserver
>
> > for replay)
>
> >
>
>

Re: ReBranch - a record-replay debugging tool

[pi3orama]

> data-flow info is not recorded because assumption is: many of the 
> non-deterministic bugs are control-flow level.
>
> so, if you do not have/or minial data-flow info, and if you only set link/branch 
> registers and instruction pointers, 
>
> then while replaying, do you think that program may not behave as in previous 
> run because of loss of data flow info. 
>
> especially it may not take some of the branches which it previously took.
>
> because in you pdf example
>
> switch (X)   /* here branch is recorded but not X where X might have been 
> changed before it, and when you reply it may take different branch */
>
> let me know if my thinking is ok in this sense.
>
> Regards,
> Oza.

ReBranch twists the branch if it goes to different target during replay.
in my example, if switch statement goes to another case during replay,
ReBranch will force the control flow back to the one in log. This is
done by gdbserver patch.

>
>
> ----- Original Message ----
> From: pi3orama <pi3orama@gmail.com>
> To: paawan oza <paawan1982@yahoo.com>; Nan Wang <pi3orama@gmail.com>
> Cc: gdb@sourceware.org
> Sent: Fri, June 10, 2011 7:25:09 AM
> Subject: Re: Re: ReBranch - a record-replay debugging tool
>
> Hi,
> ReBranch instruments all indirect branch instruction (such as jnz, jmp
> *0x12345, call *%eax..., and syscall), records their branch targets.
> For system calls, ReBranch also record their results (for write(),
> ReBranch records its return value; for read(), ReBranch records its
> return value as well as the data it retrieves). It also records the
> result of 'rdtsc'.
>
> All instrumentation is done dynamically at runtime -- no recompilation
> or relinking is required.
>
> On , paawan oza <paawan1982@yahoo.com> wrote:
>> Hi,
>>
>>
>>
>> Is it something like you do an instrumentation in object code....mostly at
>> all
>>
>> control flows and system calls.
>>
>> and record some things.
>>
>> so indirectly you do not record every instruction, but you need to modify
>> object
>>
>> code by binary instrumentation.
>>
>>
>>
>> but what I fail to understand is; what all do you record ?
>>
>>
>>
>>
>>
>> Regards,
>>
>> Oza.
>>
>>
>>
>>
>>
>>
>>
>> ----- Original Message ----
>>
>> From: Nan Wang pi3orama@gmail.com>
>>
>> To: Pedro Alves pedro@codesourcery.com>; gdb@sourceware.org
>>
>> Sent: Thu, June 9, 2011 10:09:09 PM
>>
>> Subject: Re: ReBranch - a record-replay debugging tool
>>
>>
>>
>> What I mean "control-flow only debugging" is:
>>
>>
>>
>> Sometimes user only use GDB's control-flow functions, such as 'c', 'b',
>>
>> 'n', 's' ... to watch how the program get to the bug. He or she doesn't
>>
>> care the variable name, the memory and some data-flow information.
>>
>>
>>
>> ReBranch demands "control-flow only debugging" because it only records
>>
>> every branch instruction. In current implementation (the modified
>>
>> version of gdbserver), the replayer still need to create a process and
>>
>> use ptrace to control it. When data-flow have error (caused by data-race
>>
>> in multi threading situation), the ptraced process will generate
>>
>> segfault for every instructions, which slows down the performance.
>>
>>
>>
>> ReBranch have a GUI replayer -- ReBranchK -- which is a simple
>>
>> control-flow-only debugging tool. ReBranchK doesn't really create the
>>
>> process and debug it. It 'executes' the program virtually by reads the
>>
>> log and shows corresponding source code. It implements 's', 'b' and 'c'
>>
>> command. However, when writing ReBranchK, I found that, without stack
>>
>> information, many useful control-flow command such as 'n' and 'bt' are
>>
>> hard to be implemented. Therefore, I hope someone help me to put this
>>
>> "control-flow only debugging" function into gdbserver.
>>
>>> Can you clarify what do you mean by "control-flow only debugging"?
>>> (Note: I haven't had the time yet to read your document on ReBranch,
>>> so I don't really know how it works or why would you need gdbserver
>>> for replay)
>>

Re: ReBranch - a record-replay debugging tool

[paawan oza]

ok got that point.
so in conclusion, any data-flow info which is reponsible for crash may not be 
caught by rebranch, or there is a way ?

Regards,
Oza.




----- Original Message ----
From: pi3orama <pi3orama@gmail.com>
To: paawan oza <paawan1982@yahoo.com>
Cc: gdb@sourceware.org
Sent: Fri, June 10, 2011 1:43:11 PM
Subject: Re: ReBranch - a record-replay debugging tool


> data-flow info is not recorded because assumption is: many of the 
> non-deterministic bugs are control-flow level.
>
> so, if you do not have/or minial data-flow info, and if you only set 
>link/branch 
>
> registers and instruction pointers, 
>
> then while replaying, do you think that program may not behave as in previous 
> run because of loss of data flow info. 
>
> especially it may not take some of the branches which it previously took.
>
> because in you pdf example
>
> switch (X)   /* here branch is recorded but not X where X might have been 
> changed before it, and when you reply it may take different branch */
>
> let me know if my thinking is ok in this sense.
>
> Regards,
> Oza.

ReBranch twists the branch if it goes to different target during replay.
in my example, if switch statement goes to another case during replay,
ReBranch will force the control flow back to the one in log. This is
done by gdbserver patch.

>
>
> ----- Original Message ----
> From: pi3orama <pi3orama@gmail.com>
> To: paawan oza <paawan1982@yahoo.com>; Nan Wang <pi3orama@gmail.com>
> Cc: gdb@sourceware.org
> Sent: Fri, June 10, 2011 7:25:09 AM
> Subject: Re: Re: ReBranch - a record-replay debugging tool
>
> Hi,
> ReBranch instruments all indirect branch instruction (such as jnz, jmp
> *0x12345, call *%eax..., and syscall), records their branch targets.
> For system calls, ReBranch also record their results (for write(),
> ReBranch records its return value; for read(), ReBranch records its
> return value as well as the data it retrieves). It also records the
> result of 'rdtsc'.
>
> All instrumentation is done dynamically at runtime -- no recompilation
> or relinking is required.
>
> On , paawan oza <paawan1982@yahoo.com> wrote:
>> Hi,
>>
>>
>>
>> Is it something like you do an instrumentation in object code....mostly at
>> all
>>
>> control flows and system calls.
>>
>> and record some things.
>>
>> so indirectly you do not record every instruction, but you need to modify
>> object
>>
>> code by binary instrumentation.
>>
>>
>>
>> but what I fail to understand is; what all do you record ?
>>
>>
>>
>>
>>
>> Regards,
>>
>> Oza.
>>
>>
>>
>>
>>
>>
>>
>> ----- Original Message ----
>>
>> From: Nan Wang pi3orama@gmail.com>
>>
>> To: Pedro Alves pedro@codesourcery.com>; gdb@sourceware.org
>>
>> Sent: Thu, June 9, 2011 10:09:09 PM
>>
>> Subject: Re: ReBranch - a record-replay debugging tool
>>
>>
>>
>> What I mean "control-flow only debugging" is:
>>
>>
>>
>> Sometimes user only use GDB's control-flow functions, such as 'c', 'b',
>>
>> 'n', 's' ... to watch how the program get to the bug. He or she doesn't
>>
>> care the variable name, the memory and some data-flow information.
>>
>>
>>
>> ReBranch demands "control-flow only debugging" because it only records
>>
>> every branch instruction. In current implementation (the modified
>>
>> version of gdbserver), the replayer still need to create a process and
>>
>> use ptrace to control it. When data-flow have error (caused by data-race
>>
>> in multi threading situation), the ptraced process will generate
>>
>> segfault for every instructions, which slows down the performance.
>>
>>
>>
>> ReBranch have a GUI replayer -- ReBranchK -- which is a simple
>>
>> control-flow-only debugging tool. ReBranchK doesn't really create the
>>
>> process and debug it. It 'executes' the program virtually by reads the
>>
>> log and shows corresponding source code. It implements 's', 'b' and 'c'
>>
>> command. However, when writing ReBranchK, I found that, without stack
>>
>> information, many useful control-flow command such as 'n' and 'bt' are
>>
>> hard to be implemented. Therefore, I hope someone help me to put this
>>
>> "control-flow only debugging" function into gdbserver.
>>
>>> Can you clarify what do you mean by "control-flow only debugging"?
>>> (Note: I haven't had the time yet to read your document on ReBranch,
>>> so I don't really know how it works or why would you need gdbserver
>>> for replay)
>>

Re: ReBranch - a record-replay debugging tool

[pi3orama]

In multi-thread situation, nearly all data-flow information is lost
because of propagation. However, in single thread situation, signal
processing and system call results are all recorded and replayed, all
data-flow info can still be restored.

> ok got that point.
> so in conclusion, any data-flow info which is reponsible for crash may not be 
> caught by rebranch, or there is a way ?
>
> Regards,
> Oza.
>
>
>
>
> ----- Original Message ----
> From: pi3orama <pi3orama@gmail.com>
> To: paawan oza <paawan1982@yahoo.com>
> Cc: gdb@sourceware.org
> Sent: Fri, June 10, 2011 1:43:11 PM
> Subject: Re: ReBranch - a record-replay debugging tool
>
>
>> data-flow info is not recorded because assumption is: many of the 
>> non-deterministic bugs are control-flow level.
>>
>> so, if you do not have/or minial data-flow info, and if you only set 
>> link/branch 
>>
>> registers and instruction pointers, 
>>
>> then while replaying, do you think that program may not behave as in previous 
>> run because of loss of data flow info. 
>>
>> especially it may not take some of the branches which it previously took.
>>
>> because in you pdf example
>>
>> switch (X)   /* here branch is recorded but not X where X might have been 
>> changed before it, and when you reply it may take different branch */
>>
>> let me know if my thinking is ok in this sense.
>>
>> Regards,
>> Oza.
> ReBranch twists the branch if it goes to different target during replay.
> in my example, if switch statement goes to another case during replay,
> ReBranch will force the control flow back to the one in log. This is
> done by gdbserver patch.
>
>>
>> ----- Original Message ----
>> From: pi3orama <pi3orama@gmail.com>
>> To: paawan oza <paawan1982@yahoo.com>; Nan Wang <pi3orama@gmail.com>
>> Cc: gdb@sourceware.org
>> Sent: Fri, June 10, 2011 7:25:09 AM
>> Subject: Re: Re: ReBranch - a record-replay debugging tool
>>
>> Hi,
>> ReBranch instruments all indirect branch instruction (such as jnz, jmp
>> *0x12345, call *%eax..., and syscall), records their branch targets.
>> For system calls, ReBranch also record their results (for write(),
>> ReBranch records its return value; for read(), ReBranch records its
>> return value as well as the data it retrieves). It also records the
>> result of 'rdtsc'.
>>
>> All instrumentation is done dynamically at runtime -- no recompilation
>> or relinking is required.
>>
>> On , paawan oza <paawan1982@yahoo.com> wrote:
>>> Hi,
>>>
>>>
>>>
>>> Is it something like you do an instrumentation in object code....mostly at
>>> all
>>>
>>> control flows and system calls.
>>>
>>> and record some things.
>>>
>>> so indirectly you do not record every instruction, but you need to modify
>>> object
>>>
>>> code by binary instrumentation.
>>>
>>>
>>>
>>> but what I fail to understand is; what all do you record ?
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Oza.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ----- Original Message ----
>>>
>>> From: Nan Wang pi3orama@gmail.com>
>>>
>>> To: Pedro Alves pedro@codesourcery.com>; gdb@sourceware.org
>>>
>>> Sent: Thu, June 9, 2011 10:09:09 PM
>>>
>>> Subject: Re: ReBranch - a record-replay debugging tool
>>>
>>>
>>>
>>> What I mean "control-flow only debugging" is:
>>>
>>>
>>>
>>> Sometimes user only use GDB's control-flow functions, such as 'c', 'b',
>>>
>>> 'n', 's' ... to watch how the program get to the bug. He or she doesn't
>>>
>>> care the variable name, the memory and some data-flow information.
>>>
>>>
>>>
>>> ReBranch demands "control-flow only debugging" because it only records
>>>
>>> every branch instruction. In current implementation (the modified
>>>
>>> version of gdbserver), the replayer still need to create a process and
>>>
>>> use ptrace to control it. When data-flow have error (caused by data-race
>>>
>>> in multi threading situation), the ptraced process will generate
>>>
>>> segfault for every instructions, which slows down the performance.
>>>
>>>
>>>
>>> ReBranch have a GUI replayer -- ReBranchK -- which is a simple
>>>
>>> control-flow-only debugging tool. ReBranchK doesn't really create the
>>>
>>> process and debug it. It 'executes' the program virtually by reads the
>>>
>>> log and shows corresponding source code. It implements 's', 'b' and 'c'
>>>
>>> command. However, when writing ReBranchK, I found that, without stack
>>>
>>> information, many useful control-flow command such as 'n' and 'bt' are
>>>
>>> hard to be implemented. Therefore, I hope someone help me to put this
>>>
>>> "control-flow only debugging" function into gdbserver.
>>>
>>>> Can you clarify what do you mean by "control-flow only debugging"?
>>>> (Note: I haven't had the time yet to read your document on ReBranch,
>>>> so I don't really know how it works or why would you need gdbserver
>>>> for replay)

Re: ReBranch - a record-replay debugging tool

[Yao Qi]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=GB2312, Size: 2353 bytes --]

On 06/10/2011 12:39 AM, Nan Wang wrote:
> What I mean "control-flow only debugging" is:
> 
> Sometimes user only use GDB's control-flow functions, such as 'c', 'b',
> 'n', 's' ... to watch how the program get to the bug. He or she doesn't
> care the variable name, the memory and some data-flow information.
> 

"control-flow only" is a good idea, and I think that is the reason why
your overhead, both on time and space, is so small.   Good work!

> ReBranch demands "control-flow only debugging" because it only records
> every branch instruction. In current implementation (the modified
> version of gdbserver), the replayer still need to create a process and
> use ptrace to control it. When data-flow have error (caused by data-race
> in multi threading situation), the ptraced process will generate

IIUC, you have an assumption there that "during record, program runs
correctly, while during replace, program may run incorrectly",
otherwise, you can't find the problem by comparing control-flow.

The first half of assumption is OK, but the second half, which is more
important, might not be.  Some research works are focusing on how to
expose/trigger non-deterministic program's faults.  Your current
approach in replaying looks fine, but if you want to find more bugs in
replay, that might be the way to go.

> segfault for every instructions, which slows down the performance.
> 

I don't understand why "process will generate segfault for every
instruction"?

> ReBranch have a GUI replayer -- ReBranchK -- which is a simple
> control-flow-only debugging tool. ReBranchK doesn't really create the
> process and debug it. It 'executes' the program virtually by reads the
> log and shows corresponding source code. It implements 's', 'b' and 'c'
> command. However, when writing ReBranchK, I found that, without stack
> information, many useful control-flow command such as 'n' and 'bt' are
> hard to be implemented. Therefore, I hope someone help me to put this
> "control-flow only debugging" function into gdbserver.

IIUC, you want to put your replayer function into gdbserver, in which,
record log file, and checkpoints are input.  User can connect to this
special gdbsever to debug.  Is that correct?

Generally speaking, gdbserver supports multiple archs, so is your record
log file format and checkpoints portable?

-- 
Yao (ÆëÒ¢)

Re: ReBranch - a record-replay debugging tool

[pi3orama]

>> ReBranch demands "control-flow only debugging" because it only records
>> every branch instruction. In current implementation (the modified
>> version of gdbserver), the replayer still need to create a process and
>> use ptrace to control it. When data-flow have error (caused by data-race
>> in multi threading situation), the ptraced process will generate
> IIUC, you have an assumption there that "during record, program runs
> correctly, while during replace, program may run incorrectly",
> otherwise, you can't find the problem by comparing control-flow.
>
> The first half of assumption is OK, but the second half, which is more
> important, might not be.  Some research works are focusing on how to
> expose/trigger non-deterministic program's faults.  Your current
> approach in replaying looks fine, but if you want to find more bugs in
> replay, that might be the way to go.
>

I don't think I have those assumption...

Yes, ReBranch doesn't trigger non-deterministic bugs. In fact it doesn't
know whether the program runs correctly or fault. ReBranch's job is to
faithfully record and replay the execution. If problems raise during
execution, ReBranch allows it to raise again in replay, and allows
developers to look into its control flow. Which converts
non-deterministic bugs into deterministic bugs. Developers can then
replay the program many time to find the bug. Without such a
record/replay tool, developers are hard to do this: when they notice the
bug and apply GDB, the bug will disappear.
>> segfault for every instructions, which slows down the performance.
>>
> I don't understand why "process will generate segfault for every
> instruction"?
>

Nearly every memory accessing instructions will segfault a while after a
new thread creation.

Currently, although all threads can be recorded, ReBranch is unable to
replay them simultaneously. Users can only replay and watch one thread
at a time. Therefore, because of the lost of IPC from shared memory,
sometimes ReBranch have to twist branch because the replayed thread
doesn't have the same data-flow as the original one. The effect can
propagate. For example, during original execution, a thread get a
pointer from another thread, then retrieve some data structure by it,
then do some computation. During replay, it won't get correct pointer
because there's no the other thread, but corresponding branch
instructions are twisted, so it will retrieve data from a invalid
address. A segfault is generated, then ReBranch intercepts it, let the
replay continue. So the following computation will be invalid, force
ReBranch twist more branches. In our experience, the propagate is very
fast. After the return from pthread_create function, the segfault
problem have already heavily slows down the replay speed.

>> ReBranch have a GUI replayer -- ReBranchK -- which is a simple
>> control-flow-only debugging tool. ReBranchK doesn't really create the
>> process and debug it. It 'executes' the program virtually by reads the
>> log and shows corresponding source code. It implements 's', 'b' and 'c'
>> command. However, when writing ReBranchK, I found that, without stack
>> information, many useful control-flow command such as 'n' and 'bt' are
>> hard to be implemented. Therefore, I hope someone help me to put this
>> "control-flow only debugging" function into gdbserver.
> IIUC, you want to put your replayer function into gdbserver, in which,
> record log file, and checkpoints are input.  User can connect to this
> special gdbsever to debug.  Is that correct?

Yes. Especially, I want gdbserver can handle stack frame information to
implement 'n', 'bt' and 'finish' commands.

> Generally speaking, gdbserver supports multiple archs, so is your record
> log file format and checkpoints portable?

I hope they are portable. However, currently checkpoint files and log
files are all platform specific. Checkpoint files contain some CPU
specific information; log files contain some platform specific system
call information.