* [RFA] Reverse Debugging, 5/5
@ 2008-10-01 19:21 Michael Snyder
2008-10-02 19:49 ` Eli Zaretskii
2008-10-07 3:30 ` Joel Brobecker
0 siblings, 2 replies; 15+ messages in thread
From: Michael Snyder @ 2008-10-01 19:21 UTC (permalink / raw)
To: gdb-patches; +Cc: Daniel Jacobowitz, Pedro Alves, teawater
[-- Attachment #1: Type: text/plain, Size: 0 bytes --]
[-- Attachment #2: reverse.txt --]
[-- Type: text/plain, Size: 14352 bytes --]
This patch (#5) enables all the others by adding a user interface.
The patch also includes the gdb manual documentation for reverse.
I've run all the previously existing gdb testsuite before and
after adding the five patches (including this one), with no
change in results.
I've also run the seven tests for reverse debugging
(break-reverse, consecutive-reverse, finish-reverse,
machinestate, step-reverse, until-reverse, and watch-
reverse) against the VMware reverse execution target.
I can't yet run them against teawater's implemenation
since it isn't in yet on this branch, but it passes the
testsuite fine with these basic patches on its own branch
(which any of you can verify).
2008-09-30 Michael Snyder <msnyder@vmware.com>
User interface for reverse execution.
* Makefile.in (reverse.c): New file.
* reverse.c: New file. User interface for reverse execution.
2008-09-30 Michael Snyder <msnyder@vmware.com>
* gdb.texinfo: Add documentation for reverse execution.
Index: Makefile.in
===================================================================
RCS file: /cvs/src/src/gdb/Makefile.in,v
retrieving revision 1.1051
diff -u -p -r1.1051 Makefile.in
--- Makefile.in 22 Sep 2008 15:12:19 -0000 1.1051
+++ Makefile.in 30 Sep 2008 23:57:53 -0000
@@ -639,7 +639,7 @@ SFILES = ada-exp.y ada-lang.c ada-typepr
objfiles.c osabi.c observer.c \
p-exp.y p-lang.c p-typeprint.c p-valprint.c parse.c printcmd.c \
prologue-value.c \
- regcache.c reggroups.c remote.c remote-fileio.c \
+ regcache.c reggroups.c remote.c remote-fileio.c reverse.c \
scm-exp.c scm-lang.c scm-valprint.c \
sentinel-frame.c \
serial.c ser-base.c ser-unix.c \
@@ -778,7 +778,8 @@ COMMON_OBS = $(DEPFILES) $(CONFIG_OBS) $
findcmd.o \
std-regs.o \
signals.o \
- exec.o bcache.o objfiles.o observer.o minsyms.o maint.o demangle.o \
+ exec.o reverse.o \
+ bcache.o objfiles.o observer.o minsyms.o maint.o demangle.o \
dbxread.o coffread.o coff-pe-read.o \
dwarf2read.o mipsread.o stabsread.o corefile.o \
dwarf2expr.o dwarf2loc.o dwarf2-frame.o \
Index: reverse.c
===================================================================
RCS file: reverse.c
diff -N reverse.c
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ reverse.c 30 Sep 2008 23:57:53 -0000
@@ -0,0 +1,196 @@
+/* Reverse execution and reverse debugging.
+
+ Copyright (C) 2006 Free Software Foundation, Inc.
+
+ This file is part of GDB.
+
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation; either version 2 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, write to the Free Software
+ Foundation, Inc., 51 Franklin Street, Fifth Floor,
+ Boston, MA 02110-1301, USA. */
+
+#include "defs.h"
+#include "gdb_string.h"
+#include "target.h"
+#include "top.h"
+#include "cli/cli-cmds.h"
+#include "cli/cli-decode.h"
+
+/* User interface for reverse debugging:
+ Set exec-direction / show exec-direction commands
+ (returns error unles target implements to_set_execdir method). */
+
+static const char exec_forward[] = "forward";
+static const char exec_reverse[] = "reverse";
+static const char *exec_direction = exec_forward;
+static const char *exec_direction_names[] = {
+ exec_forward,
+ exec_reverse,
+ NULL
+};
+
+static void
+set_exec_direction_func (char *args, int from_tty,
+ struct cmd_list_element *cmd)
+{
+ if (target_get_execution_direction () != EXEC_ERROR)
+ {
+ enum exec_direction_kind dir = EXEC_ERROR;
+
+ if (!strcmp (exec_direction, exec_forward))
+ dir = EXEC_FORWARD;
+ else if (!strcmp (exec_direction, exec_reverse))
+ dir = EXEC_REVERSE;
+
+ if (target_set_execution_direction (dir) != EXEC_ERROR)
+ return;
+ }
+}
+
+static void
+show_exec_direction_func (struct ui_file *out, int from_tty,
+ struct cmd_list_element *cmd, const char *value)
+{
+ enum exec_direction_kind dir = target_get_execution_direction ();
+
+ switch (dir) {
+ case EXEC_FORWARD:
+ fprintf_filtered (out, "Forward.\n");
+ break;
+ case EXEC_REVERSE:
+ fprintf_filtered (out, "Reverse.\n");
+ break;
+ case EXEC_ERROR:
+ default:
+ fprintf_filtered, (out,
+ _("Target `%s' does not support execution-direction."),
+ target_shortname);
+ break;
+ }
+}
+
+/* User interface:
+ reverse-step, reverse-next etc.
+ (returns error unles target implements to_set_execdir method). */
+
+static void execdir_default (void *notused)
+{
+ /* Return execution direction to default state. */
+ target_set_execution_direction (EXEC_FORWARD);
+}
+
+static void
+exec_reverse_once (char *cmd, char *args, int from_tty)
+{
+ /* String buffer for command consing. */
+ char reverse_command[512];
+ enum exec_direction_kind dir = target_get_execution_direction ();
+
+ if (dir == EXEC_ERROR)
+ error (_("Target %s does not support this command."), target_shortname);
+
+ if (dir == EXEC_REVERSE)
+ error (_("Already in reverse mode. Use '%s' or 'set exec-dir forward'."),
+ cmd);
+
+ if (target_set_execution_direction (EXEC_REVERSE) == EXEC_ERROR)
+ error (_("Target %s does not support this command."), target_shortname);
+
+ make_cleanup (execdir_default, NULL);
+ sprintf (reverse_command, "%s %s", cmd, args ? args : "");
+ execute_command (reverse_command, from_tty);
+}
+
+static void
+reverse_step (char *args, int from_tty)
+{
+ exec_reverse_once ("step", args, from_tty);
+}
+
+static void
+reverse_stepi (char *args, int from_tty)
+{
+ exec_reverse_once ("stepi", args, from_tty);
+}
+
+static void
+reverse_next (char *args, int from_tty)
+{
+ exec_reverse_once ("next", args, from_tty);
+}
+
+static void
+reverse_nexti (char *args, int from_tty)
+{
+ exec_reverse_once ("nexti", args, from_tty);
+}
+
+static void
+reverse_continue (char *args, int from_tty)
+{
+ exec_reverse_once ("continue", args, from_tty);
+}
+
+static void
+reverse_finish (char *args, int from_tty)
+{
+ exec_reverse_once ("finish", args, from_tty);
+}
+
+void
+_initialize_reverse (void)
+{
+ add_setshow_enum_cmd ("exec-direction", class_run, exec_direction_names,
+ &exec_direction, "Set direction of execution.\n\
+Options are 'forward' or 'reverse'.",
+ "Show direction of execution (forward/reverse).",
+ "Tells gdb whether to execute forward or backward.",
+ set_exec_direction_func, show_exec_direction_func,
+ &setlist, &showlist);
+
+ add_com ("reverse-step", class_run, reverse_step, _("\
+Step program backward until it reaches the beginning of another source line.\n\
+Argument N means do this N times (or till program stops for another reason).")
+ );
+ add_com_alias ("rs", "reverse-step", class_alias, 1);
+
+ add_com ("reverse-next", class_run, reverse_next, _("\
+Step program backward, proceeding through subroutine calls.\n\
+Like the \"reverse-step\" command as long as subroutine calls do not happen;\n\
+when they do, the call is treated as one instruction.\n\
+Argument N means do this N times (or till program stops for another reason).")
+ );
+ add_com_alias ("rn", "reverse-next", class_alias, 1);
+
+ add_com ("reverse-stepi", class_run, reverse_stepi, _("\
+Step backward exactly one instruction.\n\
+Argument N means do this N times (or till program stops for another reason).")
+ );
+ add_com_alias ("rsi", "reverse-stepi", class_alias, 0);
+
+ add_com ("reverse-nexti", class_run, reverse_nexti, _("\
+Step backward one instruction, but proceed through called subroutines.\n\
+Argument N means do this N times (or till program stops for another reason).")
+ );
+ add_com_alias ("rni", "reverse-nexti", class_alias, 0);
+
+ add_com ("reverse-continue", class_run, reverse_continue, _("\
+Continue program being debugged, running in reverse.\n\
+If proceeding from breakpoint, a number N may be used as an argument,\n\
+which means to set the ignore count of that breakpoint to N - 1 (so that\n\
+the breakpoint won't break until the Nth time it is reached)."));
+ add_com_alias ("rc", "reverse-continue", class_alias, 0);
+
+ add_com ("reverse-finish", class_run, reverse_finish, _("\
+Execute backward until just before selected stack frame is called."));
+}
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.525
diff -u -p -r1.525 gdb.texinfo
--- doc/gdb.texinfo 27 Sep 2008 21:40:48 -0000 1.525
+++ doc/gdb.texinfo 30 Sep 2008 23:57:54 -0000
@@ -143,6 +143,7 @@ software in general. We will miss him.
* Commands:: @value{GDBN} commands
* Running:: Running programs under @value{GDBN}
* Stopping:: Stopping and continuing
+* Reverse Execution:: Running programs backward
* Stack:: Examining the stack
* Source:: Examining source files
* Data:: Examining data
@@ -4842,6 +4843,109 @@ prematurely, even though your program do
+@node Reverse Execution
+@chapter Running programs backward
+
+When you are debugging a program, it is not unusual to realize that
+you have gone too far, and some event of interest has already happened.
+If the target environment supports it, @value{GDBN} can allow you to
+``rewind'' the program by running it backward.
+
+A target environment that supports reverse execution should be able
+to ``undo'' the changes in machine state that have taken place as the
+program was executing normally. Variables, registers etc. should
+revert to their previous values. Obviously this requires a great
+deal of sophistication on the part of the target environment; not
+all target environments can support reverse execution.
+
+When a program is executed in reverse, the instructions that
+have most recently been executed are ``un-executed'', in reverse
+order. The program counter runs backward, following the previous
+thread of execution in reverse. As each instruction is ``un-executed'',
+the values of memory and/or registers that were changed by that
+instruction are reverted to their previous states. After executing
+a piece of source code in reverse, all side effects of that code
+should be ``undone'', and all variables should be returned to their
+prior values.
+
+Assuming you are debugging in a target environment that supports
+reverse execution, @value{GDBN} provides the following commands.
+
+@table @code
+@kindex reverse-continue
+@kindex rc @r{(@code{reverse-continue})}
+@item reverse-continue @r{[}@var{ignore-count}@r{]}
+@itemx rc @r{[}@var{ignore-count}@r{]}
+Beginning at the point where your program last stopped, start executing
+in reverse. Reverse execution will stop for breakpoints and synchronous
+exceptions (signals), just like normal execution. Behavior of
+asynchronous signals depends on the target environment.
+
+@kindex reverse-step
+@kindex rs @r{(@code{step})}
+@item reverse-step @r{[}@var{count}@r{]}
+Run the program backward until control reaches the start of a
+different source line; then stop it, and return control to @value{GDBN}.
+
+Like the @code{step} command, @code{reverse-step} will only stop
+at the beginning of a source line. It ``un-executes'' the previously
+executed source line. If the previous source line included calls to
+debuggable functions, @code{reverse-step} will step (backward) into
+the called function, stopping at the beginning of the @emph{last}
+statement in the called function (typically a return statement).
+
+Also, as with the @code{step} command, if non-debuggable functions are
+called, @code{reverse-step} will run thru them backward without stopping.
+
+@kindex reverse-stepi
+@kindex rsi @r{(@code{reverse-stepi})}
+@item reverse-stepi @r{[}@var{count}@r{]}
+Reverse-execute one machine instruction. Note that the instruction
+to be reverse-executed is @emph{not} the one pointed to by the program
+counter, but the instruction executed prior to that one. For instance,
+if the last instruction was a jump, @code{reverse-stepi} will take you
+back from the destination of the jump to the jump instruction itself.
+
+@kindex reverse-next
+@kindex rn @r{(@code{reverse-next})}
+@item reverse-next @r{[}@var{count}@r{]}
+Run backward to the beginning of the previous line executed in
+the current (innermost) stack frame. If the line contains function
+calls, they will be ``un-executed'' without stopping. Starting from
+the first line of a function, @code{reverse-next} will take you back
+to the caller of that function, @emph{before} the function was called.
+
+@kindex reverse-nexti
+@kindex rni @r{(@code{reverse-nexti})}
+@item reverse-nexti @r{[}@var{count}@r{]}
+Like @code{nexti}, @code{reverse-nexti} executes a single instruction
+in reverse, except that called functions are ``un-executed'' atomically.
+That is, if the previously executed instruction was a return from
+another instruction, @code{reverse-nexti} will continue to execute
+in reverse until the call to that function (from the current stack
+frame) is reached.
+
+@kindex reverse-finish
+@item reverse-finish
+Just as the @code{finish} command takes you to the point where the
+current function returns, @code{reverse-finish} takes you to the point
+where it was called. Instead of ending up at the end of the current
+function invocation, you end up at the beginning.
+
+@item set exec-direction
+Set the direction of target execution.
+@itemx set exec-direction reverse
+@cindex execute forward or backward in time
+@value{GDBN} will perform all execution commands in reverse, until the
+exec-direction mode is changed to ``forward''. Affected commands include
+@code{step, stepi, next, nexti, continue, and finish}. The @code{return}
+command cannot be used in reverse mode.
+@item set exec-direction forward
+@value{GDBN} will perform all execution commands in the normal fashion.
+This is the default.
+@end table
+
+
@node Stack
@chapter Examining the Stack
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-01 19:21 [RFA] Reverse Debugging, 5/5 Michael Snyder
@ 2008-10-02 19:49 ` Eli Zaretskii
2008-10-02 21:42 ` Michael Snyder
` (2 more replies)
2008-10-07 3:30 ` Joel Brobecker
1 sibling, 3 replies; 15+ messages in thread
From: Eli Zaretskii @ 2008-10-02 19:49 UTC (permalink / raw)
To: Michael Snyder; +Cc: gdb-patches, drow, pedro, teawater
> Date: Wed, 01 Oct 2008 12:20:06 -0700
> From: Michael Snyder <msnyder@vmware.com>
> CC: Daniel Jacobowitz <drow@false.org>, Pedro Alves <pedro@codesourcery.com>, teawater <teawater@gmail.com>
>
> +static void
> +set_exec_direction_func (char *args, int from_tty,
> + struct cmd_list_element *cmd)
> +{
> + if (target_get_execution_direction () != EXEC_ERROR)
> + {
> + enum exec_direction_kind dir = EXEC_ERROR;
> +
> + if (!strcmp (exec_direction, exec_forward))
> + dir = EXEC_FORWARD;
> + else if (!strcmp (exec_direction, exec_reverse))
> + dir = EXEC_REVERSE;
> +
> + if (target_set_execution_direction (dir) != EXEC_ERROR)
> + return;
> + }
> +}
I'm not sure I get your intent with the last two lines: if
target_set_execution_direction returns anything but EXEC_ERROR, you
return, but if not, you ... return? What am I missing?
> + case EXEC_ERROR:
> + default:
> + fprintf_filtered, (out,
> + _("Target `%s' does not support execution-direction."),
> + target_shortname);
> + break;
Why print an error message? isn't it better to say the direction is
"forward" (which is documented as the default in your patch for the
manual)?
> + if (dir == EXEC_REVERSE)
> + error (_("Already in reverse mode. Use '%s' or 'set exec-dir forward'."),
> + cmd);
Isn't it better to silently do the equivalent of "cmd"?
> + add_com ("reverse-step", class_run, reverse_step, _("\
> +Step program backward until it reaches the beginning of another source line.\n\
> +Argument N means do this N times (or till program stops for another reason).")
This sounds as if you are single-stepping the program until it reaches
the previous line (as opposed to running uninterrupted until you hit
previous line). Are you?
> + add_com ("reverse-next", class_run, reverse_next, _("\
> +Step program backward, proceeding through subroutine calls.\n\
^
Won't commands like "apropos" stop at the first comma when they
display the short descriptions of commands?
> +@node Reverse Execution
> +@chapter Running programs backward
Please add a @cindex entry here.
> +program was executing normally. Variables, registers etc. should
Please put a @: after "etc.", to prevent the period being interpreted
by TeX as an end of a sentence.
> After executing
> +a piece of source code in reverse, all side effects of that code
> +should be ``undone''
ALL side effects? I thought some of them cannot be undone,
un-outputting to the various I/O devices etc.
> +Assuming you are debugging in a target environment that supports
"IF you are debugging in a target ..." sounds better, I think.
> Starting from
> +the first line of a function, @code{reverse-next} will take you back
> +to the caller of that function, @emph{before} the function was called.
Shouldn't we have some kind of caveat here regarding function prologue
and epilogue?
> +@item set exec-direction
There should be a @kindex entry here for this command.
Thanks!
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-02 19:49 ` Eli Zaretskii
@ 2008-10-02 21:42 ` Michael Snyder
2008-10-03 6:30 ` Eli Zaretskii
2008-10-02 22:43 ` Michael Snyder
2008-10-02 22:54 ` Michael Snyder
2 siblings, 1 reply; 15+ messages in thread
From: Michael Snyder @ 2008-10-02 21:42 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: gdb-patches, drow, pedro, teawater
Eli Zaretskii wrote:
>> Date: Wed, 01 Oct 2008 12:20:06 -0700
>> From: Michael Snyder <msnyder@vmware.com>
>> CC: Daniel Jacobowitz <drow@false.org>, Pedro Alves <pedro@codesourcery.com>, teawater <teawater@gmail.com>
>>
>> +static void
>> +set_exec_direction_func (char *args, int from_tty,
>> + struct cmd_list_element *cmd)
>> +{
>> + if (target_get_execution_direction () != EXEC_ERROR)
>> + {
>> + enum exec_direction_kind dir = EXEC_ERROR;
>> +
>> + if (!strcmp (exec_direction, exec_forward))
>> + dir = EXEC_FORWARD;
>> + else if (!strcmp (exec_direction, exec_reverse))
>> + dir = EXEC_REVERSE;
>> +
>> + if (target_set_execution_direction (dir) != EXEC_ERROR)
>> + return;
>> + }
>> +}
>
> I'm not sure I get your intent with the last two lines: if
> target_set_execution_direction returns anything but EXEC_ERROR, you
> return, but if not, you ... return? What am I missing?
Well, this is the implementation of a "set" command.
If target_get_execution_direction returns EXEC_ERROR,
it means that the target doesn't support reverse, in
which case there's nothing to set, so I just return.
Otherwise, I pass the request to the target.
Possibly I should output something, like "target does
not support this comand", but since it's a "set" command,
it isn't expected to output anything and it doesn't
receive a struct ui_file among its arguments.
Originally I had it call error() in that case, but I
found that caused a problem in the testsuite, default.exp.
Apparently doing an "info set" causes every registered
"set" function to be called with no arguments?
>> + case EXEC_ERROR:
>> + default:
>> + fprintf_filtered, (out,
>> + _("Target `%s' does not support execution-direction."),
>> + target_shortname);
>> + break;
>
> Why print an error message? isn't it better to say the direction is
> "forward" (which is documented as the default in your patch for the
> manual)?
Maybe. I'm open to discussion.
The context is, the user says "show exec-direction"
with a target that doesn't support reverse.
Is it better to just say "Forward", with no comment,
or is it better to let the user know that the question
is not applicable? Or both?
>> + if (dir == EXEC_REVERSE)
>> + error (_("Already in reverse mode. Use '%s' or 'set exec-dir forward'."),
>> + cmd);
>
> Isn't it better to silently do the equivalent of "cmd"?
Possibly. Again, I'm open to discussion. But I think
we've discussed this one before, and I hope we don't
get bogged down again...
Here the context is that the user has set a sticky
mode bit ("reverse"), but then has used one of the
non-sticky commands such as "reverse-step".
What should we do? A reverse-reverse-step, ie. forward?
That's logical, but is it what the user wanted?
The problem is compounded by the fact that it won't
always be easy for the user to know whether we in fact
went forward or back. If s/he realizes that we have
gone the 'wrong' way, s/he can correct it; but if not...
>> + add_com ("reverse-step", class_run, reverse_step, _("\
>> +Step program backward until it reaches the beginning of another source line.\n\
>> +Argument N means do this N times (or till program stops for another reason).")
>
> This sounds as if you are single-stepping the program until it reaches
> the previous line (as opposed to running uninterrupted until you hit
> previous line). Are you?
Of course I am single-stepping until it reaches the
previous line. It's the same way we do forward stepping.
We don't and can't implement step by using a breakpoint,
because we don't know whether the source line contains
jumps or calls. We only use the stepping-breakpoint
to get back after we have single-stepped thru a call
(or something similar).
I'm using the same step-range-start / step-range-end
logic that we use to step forward (with minor but
essential modifications).
>> + add_com ("reverse-next", class_run, reverse_next, _("\
>> +Step program backward, proceeding through subroutine calls.\n\
> ^
> Won't commands like "apropos" stop at the first comma when they
> display the short descriptions of commands?
Yep, you're right!
I should omit the comma, huh?
>> +@node Reverse Execution
>> +@chapter Running programs backward
>
> Please add a @cindex entry here.
OK.
>> +program was executing normally. Variables, registers etc. should
>
> Please put a @: after "etc.", to prevent the period being interpreted
> by TeX as an end of a sentence.
OK.
>> After executing
>> +a piece of source code in reverse, all side effects of that code
>> +should be ``undone''
>
> ALL side effects? I thought some of them cannot be undone,
> un-outputting to the various I/O devices etc.
That depends on the implementation on the target side.
For instance, VMware's implementation can undo device I/O,
so long as the device is virtual, but gdb-freeplay cannot.
GDB has no way of knowing this, but perhaps I should say
something more explicit about it in the doc?
>> +Assuming you are debugging in a target environment that supports
>
> "IF you are debugging in a target ..." sounds better, I think.
Agreed.
>> Starting from
>> +the first line of a function, @code{reverse-next} will take you back
>> +to the caller of that function, @emph{before} the function was called.
>
> Shouldn't we have some kind of caveat here regarding function prologue
> and epilogue?
Like what?
If I've done my job right, prologues and epilogues
should be handled transparently, just like they are
when stepping forward.
And that seems to be reflected by the step-reverse.exp test.
>> +@item set exec-direction
>
> There should be a @kindex entry here for this command.
OK.
Thanks a lot for your comments, Eli.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-02 19:49 ` Eli Zaretskii
2008-10-02 21:42 ` Michael Snyder
@ 2008-10-02 22:43 ` Michael Snyder
2008-10-03 6:35 ` Eli Zaretskii
2008-10-02 22:54 ` Michael Snyder
2 siblings, 1 reply; 15+ messages in thread
From: Michael Snyder @ 2008-10-02 22:43 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: gdb-patches, drow, pedro, teawater
[-- Attachment #1: Type: text/plain, Size: 121 bytes --]
Here's a revised patch for the doc only.
I added a footnote about side effects, and
followed your other suggestions.
[-- Attachment #2: gdb.texinfo.txt --]
[-- Type: text/plain, Size: 6713 bytes --]
2008-09-30 Michael Snyder <msnyder@vmware.com>
* gdb.texinfo: Add documentation for reverse execution.
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.525
diff -u -p -r1.525 gdb.texinfo
--- doc/gdb.texinfo 27 Sep 2008 21:40:48 -0000 1.525
+++ doc/gdb.texinfo 2 Oct 2008 22:27:43 -0000
@@ -143,6 +143,7 @@ software in general. We will miss him.
* Commands:: @value{GDBN} commands
* Running:: Running programs under @value{GDBN}
* Stopping:: Stopping and continuing
+* Reverse Execution:: Running programs backward
* Stack:: Examining the stack
* Source:: Examining source files
* Data:: Examining data
@@ -4841,6 +4842,123 @@ When such an event happens, a system cal
prematurely, even though your program does not appear to stop.
+@node Reverse Execution
+@chapter Running programs backward
+@cindex reverse execution
+@cindex running programs backward
+
+When you are debugging a program, it is not unusual to realize that
+you have gone too far, and some event of interest has already happened.
+If the target environment supports it, @value{GDBN} can allow you to
+``rewind'' the program by running it backward.
+
+A target environment that supports reverse execution should be able
+to ``undo'' the changes in machine state that have taken place as the
+program was executing normally. Variables, registers etc.@: should
+revert to their previous values. Obviously this requires a great
+deal of sophistication on the part of the target environment; not
+all target environments can support reverse execution.
+
+When a program is executed in reverse, the instructions that
+have most recently been executed are ``un-executed'', in reverse
+order. The program counter runs backward, following the previous
+thread of execution in reverse. As each instruction is ``un-executed'',
+the values of memory and/or registers that were changed by that
+instruction are reverted to their previous states. After executing
+a piece of source code in reverse, all side effects of that code
+should be ``undone'', and all variables should be returned to their
+prior values@footnote{
+Note that some side effects are easier to undo than others. For instance,
+memory and registers are relatively easy, but device I/O is hard. Some
+targets may be able undo things like device I/O, and some may not.
+
+The contract between @value{GDBN} and the reverse executing target
+requires only that the target do something reasonable when
+@value{GDBN} tells it to execute backwards, and then report the
+results back to @value{GDBN}. Whatever the target reports back to
+@value{GDBN}, @value{GDBN} will report back to the user. We assume
+that the memory and registers that the target reports to us are in a
+consistant state, but we accept whatever we are given.
+}.
+
+If you are debugging in a target environment that supports
+reverse execution, @value{GDBN} provides the following commands.
+
+@table @code
+@kindex reverse-continue
+@kindex rc @r{(@code{reverse-continue})}
+@item reverse-continue @r{[}@var{ignore-count}@r{]}
+@itemx rc @r{[}@var{ignore-count}@r{]}
+Beginning at the point where your program last stopped, start executing
+in reverse. Reverse execution will stop for breakpoints and synchronous
+exceptions (signals), just like normal execution. Behavior of
+asynchronous signals depends on the target environment.
+
+@kindex reverse-step
+@kindex rs @r{(@code{step})}
+@item reverse-step @r{[}@var{count}@r{]}
+Run the program backward until control reaches the start of a
+different source line; then stop it, and return control to @value{GDBN}.
+
+Like the @code{step} command, @code{reverse-step} will only stop
+at the beginning of a source line. It ``un-executes'' the previously
+executed source line. If the previous source line included calls to
+debuggable functions, @code{reverse-step} will step (backward) into
+the called function, stopping at the beginning of the @emph{last}
+statement in the called function (typically a return statement).
+
+Also, as with the @code{step} command, if non-debuggable functions are
+called, @code{reverse-step} will run thru them backward without stopping.
+
+@kindex reverse-stepi
+@kindex rsi @r{(@code{reverse-stepi})}
+@item reverse-stepi @r{[}@var{count}@r{]}
+Reverse-execute one machine instruction. Note that the instruction
+to be reverse-executed is @emph{not} the one pointed to by the program
+counter, but the instruction executed prior to that one. For instance,
+if the last instruction was a jump, @code{reverse-stepi} will take you
+back from the destination of the jump to the jump instruction itself.
+
+@kindex reverse-next
+@kindex rn @r{(@code{reverse-next})}
+@item reverse-next @r{[}@var{count}@r{]}
+Run backward to the beginning of the previous line executed in
+the current (innermost) stack frame. If the line contains function
+calls, they will be ``un-executed'' without stopping. Starting from
+the first line of a function, @code{reverse-next} will take you back
+to the caller of that function, @emph{before} the function was called.
+
+@kindex reverse-nexti
+@kindex rni @r{(@code{reverse-nexti})}
+@item reverse-nexti @r{[}@var{count}@r{]}
+Like @code{nexti}, @code{reverse-nexti} executes a single instruction
+in reverse, except that called functions are ``un-executed'' atomically.
+That is, if the previously executed instruction was a return from
+another instruction, @code{reverse-nexti} will continue to execute
+in reverse until the call to that function (from the current stack
+frame) is reached.
+
+@kindex reverse-finish
+@item reverse-finish
+Just as the @code{finish} command takes you to the point where the
+current function returns, @code{reverse-finish} takes you to the point
+where it was called. Instead of ending up at the end of the current
+function invocation, you end up at the beginning.
+
+@kindex set exec-direction
+@item set exec-direction
+Set the direction of target execution.
+@itemx set exec-direction reverse
+@cindex execute forward or backward in time
+@value{GDBN} will perform all execution commands in reverse, until the
+exec-direction mode is changed to ``forward''. Affected commands include
+@code{step, stepi, next, nexti, continue, and finish}. The @code{return}
+command cannot be used in reverse mode.
+@item set exec-direction forward
+@value{GDBN} will perform all execution commands in the normal fashion.
+This is the default.
+@end table
+
@node Stack
@chapter Examining the Stack
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-02 19:49 ` Eli Zaretskii
2008-10-02 21:42 ` Michael Snyder
2008-10-02 22:43 ` Michael Snyder
@ 2008-10-02 22:54 ` Michael Snyder
2008-10-03 6:25 ` Eli Zaretskii
2 siblings, 1 reply; 15+ messages in thread
From: Michael Snyder @ 2008-10-02 22:54 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: gdb-patches, drow, pedro, teawater
Eli Zaretskii wrote:
>> + add_com ("reverse-next", class_run, reverse_next, _("\
>> +Step program backward, proceeding through subroutine calls.\n\
> ^
> Won't commands like "apropos" stop at the first comma when they
> display the short descriptions of commands?
Hmmm, you know, "next" has the same issue.
(gdb) apropos next
[...]
next -- Step program
nexti -- Step one instruction
... why does it stop after a comma, anyways?
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-02 22:54 ` Michael Snyder
@ 2008-10-03 6:25 ` Eli Zaretskii
2008-10-03 17:45 ` Daniel Jacobowitz
2008-10-03 17:58 ` Michael Snyder
0 siblings, 2 replies; 15+ messages in thread
From: Eli Zaretskii @ 2008-10-03 6:25 UTC (permalink / raw)
To: Michael Snyder; +Cc: gdb-patches, drow, pedro, teawater
> Date: Thu, 02 Oct 2008 15:52:32 -0700
> From: Michael Snyder <msnyder@vmware.com>
> CC: "gdb-patches@sourceware.org" <gdb-patches@sourceware.org>,
> "drow@false.org" <drow@false.org>,
> "pedro@codesourcery.com" <pedro@codesourcery.com>,
> "teawater@gmail.com" <teawater@gmail.com>
>
> Eli Zaretskii wrote:
>
> >> + add_com ("reverse-next", class_run, reverse_next, _("\
> >> +Step program backward, proceeding through subroutine calls.\n\
> > ^
> > Won't commands like "apropos" stop at the first comma when they
> > display the short descriptions of commands?
>
> Hmmm, you know, "next" has the same issue.
>
> (gdb) apropos next
> [...]
> next -- Step program
> nexti -- Step one instruction
>
>
> ... why does it stop after a comma, anyways?
I have no idea, it's something that came into existence long before I
started hacking GDB.
I'd say, if not too many doc strings use the comma at the end of the
first line, let's modify them to use a period, and remove this
annoying limitation, leaving only the period. WDYT?
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-02 21:42 ` Michael Snyder
@ 2008-10-03 6:30 ` Eli Zaretskii
2008-10-03 18:07 ` Michael Snyder
0 siblings, 1 reply; 15+ messages in thread
From: Eli Zaretskii @ 2008-10-03 6:30 UTC (permalink / raw)
To: Michael Snyder; +Cc: gdb-patches, drow, pedro, teawater
> Date: Thu, 02 Oct 2008 14:40:51 -0700
> From: Michael Snyder <msnyder@vmware.com>
> CC: "gdb-patches@sourceware.org" <gdb-patches@sourceware.org>,
> "drow@false.org" <drow@false.org>,
> "pedro@codesourcery.com" <pedro@codesourcery.com>,
> "teawater@gmail.com" <teawater@gmail.com>
>
> >> + case EXEC_ERROR:
> >> + default:
> >> + fprintf_filtered, (out,
> >> + _("Target `%s' does not support execution-direction."),
> >> + target_shortname);
> >> + break;
> >
> > Why print an error message? isn't it better to say the direction is
> > "forward" (which is documented as the default in your patch for the
> > manual)?
>
> Maybe. I'm open to discussion.
>
> The context is, the user says "show exec-direction"
> with a target that doesn't support reverse.
>
> Is it better to just say "Forward", with no comment,
> or is it better to let the user know that the question
> is not applicable? Or both?
Both, I'd say.
> >> + if (dir == EXEC_REVERSE)
> >> + error (_("Already in reverse mode. Use '%s' or 'set exec-dir forward'."),
> >> + cmd);
> >
> > Isn't it better to silently do the equivalent of "cmd"?
>
> Possibly. Again, I'm open to discussion. But I think
> we've discussed this one before
Well, at least I'm consistent... ;-)
> > ALL side effects? I thought some of them cannot be undone,
> > un-outputting to the various I/O devices etc.
>
> That depends on the implementation on the target side.
> For instance, VMware's implementation can undo device I/O,
> so long as the device is virtual, but gdb-freeplay cannot.
>
> GDB has no way of knowing this, but perhaps I should say
> something more explicit about it in the doc?
Yes.
> >> Starting from
> >> +the first line of a function, @code{reverse-next} will take you back
> >> +to the caller of that function, @emph{before} the function was called.
> >
> > Shouldn't we have some kind of caveat here regarding function prologue
> > and epilogue?
>
> Like what?
>
> If I've done my job right, prologues and epilogues
> should be handled transparently, just like they are
> when stepping forward.
Are they treated transparently when we step forward? I had an
impression that in optimized code, they aren't always transparent.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-02 22:43 ` Michael Snyder
@ 2008-10-03 6:35 ` Eli Zaretskii
0 siblings, 0 replies; 15+ messages in thread
From: Eli Zaretskii @ 2008-10-03 6:35 UTC (permalink / raw)
To: Michael Snyder; +Cc: gdb-patches, drow, pedro, teawater
> Date: Thu, 02 Oct 2008 15:41:57 -0700
> From: Michael Snyder <msnyder@vmware.com>
> CC: "gdb-patches@sourceware.org" <gdb-patches@sourceware.org>,
> "drow@false.org" <drow@false.org>,
> "pedro@codesourcery.com" <pedro@codesourcery.com>,
> "teawater@gmail.com" <teawater@gmail.com>
>
> Here's a revised patch for the doc only.
> I added a footnote about side effects, and
> followed your other suggestions.
Thanks. I have only one minor comment this time:
> +The contract between @value{GDBN} and the reverse executing target
> +requires only that the target do something reasonable when
> +@value{GDBN} tells it to execute backwards, and then report the
> +results back to @value{GDBN}. Whatever the target reports back to
> +@value{GDBN}, @value{GDBN} will report back to the user. We assume
> +that the memory and registers that the target reports to us are in a
> +consistant state, but we accept whatever we are given.
This text switches from "GDB" to "we" in mid-air. I'd prefer to stay
with "GDB", as it will not be very clear to the reader who are "we" ;-)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-03 6:25 ` Eli Zaretskii
@ 2008-10-03 17:45 ` Daniel Jacobowitz
2008-10-03 17:58 ` Michael Snyder
1 sibling, 0 replies; 15+ messages in thread
From: Daniel Jacobowitz @ 2008-10-03 17:45 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: Michael Snyder, gdb-patches, pedro, teawater
On Fri, Oct 03, 2008 at 09:23:42AM +0300, Eli Zaretskii wrote:
> I'd say, if not too many doc strings use the comma at the end of the
> first line, let's modify them to use a period, and remove this
> annoying limitation, leaving only the period. WDYT?
Amen. Some day we'll have to fix this for real, and removing the
comma check is a step in the right direction.
--
Daniel Jacobowitz
CodeSourcery
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-03 6:25 ` Eli Zaretskii
2008-10-03 17:45 ` Daniel Jacobowitz
@ 2008-10-03 17:58 ` Michael Snyder
1 sibling, 0 replies; 15+ messages in thread
From: Michael Snyder @ 2008-10-03 17:58 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: gdb-patches, drow, pedro, teawater
Eli Zaretskii wrote:
>> Date: Thu, 02 Oct 2008 15:52:32 -0700
>> From: Michael Snyder <msnyder@vmware.com>
>> CC: "gdb-patches@sourceware.org" <gdb-patches@sourceware.org>,
>> "drow@false.org" <drow@false.org>,
>> "pedro@codesourcery.com" <pedro@codesourcery.com>,
>> "teawater@gmail.com" <teawater@gmail.com>
>>
>> Eli Zaretskii wrote:
>>
>>>> + add_com ("reverse-next", class_run, reverse_next, _("\
>>>> +Step program backward, proceeding through subroutine calls.\n\
>>> ^
>>> Won't commands like "apropos" stop at the first comma when they
>>> display the short descriptions of commands?
>> Hmmm, you know, "next" has the same issue.
>>
>> (gdb) apropos next
>> [...]
>> next -- Step program
>> nexti -- Step one instruction
>>
>>
>> ... why does it stop after a comma, anyways?
>
> I have no idea, it's something that came into existence long before I
> started hacking GDB.
Daniel Berlin, May 2000
> I'd say, if not too many doc strings use the comma at the end of the
> first line, let's modify them to use a period, and remove this
> annoying limitation, leaving only the period. WDYT?
Sounds good to me...
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-03 6:30 ` Eli Zaretskii
@ 2008-10-03 18:07 ` Michael Snyder
2008-10-04 8:13 ` Eli Zaretskii
0 siblings, 1 reply; 15+ messages in thread
From: Michael Snyder @ 2008-10-03 18:07 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: gdb-patches, drow, pedro, teawater
Eli Zaretskii wrote:
>> Date: Thu, 02 Oct 2008 14:40:51 -0700
>> From: Michael Snyder <msnyder@vmware.com>
>> The context is, the user says "show exec-direction"
>> with a target that doesn't support reverse.
>>
>> Is it better to just say "Forward", with no comment,
>> or is it better to let the user know that the question
>> is not applicable? Or both?
>
> Both, I'd say.
OK, how about this?
(gdb) show exec-direction
Forward (target `None' does not support exec-direction).
>>> Shouldn't we have some kind of caveat here regarding function prologue
>>> and epilogue?
>> Like what?
>>
>> If I've done my job right, prologues and epilogues
>> should be handled transparently, just like they are
>> when stepping forward.
>
> Are they treated transparently when we step forward? I had an
> impression that in optimized code, they aren't always transparent.
OK, I should have said "we do our best to treat them
transparently". I suppose if the code is too optimized
for us to do a good job when we're going forward, we will
also have problems in reverse.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-03 18:07 ` Michael Snyder
@ 2008-10-04 8:13 ` Eli Zaretskii
2008-10-04 17:56 ` Michael Snyder
0 siblings, 1 reply; 15+ messages in thread
From: Eli Zaretskii @ 2008-10-04 8:13 UTC (permalink / raw)
To: Michael Snyder; +Cc: gdb-patches, drow, pedro, teawater
> Date: Fri, 03 Oct 2008 11:05:17 -0700
> From: Michael Snyder <msnyder@vmware.com>
> CC: "gdb-patches@sourceware.org" <gdb-patches@sourceware.org>, "drow@false.org" <drow@false.org>, "pedro@codesourcery.com" <pedro@codesourcery.com>, "teawater@gmail.com" <teawater@gmail.com>
>
> Eli Zaretskii wrote:
> >> Date: Thu, 02 Oct 2008 14:40:51 -0700
> >> From: Michael Snyder <msnyder@vmware.com>
>
> >> The context is, the user says "show exec-direction"
> >> with a target that doesn't support reverse.
> >>
> >> Is it better to just say "Forward", with no comment,
> >> or is it better to let the user know that the question
> >> is not applicable? Or both?
> >
> > Both, I'd say.
>
> OK, how about this?
>
> (gdb) show exec-direction
> Forward (target `None' does not support exec-direction).
Fine with me.
> >>> Shouldn't we have some kind of caveat here regarding function prologue
> >>> and epilogue?
> >> Like what?
> >>
> >> If I've done my job right, prologues and epilogues
> >> should be handled transparently, just like they are
> >> when stepping forward.
> >
> > Are they treated transparently when we step forward? I had an
> > impression that in optimized code, they aren't always transparent.
>
> OK, I should have said "we do our best to treat them
> transparently". I suppose if the code is too optimized
> for us to do a good job when we're going forward, we will
> also have problems in reverse.
Yes, that's what I had in mind. So perhaps the text should be a
little vaguer and maybe with some reasonable qualifier, like "unless
the code is too heavily optimized". Maybe have that in a @footnote.
^ permalink raw reply [flat|nested] 15+ messages in thread
* RE: [RFA] Reverse Debugging, 5/5
2008-10-04 8:13 ` Eli Zaretskii
@ 2008-10-04 17:56 ` Michael Snyder
0 siblings, 0 replies; 15+ messages in thread
From: Michael Snyder @ 2008-10-04 17:56 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: gdb-patches, drow, pedro, teawater
Done and done.
Excellent! We seem to be approaching convergence!
________________________________________
From: Eli Zaretskii [eliz@gnu.org]
Sent: Saturday, October 04, 2008 1:11 AM
To: Michael Snyder
Cc: gdb-patches@sourceware.org; drow@false.org; pedro@codesourcery.com; teawater@gmail.com
Subject: Re: [RFA] Reverse Debugging, 5/5
> Date: Fri, 03 Oct 2008 11:05:17 -0700
> From: Michael Snyder <msnyder@vmware.com>
> CC: "gdb-patches@sourceware.org" <gdb-patches@sourceware.org>, "drow@false.org" <drow@false.org>, "pedro@codesourcery.com" <pedro@codesourcery.com>, "teawater@gmail.com" <teawater@gmail.com>
>
> Eli Zaretskii wrote:
> >> Date: Thu, 02 Oct 2008 14:40:51 -0700
> >> From: Michael Snyder <msnyder@vmware.com>
>
> >> The context is, the user says "show exec-direction"
> >> with a target that doesn't support reverse.
> >>
> >> Is it better to just say "Forward", with no comment,
> >> or is it better to let the user know that the question
> >> is not applicable? Or both?
> >
> > Both, I'd say.
>
> OK, how about this?
>
> (gdb) show exec-direction
> Forward (target `None' does not support exec-direction).
Fine with me.
> >>> Shouldn't we have some kind of caveat here regarding function prologue
> >>> and epilogue?
> >> Like what?
> >>
> >> If I've done my job right, prologues and epilogues
> >> should be handled transparently, just like they are
> >> when stepping forward.
> >
> > Are they treated transparently when we step forward? I had an
> > impression that in optimized code, they aren't always transparent.
>
> OK, I should have said "we do our best to treat them
> transparently". I suppose if the code is too optimized
> for us to do a good job when we're going forward, we will
> also have problems in reverse.
Yes, that's what I had in mind. So perhaps the text should be a
little vaguer and maybe with some reasonable qualifier, like "unless
the code is too heavily optimized". Maybe have that in a @footnote.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-01 19:21 [RFA] Reverse Debugging, 5/5 Michael Snyder
2008-10-02 19:49 ` Eli Zaretskii
@ 2008-10-07 3:30 ` Joel Brobecker
2008-10-07 18:25 ` Michael Snyder
1 sibling, 1 reply; 15+ messages in thread
From: Joel Brobecker @ 2008-10-07 3:30 UTC (permalink / raw)
To: Michael Snyder; +Cc: gdb-patches, Daniel Jacobowitz, Pedro Alves, teawater
I think that the implementation of this file will change slightly based
on the outcome of some of the discussions we had with the previous
patch. But I noticed a few little things in addition to what Eli
already said.
> + Copyright (C) 2006 Free Software Foundation, Inc.
The copyright date should include 2008.
> + (returns error unles target implements to_set_execdir method). */
^^^^^
unless
> +static void
> +exec_reverse_once (char *cmd, char *args, int from_tty)
This function needs a comment describing what it does. I was almost
at the end of the function before I understood its purpose.
> + add_setshow_enum_cmd ("exec-direction", class_run, exec_direction_names,
> + &exec_direction, "Set direction of execution.\n\
> +Options are 'forward' or 'reverse'.",
> + "Show direction of execution (forward/reverse).",
> + "Tells gdb whether to execute forward or backward.",
> + set_exec_direction_func, show_exec_direction_func,
> + &setlist, &showlist);
Minor nit: i18n.
--
Joel
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [RFA] Reverse Debugging, 5/5
2008-10-07 3:30 ` Joel Brobecker
@ 2008-10-07 18:25 ` Michael Snyder
0 siblings, 0 replies; 15+ messages in thread
From: Michael Snyder @ 2008-10-07 18:25 UTC (permalink / raw)
To: Joel Brobecker; +Cc: gdb-patches, Daniel Jacobowitz, Pedro Alves, teawater
Joel Brobecker wrote:
> I think that the implementation of this file will change slightly based
> on the outcome of some of the discussions we had with the previous
> patch. But I noticed a few little things in addition to what Eli
> already said.
>
>> + Copyright (C) 2006 Free Software Foundation, Inc.
>
> The copyright date should include 2008.
OK
>> + (returns error unles target implements to_set_execdir method). */
> ^^^^^
> unless
>> +static void
>> +exec_reverse_once (char *cmd, char *args, int from_tty)
>
> This function needs a comment describing what it does. I was almost
> at the end of the function before I understood its purpose.
OK
>> + add_setshow_enum_cmd ("exec-direction", class_run, exec_direction_names,
>> + &exec_direction, "Set direction of execution.\n\
>> +Options are 'forward' or 'reverse'.",
>> + "Show direction of execution (forward/reverse).",
>> + "Tells gdb whether to execute forward or backward.",
>> + set_exec_direction_func, show_exec_direction_func,
>> + &setlist, &showlist);
>
> Minor nit: i18n.
OK
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2008-10-07 18:25 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-10-01 19:21 [RFA] Reverse Debugging, 5/5 Michael Snyder
2008-10-02 19:49 ` Eli Zaretskii
2008-10-02 21:42 ` Michael Snyder
2008-10-03 6:30 ` Eli Zaretskii
2008-10-03 18:07 ` Michael Snyder
2008-10-04 8:13 ` Eli Zaretskii
2008-10-04 17:56 ` Michael Snyder
2008-10-02 22:43 ` Michael Snyder
2008-10-03 6:35 ` Eli Zaretskii
2008-10-02 22:54 ` Michael Snyder
2008-10-03 6:25 ` Eli Zaretskii
2008-10-03 17:45 ` Daniel Jacobowitz
2008-10-03 17:58 ` Michael Snyder
2008-10-07 3:30 ` Joel Brobecker
2008-10-07 18:25 ` Michael Snyder
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox