Re: [RFA/RFC] Prec multi-thread support [4/4] record target

[Hui Zhu <teawater@gmail.com>]

Sorry for I forget it.  To use this patch need patch
http://sourceware.org/ml/gdb-patches/2009-11/msg00520.html first.

prec-fix-error-handler.txt is still in discussion.  But looks we are moving on.
If we don't finish prec-fix-error-handler.txt in one or two days, I
will post a new prec-thread-target.txt for cvs head.

BTW, even if new prec-fix-error-handler.txt check in, I still need
post a new prec-thread-target.txt too.  :P

Thanks,
Hui

On Thu, Nov 26, 2009 at 03:35, Michael Snyder <msnyder@vmware.com> wrote:
> Hi Hui,
>
> This one (4 of 4) does not seem to apply cleanly against cvs head
> (record.c is cvs revision 1.40).
>
> Since the rejected diffs seem to have to do with do_record_message,
> I'm guessing that you may have experienced some "bleed" between this
> patch and another one that you're working on.
>
> Can you sort it out?  This patch should only include thread
> support, not fixes for the do_record_message issue.
>
> Thanks,
> Michael
>
>
> Hui Zhu wrote:
>>
>> To record the multithread inferior, record let each thread exec one by
>> one. For example:
>> Inferior have 2 threads t1 and t2.
>> 1. set t1 to inferior_ptid.
>> 2. record_message t1.
>> 3. resume t1.
>> 4. wait t1.
>> 5. Check if t1 get breakpoint or someting.  If so, return.
>> 5. set t2 to inferior_ptid.
>> 6. record_message t2.
>> 7. resume t2.
>> 8. wait t2.
>> 9. Check if t2 get breakpoint or someting.  If so, return.
>> 10. Check if step, if so return.  If not, goto 1.
>>
>> But there is a problem that thread will hang by a lock sometime.  If
>> t1 is waiting a lock and t2 have it.  GDB and inferior will dead lock
>> in there.
>> So record call wait with TARGET_WNOHANG. After we get
>> TARGET_WAITKIND_IGNORE (thread is keep running), we will set
>> tp->record_is_waiting to 1 that flag thread is running.
>> And record set non_stop to 1, because if we want linux-nat handle 2
>> thread running.  This is the only way.  If not set, GDB will get a lot
>> of internal_error.
>> Then the inferior exec will be:
>> 1. set t1 to inferior_ptid.
>> 2. If t1->record_is_waiting is 0, record_message t1.
>> 3. If t1->record_is_waiting is 0, resume t1.
>> 4. wait t1.
>> 5. If get TARGET_WAITKIND_IGNORE, set t1->record_is_waiting to 1.
>> Check if t1 get breakpoint or someting.  If so, return.
>> 5. set t2 to inferior_ptid.
>> 6. If t2->record_is_waiting is 0, record_message t2.
>> 7. If t2->record_is_waiting is 0, resume t2.
>> 8. wait t2.
>> 9. If get TARGET_WAITKIND_IGNORE, set t2->record_is_waiting to 1.
>> Check if t2 get breakpoint or someting.  If so, return.
>> 10. Check if step, if so return.  If not, goto 1.
>>
>> Before record_wait return, it will call iterate_over_threads
>> (record_thread_stop_callback, NULL) to stop each thread that
>> record_is_waiting is 1.
>>
>>
>> And if this is a single thread or just one thread exec, record will
>> exec it like before.
>> record_new_thread_observer and record_thread_exit_observer make record
>> know how much thread in inferior.
>>
>>
>> 2009-11-25  Hui Zhu  <teawater@gmail.com>
>>
>>        * record.c (observer.h): New include.
>>        (RECORD_THREAD): New macro.
>>        (record_type): Add record_ptid.
>>        (record_entry): Add ptid.
>>        (record_prev_ptid,
>>        record_new_thread_observer,
>>        record_thread_exit_observer): New variable.
>>        (record_ptid_alloc, record_ptid_release): New function.
>>        (record_entry_release): Add record_ptid.
>>        (record_entry_remove_from_list,
>>        record_arch_list_add_ptid): New function.
>>        (record_message_args): Remove regcache.
>>        (record_message): Add code for multithread.
>>        (do_record_message): Remove regcache.
>>        (record_exec_insn): Remove regcache, gdbarch.  Add
>>        regcache_p, gdbarch_p, aspace_p.
>>        Add suport for record_ptid.
>>        (record_thread_number): New variable.
>>        (record_thread_number_count,
>>        record_thread_number_reset,
>>        record_new_thread_handler,
>>        record_thread_exit_handler): New function.
>>        (record_open_1): Add record_new_thread_observer
>>        and record_thread_exit_observer.
>>        (record_open): Add record_prev_ptid
>>        and record_thread_number_reset.
>>        (record_close): Add record_new_thread_observer
>>        and record_thread_exit_observer.
>>        (record_resume_ptid, record_resume_signal): New variable.
>>        (record_resume): Add code for multithread.
>>        (record_thread_wait_args): New struct.
>>        (record_thread_reset_callback,
>>        record_thread_stop_callback,
>>        record_thread_wait,
>>        record_thread_wait_callback,
>>        record_wait_signal_cleanups) New function.
>>        (record_wait_cleanups): Change to record_wait_replay_cleanups.
>>        (record_wait_mthread_cleanups): New function.
>>        (record_wait): Add code for multithread.
>>        (record_registers_change): Ditto.
>>        (record_xfer_partial): Ditto.
>>        (record_restore): Add code restore record_ptid entry.
>>        (cmd_record_save): Add code save record_ptid entry.
>>        (record_goto_insn): Update argument to call record_exec_insn.
>>        record.h (record_step): New extern.
>>
>> ---
>>  record.c |  734
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++---------
>>  record.h |    1
>>  2 files changed, 631 insertions(+), 104 deletions(-)
>>
>> --- a/record.c
>> +++ b/record.c
>> @@ -30,6 +30,7 @@
>>  #include "record.h"
>>  #include "elf-bfd.h"
>>  #include "gcore.h"
>> +#include "observer.h"
>>
>>  #include <signal.h>
>>
>> @@ -60,6 +61,8 @@
>>
>>  #define RECORD_FILE_MAGIC      netorder32(0x20091016)
>>
>> +#define RECORD_THREAD  (current_target.beneath->beneath->to_stratum ==
>> thread_stratum)
>> +
>>  /* These are the core structs of the process record functionality.
>>
>>    A record_entry is a record of the value change of a register
>> @@ -106,7 +109,8 @@ enum record_type
>>  {
>>   record_end = 0,
>>   record_reg,
>> -  record_mem
>> +  record_mem,
>> +  record_ptid
>>  };
>>
>>  /* This is the data structure that makes up the execution log.
>> @@ -144,6 +148,8 @@ struct record_entry
>>     struct record_reg_entry reg;
>>     /* mem */
>>     struct record_mem_entry mem;
>> +    /* ptid */
>> +    ptid_t ptid;
>>     /* end */
>>     struct record_end_entry end;
>>   } u;
>> @@ -196,10 +202,19 @@ static int record_insn_num = 0;
>>    than count of insns presently in execution log).  */
>>  static ULONGEST record_insn_count;
>>
>> +/* In record mode, it's the step of next resume.  */
>> +int record_step;
>> +
>> +/* The current inferior_ptid that is recorded.  */
>> +static ptid_t record_prev_ptid;
>> +
>>  /* The target_ops of process record.  */
>>  static struct target_ops record_ops;
>>  static struct target_ops record_core_ops;
>>
>> +static struct observer *record_new_thread_observer = NULL;
>> +static struct observer *record_thread_exit_observer = NULL;
>> +
>>  /* The beneath function pointers.  */
>>  static struct target_ops *record_beneath_to_resume_ops;
>>  static void (*record_beneath_to_resume) (struct target_ops *, ptid_t,
>> int,
>> @@ -288,6 +303,27 @@ record_mem_release (struct record_entry
>>   xfree (rec);
>>  }
>>
>> +/* Alloc a record_ptid record entry.  */
>> +
>> +static inline struct record_entry *
>> +record_ptid_alloc (void)
>> +{
>> +  struct record_entry *rec;
>> +
>> +  rec = (struct record_entry *) xcalloc (1, sizeof (struct
>> record_entry));
>> +  rec->type = record_ptid;
>> +
>> +  return rec;
>> +}
>> +
>> +/* Free a record_ptid record entry.  */
>> +
>> +static inline void
>> +record_ptid_release (struct record_entry *rec)
>> +{
>> +  xfree (rec);
>> +}
>> +
>>  /* Alloc a record_end record entry.  */
>>
>>  static inline struct record_entry *
>> @@ -324,6 +360,9 @@ record_entry_release (struct record_entr
>>   case record_mem:
>>     record_mem_release (rec);
>>     break;
>> +  case record_ptid:
>> +    record_ptid_release (rec);
>> +    break;
>>   case record_end:
>>     record_end_release (rec);
>>     break;
>> @@ -331,6 +370,16 @@ record_entry_release (struct record_entr
>>   return type;
>>  }
>>
>> +/* Remove one record entry from record_list.  */
>> +
>> +static void
>> +record_entry_remove_from_list (struct record_entry *rec)
>> +{
>> +  if (rec->next)
>> +    rec->next->prev = rec->prev;
>> +  rec->prev->next = rec->next;
>> +}
>> +
>>  /* Free all record entries in list pointed to by REC.  */
>>
>>  static void
>> @@ -512,6 +561,24 @@ record_arch_list_add_mem (CORE_ADDR addr
>>   return 0;
>>  }
>>
>> +static void
>> +record_arch_list_add_ptid (ptid_t *ptid)
>> +{
>> +  struct record_entry *rec;
>> +
>> +  if (record_debug > 1)
>> +    fprintf_unfiltered (gdb_stdlog,
>> +                       "Process record: add ptid = %s to "
>> +                       "record list.\n",
>> +                       target_pid_to_str (*ptid));
>> +
>> +  rec = record_ptid_alloc ();
>> +
>> +  rec->u.ptid = *ptid;
>> +
>> +  record_arch_list_add (rec);
>> +}
>> +
>>  /* Add a record_end type struct record_entry to record_arch_list.  */
>>
>>  int
>> @@ -573,7 +640,6 @@ record_arch_list_cleanups (void *ignore)
>>    record_arch_list, and add it to record_list.  */
>>
>>  struct record_message_args {
>> -  struct regcache *regcache;
>>   enum target_signal signal;
>>  };
>>
>> @@ -582,15 +648,21 @@ record_message (void *args)
>>  {
>>   int ret;
>>   struct record_message_args *myargs = args;
>> -  struct gdbarch *gdbarch = get_regcache_arch (myargs->regcache);
>> +  struct regcache *regcache;
>> +  struct gdbarch *gdbarch;
>>   struct cleanup *old_cleanups = make_cleanup (record_arch_list_cleanups,
>> 0);
>>
>> +  record_step = 1;
>> +
>>   record_arch_list_head = NULL;
>>   record_arch_list_tail = NULL;
>>
>>   /* Check record_insn_num.  */
>>   record_check_insn_num (1);
>>
>> +  regcache = get_current_regcache ();
>> +  gdbarch = get_regcache_arch (regcache);
>> +
>>   /* If gdb sends a signal value to target_resume,
>>      save it in the 'end' field of the previous instruction.
>>
>> @@ -613,20 +685,45 @@ record_message (void *args)
>>      But we should still deliver the signal to gdb during the replay,
>>      if we delivered it during the recording.  Therefore we should
>>      record the signal during record_wait, not record_resume.  */
>> -  if (record_list != &record_first)    /* FIXME better way to check */
>> +  if (ptid_equal (record_prev_ptid, inferior_ptid))
>>     {
>> -      gdb_assert (record_list->type == record_end);
>> -      record_list->u.end.sigval = myargs->signal;
>> +      if (myargs->signal != TARGET_SIGNAL_0)
>> +        {
>> +          gdb_assert (record_list->type == record_end);
>> +          record_list->u.end.sigval = myargs->signal;
>> +        }
>> +    }
>> +  else
>> +    {
>> +      if (myargs->signal != TARGET_SIGNAL_0)
>> +        {
>> +          struct record_entry *rec;
>> +          for (rec = record_list; rec != record_first.next; rec =
>> rec->prev)
>> +            {
>> +              if (rec->type == record_ptid
>> +                  && ptid_equal (rec->u.ptid, inferior_ptid))
>> +                {
>> +                  gdb_assert (rec->prev->type == record_end);
>> +                  rec->prev->u.end.sigval = myargs->signal;
>> +                  break;
>> +                }
>> +            }
>> +        }
>> +
>> +      /* If the inferior_ptid change (inferior_ptid is not same with
>> +         record_prev_ptid), record record_prev_ptid to record list.  */
>> +      record_arch_list_add_ptid (&record_prev_ptid);
>> +      record_prev_ptid = inferior_ptid;
>>     }
>>
>>   if (myargs->signal == TARGET_SIGNAL_0
>>       || !gdbarch_process_record_signal_p (gdbarch))
>>     ret = gdbarch_process_record (gdbarch,
>> -                                 myargs->regcache,
>> -                                 regcache_read_pc (myargs->regcache));
>> +                                 regcache,
>> +                                 regcache_read_pc (regcache));
>>   else
>>     ret = gdbarch_process_record_signal (gdbarch,
>> -                                        myargs->regcache,
>> +                                        regcache,
>>                                         myargs->signal);
>>
>>   if (ret > 0)
>> @@ -649,12 +746,10 @@ record_message (void *args)
>>  }
>>
>>  static int
>> -do_record_message (struct regcache *regcache,
>> -                  enum target_signal signal, int catch)
>> +do_record_message (enum target_signal signal, int catch)
>>  {
>>   struct record_message_args args;
>>
>> -  args.regcache = regcache;
>>   args.signal = signal;
>>
>>   if (catch)
>> @@ -688,9 +783,14 @@ static int record_hw_watchpoint = 0;
>>    entries and memory entries, followed by an 'end' entry.  */
>>
>>  static inline void
>> -record_exec_insn (struct regcache *regcache, struct gdbarch *gdbarch,
>> +record_exec_insn (struct regcache **regcache_p,
>> +                  struct gdbarch **gdbarch_p,
>> +                  struct address_space **aspace_p,
>>                  struct record_entry *entry)
>>  {
>> +  struct regcache *regcache = *regcache_p;
>> +  struct gdbarch *gdbarch = *gdbarch_p;
>> +
>>   switch (entry->type)
>>     {
>>     case record_reg: /* reg */
>> @@ -769,6 +869,110 @@ record_exec_insn (struct regcache *regca
>>           }
>>       }
>>       break;
>> +
>> +    case record_ptid: /* ptid */
>> +      {
>> +        ptid_t tmp_ptid;
>> +
>> +        if (record_debug > 1)
>> +          fprintf_unfiltered (gdb_stdlog,
>> +                              "Process record: record_ptid %s to "
>> +                              "inferior ptid = %s.\n",
>> +                              host_address_to_string (entry),
>> +                              target_pid_to_str (entry->u.ptid));
>> +
>> +        if (!ptid_equal (entry->u.ptid, null_ptid))
>> +          {
>> +            inferior_ptid = entry->u.ptid;
>> +            *regcache_p = get_current_regcache ();
>> +            *gdbarch_p = get_regcache_arch (*regcache_p);
>> +            *aspace_p = get_regcache_aspace (*regcache_p);
>> +          }
>> +        tmp_ptid = entry->u.ptid;
>> +        entry->u.ptid = record_prev_ptid;
>> +        record_prev_ptid = tmp_ptid;
>> +      }
>> +      break;
>> +    }
>> +}
>> +
>> +static int record_thread_number;
>> +
>> +static int
>> +record_thread_number_count (struct thread_info *thread, void *unused)
>> +{
>> +  if (thread->state_ != THREAD_EXITED)
>> +    record_thread_number ++;
>> +
>> +  return 0;
>> +}
>> +
>> +static void
>> +record_thread_number_reset (void)
>> +{
>> +  record_thread_number = 0;
>> +
>> +  if (RECORD_THREAD)
>> +    iterate_over_threads (record_thread_number_count, NULL);
>> +}
>> +
>> +static void
>> +record_new_thread_handler (struct thread_info *tp)
>> +{
>> +  tp->record_is_waiting = 0;
>> +
>> +  record_thread_number_reset ();
>> +}
>> +
>> +static void
>> +record_thread_exit_handler (struct thread_info *tp, int silent)
>> +{
>> +  struct record_entry *rec;
>> +  ptid_t cur_ptid = record_prev_ptid;
>> +
>> +  if (!record_first.next)
>> +    return;
>> +
>> +  tp->record_is_waiting = 0;
>> +
>> +  if (tp->state_ != THREAD_EXITED)
>> +    {
>> +      gdb_assert (record_thread_number > 0);
>> +      record_thread_number --;
>> +    }
>> +
>> +  /* Delete all the record_reg and record_ptid for tp->ptid
>> +     from record list.  */
>> +  for (rec = record_list; rec != &record_first;)
>> +    {
>> +      struct record_entry *tmp = rec;
>> +      rec = rec->prev;
>> +
>> +      switch (tmp->type) {
>> +        case record_reg:
>> +          /* If this record_reg is for tp->ptid, delte it.  */
>> +          if (ptid_equal (cur_ptid, tp->ptid))
>> +            {
>> +              record_entry_remove_from_list (tmp);
>> +              record_entry_release (tmp);
>> +            }
>> +          break;
>> +        case record_ptid:
>> +          /* If this record_ptid is for tp->ptid, delte it.  */
>> +          cur_ptid = tmp->u.ptid;
>> +          if (ptid_equal (record_prev_ptid, tp->ptid))
>> +            {
>> +              record_prev_ptid = cur_ptid;
>> +              record_entry_remove_from_list (tmp);
>> +              record_entry_release (tmp);
>> +            }
>> +          else if (ptid_equal (cur_ptid, tp->ptid))
>> +            {
>> +              record_entry_remove_from_list (tmp);
>> +              record_entry_release (tmp);
>> +            }
>> +          break;
>> +      }
>>     }
>>  }
>>
>> @@ -865,6 +1069,9 @@ record_open_1 (char *name, int from_tty)
>>     error (_("Could not find 'to_remove_breakpoint' method on the
>> target stack."));
>>
>>   push_target (&record_ops);
>> +
>> +  record_new_thread_observer = observer_attach_new_thread
>> (record_new_thread_handler);
>> +  record_thread_exit_observer = observer_attach_thread_exit
>> (record_thread_exit_handler);
>>  }
>>
>>  /* "to_open" target method.  Open the process record target.  */
>> @@ -934,6 +1141,7 @@ record_open (char *name, int from_tty)
>>   record_insn_count = 0;
>>   record_list = &record_first;
>>   record_list->next = NULL;
>> +  record_prev_ptid = null_ptid;
>>
>>   /* Set the tmp beneath pointers to beneath pointers.  */
>>   record_beneath_to_resume_ops = tmp_to_resume_ops;
>> @@ -953,6 +1161,8 @@ record_open (char *name, int from_tty)
>>     record_core_open_1 (name, from_tty);
>>   else
>>     record_open_1 (name, from_tty);
>> +
>> +  record_thread_number_reset ();
>>  }
>>
>>  /* "to_close" target method.  Close the process record target.  */
>> @@ -965,6 +1175,17 @@ record_close (int quitting)
>>   if (record_debug)
>>     fprintf_unfiltered (gdb_stdlog, "Process record: record_close\n");
>>
>> +  if (record_new_thread_observer)
>> +    {
>> +      observer_detach_new_thread (record_new_thread_observer);
>> +      record_new_thread_observer = NULL;
>> +    }
>> +  if (record_thread_exit_observer)
>> +    {
>> +      observer_detach_thread_exit (record_thread_exit_observer);
>> +      record_thread_exit_observer = NULL;
>> +    }
>> +
>>   record_list_release (record_list);
>>
>>   /* Release record_core_regbuf.  */
>> @@ -987,6 +1208,8 @@ record_close (int quitting)
>>  }
>>
>>  static int record_resume_step = 0;
>> +static ptid_t record_resume_ptid;
>> +static enum target_signal record_resume_signal = TARGET_SIGNAL_0;
>>
>>  /* "to_resume" target method.  Resume the process record target.  */
>>
>> @@ -998,10 +1221,201 @@ record_resume (struct target_ops *ops, p
>>
>>   if (!RECORD_IS_REPLAY)
>>     {
>> -      do_record_message (get_current_regcache (), signal, 0);
>> -      record_beneath_to_resume (record_beneath_to_resume_ops, ptid, 1,
>> -                                signal);
>> +      record_resume_ptid = ptid;
>> +      if (record_thread_number > 1 && !ptid_equal (ptid, inferior_ptid))
>> +        {
>> +          record_resume_signal = signal;
>> +        }
>> +      else
>> +        {
>> +          do_record_message (signal, 0);
>> +          record_beneath_to_resume (record_beneath_to_resume_ops, ptid,
>> +                                    record_step, signal);
>> +        }
>> +    }
>> +}
>> +
>> +struct record_thread_wait_args {
>> +  ptid_t real_inferior_ptid;
>> +  struct thread_info *real_inferior_tp;
>> +  struct target_waitstatus real_inferior_status;
>> +
>> +  struct target_waitstatus *status;
>> +  int options;
>> +};
>> +
>> +static int
>> +record_thread_reset_callback (struct thread_info *tp, void *data)
>> +{
>> +  tp->record_is_waiting = 0;
>> +
>> +  return 0;
>> +}
>> +
>> +static int
>> +record_thread_stop_callback (struct thread_info *tp, void *data)
>> +{
>> +  if (tp->state_ != THREAD_EXITED && tp->record_is_waiting)
>> +    target_stop (tp->ptid);
>> +
>> +  return 0;
>> +}
>> +
>> +/* Return 1 if it is not a simple step.
>> +   Return 0 if it is a simple step.  */
>> +
>> +static int
>> +record_thread_wait (ptid_t ptid, struct target_waitstatus *status,
>> +                    int options, ptid_t *ret_ptid)
>> +{
>> +  ptid_t wait_ptid = record_beneath_to_wait (record_beneath_to_wait_ops,
>> +                                             ptid, status, options);
>> +
>> +  if (ret_ptid)
>> +    *ret_ptid = wait_ptid;
>> +
>> +  if (status->kind == TARGET_WAITKIND_IGNORE)
>> +    return 0;
>> +
>> +  inferior_ptid = wait_ptid;
>> +
>> +  /* Is this a SIGTRAP?  */
>> +  if (status->kind == TARGET_WAITKIND_STOPPED
>> +      && status->value.sig == TARGET_SIGNAL_TRAP)
>> +    {
>> +      CORE_ADDR tmp_pc;
>> +      struct regcache *regcache;
>> +      struct gdbarch *gdbarch;
>> +      struct address_space *aspace;
>> +      CORE_ADDR decr_pc_after_break;
>> +
>> +      /* Yes -- this is likely our single-step finishing,
>> +         but check if there's any reason the core would be
>> +         interested in the event.  */
>> +
>> +      registers_changed ();
>> +      regcache = get_current_regcache ();
>> +      gdbarch = get_regcache_arch (regcache);
>> +      tmp_pc = regcache_read_pc (regcache);
>> +      aspace = get_regcache_aspace (regcache);
>> +      decr_pc_after_break = gdbarch_decr_pc_after_break (gdbarch);
>> +
>> +      /* To check if this is breakpoint need the right pc, if this is not
>> +         a single step.  So we need adjust tmp_pc to make it right.  */
>> +      if (!record_step)
>> +        tmp_pc -= decr_pc_after_break;
>> +
>> +      if (target_stopped_by_watchpoint ())
>> +        {
>> +          /* Always interested in watchpoints.  */
>> +          return 1;
>> +        }
>> +      else if (breakpoint_inserted_here_p (aspace, tmp_pc))
>> +        {
>> +          /* There is a breakpoint here.  Let the core handle it.  */
>> +          if (record_step
>> +              && software_breakpoint_inserted_here_p (aspace, tmp_pc))
>> +            {
>> +              if (decr_pc_after_break)
>> +                regcache_write_pc (regcache, tmp_pc +
>> decr_pc_after_break);
>> +            }
>> +
>> +          return 1;
>> +        }
>> +
>> +      return 0;
>> +    }
>> +
>> +  return 1;
>> +}
>> +
>> +static int
>> +record_thread_wait_callback (struct thread_info *tp, void *data)
>> +{
>> +  struct record_thread_wait_args *args = data;
>> +  enum target_signal sig = TARGET_SIGNAL_0;
>> +  int options = args->options;
>> +  int ret;
>> +
>> +  if (tp->state_ == THREAD_EXITED)
>> +    return 0;
>> +
>> +  if (record_debug > 1)
>> +    fprintf_unfiltered (gdb_stdlog,
>> +                        "Process record: record_thread_wait_callback "
>> +                        "resume %s\n", target_pid_to_str (tp->ptid));
>> +
>> +  inferior_ptid = tp->ptid;
>> +
>> +  if (!tp->record_is_waiting)
>> +    {
>> +      if (record_resume_signal != TARGET_SIGNAL_0
>> +          && ptid_equal (inferior_ptid, args->real_inferior_ptid))
>> +        {
>> +          sig = record_resume_signal;
>> +          record_resume_signal = TARGET_SIGNAL_0;
>> +        }
>> +      if (!do_record_message (sig, 1))
>> +        {
>> +          args->status->kind = TARGET_WAITKIND_STOPPED;
>> +          args->status->value.sig = TARGET_SIGNAL_0;
>> +          return 1;
>> +        }
>> +
>> +      if (record_step == 0)
>> +        {
>> +          /* The next insn is a sys_clone.  */
>> +          iterate_over_threads (record_thread_stop_callback, NULL);
>> +          non_stop = 0;
>> +        }
>> +
>> +      record_beneath_to_resume (record_beneath_to_resume_ops,
>> inferior_ptid,
>> +                                record_step, sig);
>> +    }
>> +  else
>> +    {
>> +      if (record_debug > 1)
>> +        fprintf_unfiltered (gdb_stdlog,
>> +                            "Process record: record_thread_wait_callback
>> "
>> +                            "not resume %s\n", target_pid_to_str
>> (tp->ptid));
>> +    }
>> +
>> +  if (record_step)
>> +    options |= TARGET_WNOHANG;
>> +  ret = record_thread_wait (inferior_ptid, args->status,
>> +                            options, NULL);
>> +  if (args->status->kind == TARGET_WAITKIND_IGNORE)
>> +    {
>> +      if (!tp->record_is_waiting)
>> +        {
>> +          tp->record_is_waiting = 1;
>> +          if (record_debug > 1)
>> +            fprintf_unfiltered (gdb_stdlog,
>> +                                "Process record:
>> record_thread_wait_callback "
>> +                                "start waiting %s.\n",
>> +                               target_pid_to_str (tp->ptid));
>> +        }
>> +    }
>> +  else
>> +    {
>> +      if (tp->record_is_waiting)
>> +        {
>> +          tp->record_is_waiting = 0;
>> +          if (record_debug > 1)
>> +            fprintf_unfiltered (gdb_stdlog,
>> +                                "Process record:
>> record_thread_wait_callback "
>> +                                "stop waiting %s.\n",
>> +                               target_pid_to_str (tp->ptid));
>> +        }
>> +
>> +      if (!args->real_inferior_tp
>> +          && ptid_equal (inferior_ptid, args->real_inferior_ptid))
>> +        args->real_inferior_tp = tp;
>> +      if (args->real_inferior_tp == tp)
>> +        args->real_inferior_status = *args->status;
>>     }
>> +
>> +  return ret;
>>  }
>>
>>  static int record_get_sig = 0;
>> @@ -1023,7 +1437,13 @@ record_sig_handler (int signo)
>>  }
>>
>>  static void
>> -record_wait_cleanups (void *ignore)
>> +record_wait_signal_cleanups (void *ignore)
>> +{
>> +  signal (SIGINT, handle_sigint);
>> +}
>> +
>> +static void
>> +record_wait_replay_cleanups (void *ignore)
>>  {
>>   if (execution_direction == EXEC_REVERSE)
>>     {
>> @@ -1034,6 +1454,17 @@ record_wait_cleanups (void *ignore)
>>     record_list = record_list->prev;
>>  }
>>
>> +static void
>> +record_wait_mthread_cleanups (void *ignore)
>> +{
>> +  non_stop = 1;
>> +
>> +  /* Stop the threads that still running.  */
>> +  iterate_over_threads (record_thread_stop_callback, NULL);
>> +
>> +  non_stop = 0;
>> +}
>> +
>>  /* "to_wait" target method for process record target.
>>
>>    In record mode, the target is always run in singlestep mode
>> @@ -1052,7 +1483,10 @@ record_wait (struct target_ops *ops,
>>             ptid_t ptid, struct target_waitstatus *status,
>>             int options)
>>  {
>> +  ptid_t ret_ptid;
>>   struct cleanup *set_cleanups = record_gdb_operation_disable_set ();
>> +  struct cleanup *signal_cleanups
>> +                    = make_cleanup (record_wait_signal_cleanups, 0);
>>
>>   if (record_debug)
>>     fprintf_unfiltered (gdb_stdlog,
>> @@ -1060,93 +1494,112 @@ record_wait (struct target_ops *ops,
>>                        "record_resume_step = %d\n",
>>                        record_resume_step);
>>
>> +  record_get_sig = 0;
>> +  signal (SIGINT, record_sig_handler);
>> +
>>   if (!RECORD_IS_REPLAY && ops != &record_core_ops)
>>     {
>> -      if (record_resume_step)
>> -       {
>> -         /* This is a single step.  */
>> -         return record_beneath_to_wait (record_beneath_to_wait_ops,
>> -                                        ptid, status, options);
>> -       }
>> -      else
>> -       {
>> -         /* This is not a single step.  */
>> -         ptid_t ret;
>> -         CORE_ADDR tmp_pc;
>> +      /* Record mode.  */
>> +      if (record_thread_number > 1 && !ptid_equal (record_resume_ptid,
>> +                                                   inferior_ptid))
>> +        {
>> +          /* Multi-threads record.  */
>> +          struct record_thread_wait_args args;
>> +          struct thread_info *tp;
>> +          struct cleanup *mthread_cleanups
>> +                            = make_cleanup (record_wait_mthread_cleanups,
>> 0);
>>
>> -         while (1)
>> -           {
>> -             ret = record_beneath_to_wait (record_beneath_to_wait_ops,
>> -                                           ptid, status, options);
>> +          args.real_inferior_ptid = inferior_ptid;
>> +          args.real_inferior_tp = NULL;
>> +          args.status = status;
>> +          args.options = options;
>> +          non_stop = 1;
>> +          iterate_over_threads (record_thread_reset_callback, NULL);
>>
>> -             /* Is this a SIGTRAP?  */
>> -             if (status->kind == TARGET_WAITKIND_STOPPED
>> -                 && status->value.sig == TARGET_SIGNAL_TRAP)
>> -               {
>> -                 struct regcache *regcache;
>> -                 struct address_space *aspace;
>> +          while (1)
>> +            {
>> +              tp = iterate_over_threads (record_thread_wait_callback,
>> &args);
>> +              if (tp)
>> +                break;
>> +              if (record_resume_step
>> +                  && args.real_inferior_tp
>> +                  && !args.real_inferior_tp->record_is_waiting)
>> +                {
>> +                  *status = args.real_inferior_status;
>> +                  inferior_ptid = args.real_inferior_ptid;
>> +                  break;
>> +                }
>> +            }
>>
>> -                 /* Yes -- this is likely our single-step finishing,
>> -                    but check if there's any reason the core would be
>> -                    interested in the event.  */
>> +          do_cleanups (mthread_cleanups);
>>
>> -                 registers_changed ();
>> -                 regcache = get_current_regcache ();
>> -                 tmp_pc = regcache_read_pc (regcache);
>> -                 aspace = get_regcache_aspace (regcache);
>> +          if (tp && tp->state_ != THREAD_EXITED)
>> +            {
>> +              inferior_ptid = tp->ptid;
>> +            }
>> +          ret_ptid = inferior_ptid;
>> +        }
>> +      else
>> +        {
>> +          /* Single-thread record.  */
>> +          if (record_thread_number > 1)
>> +            ptid = record_resume_ptid;
>>
>> -                 if (target_stopped_by_watchpoint ())
>> -                   {
>> -                     /* Always interested in watchpoints.  */
>> -                   }
>> -                 else if (breakpoint_inserted_here_p (aspace, tmp_pc))
>> -                   {
>> -                     /* There is a breakpoint here.  Let the core
>> -                        handle it.  */
>> -                     if (software_breakpoint_inserted_here_p (aspace,
>> tmp_pc))
>> -                       {
>> -                         struct gdbarch *gdbarch = get_regcache_arch
>> (regcache);
>> -                         CORE_ADDR decr_pc_after_break
>> -                           = gdbarch_decr_pc_after_break (gdbarch);
>> -                         if (decr_pc_after_break)
>> -                           regcache_write_pc (regcache,
>> -                                              tmp_pc +
>> decr_pc_after_break);
>> -                       }
>> -                   }
>> -                 else
>> -                   {
>> -                     /* This must be a single-step trap.  Record the
>> -                        insn and issue another step.  */
>> -                     if (!do_record_message (regcache, TARGET_SIGNAL_0,
>> 1))
>> -                       {
>> -                           status->kind = TARGET_WAITKIND_STOPPED;
>> -                           status->value.sig = TARGET_SIGNAL_0;
>> -                           break;
>> -                       }
>> +          if (record_resume_step)
>> +            {
>> +             /* This is a single step.  */
>> +             ret_ptid = record_beneath_to_wait
>> (record_beneath_to_wait_ops,
>> +                                                 ptid, status, options);
>> +           }
>> +          else
>> +            {
>> +              /* This is not a single step.  */
>> +              while (1)
>> +                {
>> +                  if (record_thread_wait (ptid, status,
>> +                                          options, &ret_ptid))
>> +                    break;
>>
>> -                     record_beneath_to_resume
>> (record_beneath_to_resume_ops,
>> -                                               ptid, 1,
>> -                                               TARGET_SIGNAL_0);
>> -                     continue;
>> -                   }
>> -               }
>> +                  if (record_resume_step)
>> +                    break;
>>
>> -             /* The inferior is broken by a breakpoint or a signal.  */
>> -             break;
>> -           }
>> +                  /* There is not a breakpoint, and gdb is not
>> +                     stepping, therefore gdb will not stop.
>> +                     Therefore we will not return to gdb.
>> +                     Record the insn and resume.  */
>> +                 if (!do_record_message (TARGET_SIGNAL_0, 1))
>> +                    {
>> +                      ret_ptid = inferior_ptid;
>> +                      status->kind = TARGET_WAITKIND_STOPPED;
>> +                      status->value.sig = TARGET_SIGNAL_0;
>> +                      break;
>> +                    }
>>
>> -         return ret;
>> -       }
>> +                 record_beneath_to_resume (record_beneath_to_resume_ops,
>> +                                           record_resume_ptid,
>> record_step,
>> +                                           TARGET_SIGNAL_0);
>> +                }
>> +            }
>> +        }
>>     }
>>   else
>>     {
>> -      struct regcache *regcache = get_current_regcache ();
>> -      struct gdbarch *gdbarch = get_regcache_arch (regcache);
>> -      struct address_space *aspace = get_regcache_aspace (regcache);
>> +      /* Replay mode.  */
>> +      struct regcache *regcache;
>> +      struct gdbarch *gdbarch;
>> +      struct address_space *aspace;
>>       int continue_flag = 1;
>>       int first_record_end = 1;
>> -      struct cleanup *old_cleanups = make_cleanup (record_wait_cleanups,
>> 0);
>> +      struct cleanup *old_cleanups =
>> +        make_cleanup (record_wait_replay_cleanups, 0);
>>       CORE_ADDR tmp_pc;
>> +      ptid_t old_inferior_ptid = inferior_ptid;
>> +
>> +      if (!ptid_equal (record_prev_ptid, null_ptid))
>> +        inferior_ptid = record_prev_ptid;
>> +      regcache = get_current_regcache ();
>> +      gdbarch = get_regcache_arch (regcache);
>> +      aspace = get_regcache_aspace (regcache);
>>
>>       record_hw_watchpoint = 0;
>>       status->kind = TARGET_WAITKIND_STOPPED;
>> @@ -1173,8 +1626,6 @@ record_wait (struct target_ops *ops,
>>            }
>>        }
>>
>> -      record_get_sig = 0;
>> -      signal (SIGINT, record_sig_handler);
>>       /* If GDB is in terminal_inferior mode, it will not get the signal.
>>          And in GDB replay mode, GDB doesn't need to be in
>> terminal_inferior
>>          mode, because inferior will not executed.
>> @@ -1205,7 +1656,7 @@ record_wait (struct target_ops *ops,
>>              break;
>>            }
>>
>> -          record_exec_insn (regcache, gdbarch, record_list);
>> +          record_exec_insn (&regcache, &gdbarch, &aspace, record_list);
>>
>>          if (record_list->type == record_end)
>>            {
>> @@ -1228,19 +1679,20 @@ record_wait (struct target_ops *ops,
>>                     In EXEC_FORWARD mode, this is the record_end of
>> current
>>                     instruction.  */
>>                  /* step */
>> -                 if (record_resume_step)
>> +                 if (record_resume_step
>> +                      && ptid_equal (inferior_ptid, old_inferior_ptid))
>>                    {
>>                      if (record_debug > 1)
>>                        fprintf_unfiltered (gdb_stdlog,
>>                                            "Process record: step.\n");
>> -                     continue_flag = 0;
>> +                      continue_flag = 0;
>>                    }
>>
>>                  /* check breakpoint */
>>                  tmp_pc = regcache_read_pc (regcache);
>>                  if (breakpoint_inserted_here_p (aspace, tmp_pc))
>>                    {
>> -                     int decr_pc_after_break
>> +                     CORE_ADDR decr_pc_after_break
>>                        = gdbarch_decr_pc_after_break (gdbarch);
>>
>>                      if (record_debug)
>> @@ -1288,22 +1740,23 @@ Process record: hit hw watchpoint.\n");
>>        }
>>       while (continue_flag);
>>
>> -      signal (SIGINT, handle_sigint);
>> -
>>  replay_out:
>> -      if (record_get_sig)
>> -       status->value.sig = TARGET_SIGNAL_INT;
>> -      else if (record_list->u.end.sigval != TARGET_SIGNAL_0)
>> +      if (record_list->u.end.sigval != TARGET_SIGNAL_0)
>>        /* FIXME: better way to check */
>>        status->value.sig = record_list->u.end.sigval;
>>       else
>>        status->value.sig = TARGET_SIGNAL_TRAP;
>> +      ret_ptid = inferior_ptid;
>>
>>       discard_cleanups (old_cleanups);
>>     }
>>
>> +  if (record_get_sig)
>> +    status->value.sig = TARGET_SIGNAL_INT;
>> +
>> +  do_cleanups (signal_cleanups);
>>   do_cleanups (set_cleanups);
>> -  return inferior_ptid;
>> +  return ret_ptid;
>>  }
>>
>>  static int
>> @@ -1384,6 +1837,12 @@ record_registers_change (struct regcache
>>   record_arch_list_head = NULL;
>>   record_arch_list_tail = NULL;
>>
>> +  if (!ptid_equal (record_prev_ptid, inferior_ptid))
>> +    {
>> +      record_arch_list_add_ptid (&record_prev_ptid);
>> +      record_prev_ptid = inferior_ptid;
>> +    }
>> +
>>   if (regnum < 0)
>>     {
>>       int i;
>> @@ -1506,6 +1965,11 @@ record_xfer_partial (struct target_ops *
>>       /* Record registers change to list as an instruction.  */
>>       record_arch_list_head = NULL;
>>       record_arch_list_tail = NULL;
>> +      if (!ptid_equal (record_prev_ptid, inferior_ptid))
>> +        {
>> +          record_arch_list_add_ptid (&record_prev_ptid);
>> +          record_prev_ptid = inferior_ptid;
>> +        }
>>       if (record_arch_list_add_mem (offset, len))
>>        {
>>          record_list_release (record_arch_list_tail);
>> @@ -2061,6 +2525,17 @@ info_record_command (char *args, int fro
>>        4 bytes: memory length (network byte order).
>>        8 bytes: memory address (network byte order).
>>        n bytes: memory value (n == memory length).
>> +     record_mem:
>> +       1 byte:  record type (record_mem, see enum record_type).
>> +       4 bytes: memory length (network byte order).
>> +       8 bytes: memory address (network byte order).
>> +       n bytes: memory value (n == memory length).
>> +     record_ptid:
>> +       1 byte:  record type (record_ptid, see enum record_type).
>> +       8 bytes: process id (network byte order).
>> +                It must same with CORELOW_PID (1).
>> +       8 bytes: lightweight process id (network byte order).
>> +       8 bytes: thread id (network byte order).
>>
>>  */
>>
>> @@ -2222,6 +2697,32 @@ record_restore (void)
>>                                rec->u.mem.len);
>>           break;
>>
>> +        case record_ptid: /* ptid */
>> +          rec = record_ptid_alloc ();
>> +
>> +         /* Get Process id.  */
>> +         bfdcore_read (core_bfd, osec, &addr,
>> +                       sizeof (addr), &bfd_offset);
>> +         rec->u.ptid.pid = (int) netorder64 (addr);
>> +
>> +         /* Get Lightweight process id.  */
>> +         bfdcore_read (core_bfd, osec, &addr,
>> +                       sizeof (addr), &bfd_offset);
>> +         rec->u.ptid.lwp = (long) netorder64 (addr);
>> +
>> +         /* Get Thread id.  */
>> +         bfdcore_read (core_bfd, osec, &addr,
>> +                       sizeof (addr), &bfd_offset);
>> +         rec->u.ptid.tid = (long) netorder64 (addr);
>> +
>> +         if (record_debug)
>> +           fprintf_unfiltered (gdb_stdlog, "\
>> +  Reading ptid %s (1 + 8 + 8 + 8 bytes), offset == %s\n",
>> +                               target_pid_to_str (record_list->u.ptid),
>> +                               paddress (get_current_arch (),
>> +                                         bfd_offset));
>> +          break;
>> +
>>         case record_end: /* end */
>>           rec = record_end_alloc ();
>>           record_insn_num ++;
>> @@ -2327,6 +2828,7 @@ cmd_record_save (char *args, int from_tt
>>   uint32_t magic;
>>   struct regcache *regcache;
>>   struct gdbarch *gdbarch;
>> +  struct address_space *aspace;
>>   struct cleanup *old_cleanups;
>>   struct cleanup *set_cleanups;
>>   bfd *obfd;
>> @@ -2363,6 +2865,7 @@ cmd_record_save (char *args, int from_tt
>>   /* Get the values of regcache and gdbarch.  */
>>   regcache = get_current_regcache ();
>>   gdbarch = get_regcache_arch (regcache);
>> +  aspace = get_regcache_aspace (regcache);
>>
>>   /* Disable the GDB operation record.  */
>>   set_cleanups = record_gdb_operation_disable_set ();
>> @@ -2374,7 +2877,7 @@ cmd_record_save (char *args, int from_tt
>>       if (record_list == &record_first)
>>         break;
>>
>> -      record_exec_insn (regcache, gdbarch, record_list);
>> +      record_exec_insn (&regcache, &gdbarch, &aspace, record_list);
>>
>>       if (record_list->prev)
>>         record_list = record_list->prev;
>> @@ -2395,6 +2898,9 @@ cmd_record_save (char *args, int from_tt
>>       case record_mem:
>>        save_size += 1 + 4 + 8 + record_list->u.mem.len;
>>        break;
>> +      case record_ptid:
>> +       save_size += 1 + 8 + 8 + 8;
>> +       break;
>>       }
>>
>>   /* Make the new bfd section.  */
>> @@ -2481,6 +2987,25 @@ cmd_record_save (char *args, int from_tt
>>                             record_list->u.mem.len, &bfd_offset);
>>               break;
>>
>> +            case record_ptid: /* ptid */
>> +             if (record_debug)
>> +               fprintf_unfiltered (gdb_stdlog, "\
>> +  Writing ptid %s (1 plus 8 plus 8 plus 8 bytes)\n",
>> +                                   target_pid_to_str
>> (record_list->u.ptid));
>> +
>> +             /* Write Process id.  */
>> +             addr = netorder64 ((uint64_t) 1);
>> +             bfdcore_write (obfd, osec, &addr, sizeof (addr),
>> &bfd_offset);
>> +
>> +             /* Write Lightweight process id.  */
>> +             addr = netorder64 ((uint64_t) record_list->u.ptid.lwp);
>> +             bfdcore_write (obfd, osec, &addr, sizeof (addr),
>> &bfd_offset);
>> +
>> +             /* Write Thread id.  */
>> +             addr = netorder64 ((uint64_t) record_list->u.ptid.tid);
>> +             bfdcore_write (obfd, osec, &addr, sizeof (addr),
>> &bfd_offset);
>> +              break;
>> +
>>               case record_end:
>>                if (record_debug)
>>                  fprintf_unfiltered (gdb_stdlog, "\
>> @@ -2501,7 +3026,7 @@ cmd_record_save (char *args, int from_tt
>>         }
>>
>>       /* Execute entry.  */
>> -      record_exec_insn (regcache, gdbarch, record_list);
>> +      record_exec_insn (&regcache, &gdbarch, &aspace, record_list);
>>
>>       if (record_list->next)
>>         record_list = record_list->next;
>> @@ -2516,7 +3041,7 @@ cmd_record_save (char *args, int from_tt
>>       if (record_list == cur_record_list)
>>         break;
>>
>> -      record_exec_insn (regcache, gdbarch, record_list);
>> +      record_exec_insn (&regcache, &gdbarch, &aspace, record_list);
>>
>>       if (record_list->prev)
>>         record_list = record_list->prev;
>> @@ -2542,6 +3067,7 @@ record_goto_insn (struct record_entry *e
>>   struct cleanup *set_cleanups = record_gdb_operation_disable_set ();
>>   struct regcache *regcache = get_current_regcache ();
>>   struct gdbarch *gdbarch = get_regcache_arch (regcache);
>> +  struct address_space *aspace = get_regcache_aspace (regcache);
>>
>>   /* Assume everything is valid: we will hit the entry,
>>      and we will not hit the end of the recording.  */
>> @@ -2551,7 +3077,7 @@ record_goto_insn (struct record_entry *e
>>
>>   do
>>     {
>> -      record_exec_insn (regcache, gdbarch, record_list);
>> +      record_exec_insn (&regcache, &gdbarch, &aspace, record_list);
>>       if (dir == EXEC_REVERSE)
>>        record_list = record_list->prev;
>>       else
>> --- a/record.h
>> +++ b/record.h
>> @@ -23,6 +23,7 @@
>>  #define RECORD_IS_USED (current_target.to_stratum == record_stratum)
>>
>>  extern int record_debug;
>> +extern int record_step;
>>
>>  extern int record_arch_list_add_reg (struct regcache *regcache, int num);
>>  extern int record_arch_list_add_mem (CORE_ADDR addr, int len);
>
>