"record save" failure

"record save" failure

[Cedric Roux]

Hi GNU gdb people,

I'm trying to analyse an unfriendly program and I gave a try
to the "record" thing gdb provides.

Except it fails.

Below is an excerpt of the log.
Am I doing something wrong? forgetting to type something?
Does the "record" thing works with multi-threaded programs?
I used the "set scheduler-locking on" command to run only the
desired thread.

As you see, I record more than 5 millions instructions between
breakpoint 3 and breakpoint 2. The "record save" fails and a
following "info record" crashes gdb. My computer is a 64b intel p6000
thing with debian testing (last updated in october of last year or
something). I compiled gdb 7.6.2 locally with gcc 4.8.1.

I hacked a bit gdb (but just a bit) to dump the "record list" at
the beginning of "record_full_save" and I get a different dump
compared to "record_full_save" (which does more than dump the
list, it replays the instructions, which I don't do). So I'm a
bit lost. (I compared with a simple multi-threaded program I quickly
wrote, and for which "record" works.)

Any hint or whatever (maybe there is a bug somewhere in gdb...) is
very welcome.

Regards and thanks for gdb which is a great program,
Cédric.

momo@sed:~$ dbg/bin/gdb -p 23553
GNU gdb (GDB) 7.6.2
[...]
This GDB was configured as "x86_64-unknown-linux-gnu".
[...]
Attaching to process 23553
[...]
[New LWP 23554]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[...]
0x00007ffff325824d in poll () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) hb *0x00007fffe8c63f22
Hardware assisted breakpoint 1 at 0x7fffe8c63f22
(gdb) cont
Continuing.
[New Thread 0x7fffe4372700 (LWP 23570)]
[New Thread 0x7fffe4b73700 (LWP 23571)]
[New Thread 0x7fffdeaff700 (LWP 23572)]
[New Thread 0x7fffca553700 (LWP 23594)]
[New Thread 0x7fffc87ff700 (LWP 23595)]
[New Thread 0x7fffc7ffe700 (LWP 23596)]
[New Thread 0x7fffc77fd700 (LWP 23597)]
[New Thread 0x7fffc6ce9700 (LWP 23598)]
[New Thread 0x7fffc64e8700 (LWP 23599)]
[Thread 0x7fffc77fd700 (LWP 23597) exited]
[Thread 0x7fffc7ffe700 (LWP 23596) exited]
[Thread 0x7fffc87ff700 (LWP 23595) exited]
[Thread 0x7fffc6ce9700 (LWP 23598) exited]
[Thread 0x7fffc64e8700 (LWP 23599) exited]
[Thread 0x7fffca553700 (LWP 23594) exited]
[New Thread 0x7fffca553700 (LWP 23600)]
[New Thread 0x7fffc64e8700 (LWP 23601)]
[Thread 0x7fffc64e8700 (LWP 23601) exited]
[Switching to Thread 0x7fffca553700 (LWP 23600)]

Breakpoint 1, 0x00007fffe8c63f22 in ?? () from XXX.so
(gdb) cont
Continuing.

Breakpoint 1, 0x00007fffe8c63f22 in ?? () from XXX.so
(gdb) hb *0x00007fffe8ad361c
Hardware assisted breakpoint 2 at 0x7fffe8ad361c
(gdb) hb *0x00007fffe8c0f066
Hardware assisted breakpoint 3 at 0x7fffe8c0f066
(gdb) cont
Continuing.

Breakpoint 3, 0x00007fffe8c0f066 in ?? () from XXX.so
(gdb) set record full insn-number-max 10000000
(gdb) record full
(gdb) set scheduler-locking on
(gdb) cont
Continuing.

Breakpoint 2, 0x00007fffe8ad361c in ?? () from XXX.so
(gdb) info record
Active record target: record-full
Record mode:
Lowest recorded instruction number is 1.
Highest recorded instruction number is 5681093.
Log contains 5681093 instructions.
Max logged instructions is 10000000.
(gdb) record save TOTOTATA
warning: target file /proc/23553/cmdline contained unexpected null 
characters
Couldn't get registers: No such process.
(gdb) info record
Active record target: record-full
Segmentation fault

Re: "record save" failure

[Hui Zhu]

Prec didn't support multithread.

Thanks,
Hui

On Tue, Jan 21, 2014 at 9:31 PM, Cedric Roux <sed@free.fr> wrote:
> Hi GNU gdb people,
>
> I'm trying to analyse an unfriendly program and I gave a try
> to the "record" thing gdb provides.
>
> Except it fails.
>
> Below is an excerpt of the log.
> Am I doing something wrong? forgetting to type something?
> Does the "record" thing works with multi-threaded programs?
> I used the "set scheduler-locking on" command to run only the
> desired thread.
>
> As you see, I record more than 5 millions instructions between
> breakpoint 3 and breakpoint 2. The "record save" fails and a
> following "info record" crashes gdb. My computer is a 64b intel p6000
> thing with debian testing (last updated in october of last year or
> something). I compiled gdb 7.6.2 locally with gcc 4.8.1.
>
> I hacked a bit gdb (but just a bit) to dump the "record list" at
> the beginning of "record_full_save" and I get a different dump
> compared to "record_full_save" (which does more than dump the
> list, it replays the instructions, which I don't do). So I'm a
> bit lost. (I compared with a simple multi-threaded program I quickly
> wrote, and for which "record" works.)
>
> Any hint or whatever (maybe there is a bug somewhere in gdb...) is
> very welcome.
>
> Regards and thanks for gdb which is a great program,
> Cédric.
>
> momo@sed:~$ dbg/bin/gdb -p 23553
> GNU gdb (GDB) 7.6.2
> [...]
> This GDB was configured as "x86_64-unknown-linux-gnu".
> [...]
> Attaching to process 23553
> [...]
> [New LWP 23554]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> [...]
> 0x00007ffff325824d in poll () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) hb *0x00007fffe8c63f22
> Hardware assisted breakpoint 1 at 0x7fffe8c63f22
> (gdb) cont
> Continuing.
> [New Thread 0x7fffe4372700 (LWP 23570)]
> [New Thread 0x7fffe4b73700 (LWP 23571)]
> [New Thread 0x7fffdeaff700 (LWP 23572)]
> [New Thread 0x7fffca553700 (LWP 23594)]
> [New Thread 0x7fffc87ff700 (LWP 23595)]
> [New Thread 0x7fffc7ffe700 (LWP 23596)]
> [New Thread 0x7fffc77fd700 (LWP 23597)]
> [New Thread 0x7fffc6ce9700 (LWP 23598)]
> [New Thread 0x7fffc64e8700 (LWP 23599)]
> [Thread 0x7fffc77fd700 (LWP 23597) exited]
> [Thread 0x7fffc7ffe700 (LWP 23596) exited]
> [Thread 0x7fffc87ff700 (LWP 23595) exited]
> [Thread 0x7fffc6ce9700 (LWP 23598) exited]
> [Thread 0x7fffc64e8700 (LWP 23599) exited]
> [Thread 0x7fffca553700 (LWP 23594) exited]
> [New Thread 0x7fffca553700 (LWP 23600)]
> [New Thread 0x7fffc64e8700 (LWP 23601)]
> [Thread 0x7fffc64e8700 (LWP 23601) exited]
> [Switching to Thread 0x7fffca553700 (LWP 23600)]
>
> Breakpoint 1, 0x00007fffe8c63f22 in ?? () from XXX.so
> (gdb) cont
> Continuing.
>
> Breakpoint 1, 0x00007fffe8c63f22 in ?? () from XXX.so
> (gdb) hb *0x00007fffe8ad361c
> Hardware assisted breakpoint 2 at 0x7fffe8ad361c
> (gdb) hb *0x00007fffe8c0f066
> Hardware assisted breakpoint 3 at 0x7fffe8c0f066
> (gdb) cont
> Continuing.
>
> Breakpoint 3, 0x00007fffe8c0f066 in ?? () from XXX.so
> (gdb) set record full insn-number-max 10000000
> (gdb) record full
> (gdb) set scheduler-locking on
> (gdb) cont
> Continuing.
>
> Breakpoint 2, 0x00007fffe8ad361c in ?? () from XXX.so
> (gdb) info record
> Active record target: record-full
> Record mode:
> Lowest recorded instruction number is 1.
> Highest recorded instruction number is 5681093.
> Log contains 5681093 instructions.
> Max logged instructions is 10000000.
> (gdb) record save TOTOTATA
> warning: target file /proc/23553/cmdline contained unexpected null
> characters
> Couldn't get registers: No such process.
> (gdb) info record
> Active record target: record-full
> Segmentation fault

Re: "record save" failure

[Sergio Durigan Junior]

On Tuesday, January 21 2014, Hui Zhu wrote:

> Prec didn't support multithread.

Even so, GDB shouldn't segfault like that.

Cedric, thanks for the report.  Would it be possible for you to test
this with our git HEAD?  Also, if you still find the issue with git
HEAD, could you please file a bug in our bugzilla about this issue?
Thanks.

-- 
Sergio

Re: "record save" failure

[Cedric Roux]

[-- Attachment #1: Type: text/plain, Size: 1833 bytes --]

On 01/22/2014 04:11 AM, Sergio Durigan Junior wrote:
> On Tuesday, January 21 2014, Hui Zhu wrote:
>
>> Prec didn't support multithread.
>
> Even so, GDB shouldn't segfault like that.
>
> Cedric, thanks for the report.  Would it be possible for you to test
> this with our git HEAD?  Also, if you still find the issue with git
> HEAD, could you please file a bug in our bugzilla about this issue?
> Thanks.

fails too with GNU gdb (GDB) 7.7.50.20140121-cvs (git cloned yesterday)
Attached is a little C program that exhibits the problem.
You need two terminals: Tprog and Tgdb.
In Tprog: compile and run recthr.
When it stops, in Tgdb, launch gdb.
Then press enter in Tprog, and, well, follow what the program
asks you to do.

What it does is: create a thread th1, wait for gdb, th1 quits
(so that gdb sees th1 leaving, I think that's the triggering
event) and th2 is created, then gdb records a function run by that
thread and when it's done the user asks for "record save"
which fails. Then the user types "info record" and everyone
goes back to nowhere. (Maybe th2 is not necessary here.)

I think you set a global variable to the TID of th1. But th1 has gone
away when I record th2, so asking the kernel for th1's registers' values
fails. Or something like that.

I still have a 64b computer here, with debian. But I bet it would fail
with something else. (Maybe you have to run the linux kernel, not
even sure about that...)

I let you deal with your bug tracking stuff, I'm not good with those. :)

On my side I will try to change the TID for the one I record but since
you don't support multi-threading recording I won't bother you with my
little hacks, which, I fear, will be terribly dirty. (Note though
that the recording is done with "scheduler-locking on", which means I
record a single thread.)

Happy hacking!
Cédric.

[-- Attachment #2: recthr.c --]
[-- Type: text/x-csrc, Size: 1104 bytes --]

/* compile with: gcc -pthread -Wall recthr.c -o recthr */
#include <stdio.h>
#include <pthread.h>
#include <unistd.h>

volatile int th1_done;

void *th1(void *_)
{
  printf("run\ngdb -p %d\nthen in gdb type:\ncont\nand press enter in this term\n", getpid());
  getchar();
  th1_done = 1;
  return 0;
}

int traceme(void)
{
  int x=0;
  int i=0;
  for (x = 0; x < 10; x++) i += x*2;
  return i;
}

void after_traceme(int x)
{
  printf("%d\n", x);
}

void *th2(void *x)
{
  int i = traceme();
  after_traceme(i);
  return 0;
}

void new_thread(void *(*f)(void *))
{
  pthread_t th;
  pthread_create(&th, 0, f, 0);
}

int main(void)
{
  new_thread(th1);
  while (!th1_done) usleep(10*1000);
  printf("type:\nctrl+c\nhb *%p\nhb *%p\ncont\nin gdb and press enter in this term\n",
         traceme, after_traceme);
  getchar();
  printf("type:\nset scheduler-locking on\nrecord full\ncont\nin gdb\n");
  printf("then:\nrecord save /tmp/XX\nit should fail\n");
  printf("then type:\ninfo record\nit should segfault\n");
  printf("and happy debugging to you too.:)\n");
  new_thread(th2);
  while (1) pause();
}