Run multiple parallel instances of gdb

Run multiple parallel instances of gdb

[LE GARREC Vincent]

Hi everybody,

I would like to sort crashes found by fuzzing. So I have around 1000 files
that make my application crashes. I made a small program to run gdb and to
extract backtraces to file. To increase speed, I run parallel instances.

Problem, with parallel instances, my program stopped. It doesn't crashes,
it stops. I have to run "fg" from terminal to continue and it's happening
very often. So actually, I'm running with single thread.

Is it normal ? Did I do something wrong ? If you need more information, I
can give you.

Please find after simple steps to reproduce the case,

Thanks for you advices,

Vincent Le Garrec

[1] : crash program
main.c (in /tmp folder)

int main()
{
  int *t = 0xDEADBEEF;
  *t = 1;
}

Run it and it should crash.

[2] : multiple execution of gdb
loopgdb.cpp

#include <thread>
#include <future>
#include <vector>
#include <functional>
#include <unistd.h>
#include <iostream>
#include <sys/types.h>
#include <sys/wait.h>

void run_gdb()
{
  pid_t child_pid = fork();
  std::cout << "run" << std::endl;
  if (child_pid != 0)
  {
    pid_t wait_pid;
    {
      wait_pid = waitpid(child_pid, nullptr, WNOHANG);
      sleep(1);
    }
    while(wait_pid == 0);
  }
  else
  {
    execlp("/usr/bin/gdb", "-batch-silent", "-ex", "run", "-ex", "set
logging overwrite on", "-ex", "set logging on", "-ex", "set pagination
off", "-ex", "handle SIG33 pass nostop noprint", "-ex", "backtrace full",
"-ex", "set logging off", "-ex", "quit", "--args", "/tmp/main", nullptr);
  }
}

int main()
{
  int nthreads = std::thread::hardware_concurrency();
  std::vector<std::future<void>> threads(nthreads);
  for (size_t t = 0; t < nthreads; t++)
  {
    threads[t] = std::async(std::launch::async, std::bind(
        []() {
          while(true)
          {
            run_gdb();
          }
        }));
  }
  for (std::future<void> & t : threads)
  {
    t.get();
  }

}

Run it (g++ loopgdb.cpp -o loopgdb -lpthread) and you will see that the
program in command-line will stopped.

[3] gdb --version
GNU gdb (Gentoo 8.1 p1) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".

Re: Run multiple parallel instances of gdb

[Simon Marchi]

On 2018-03-11 16:15, LE GARREC Vincent wrote:
> Hi everybody,
> 
> I would like to sort crashes found by fuzzing. So I have around 1000 
> files
> that make my application crashes. I made a small program to run gdb and 
> to
> extract backtraces to file. To increase speed, I run parallel 
> instances.
> 
> Problem, with parallel instances, my program stopped. It doesn't 
> crashes,
> it stops. I have to run "fg" from terminal to continue and it's 
> happening
> very often. So actually, I'm running with single thread.
> 
> Is it normal ? Did I do something wrong ? If you need more information, 
> I
> can give you.
> 
> Please find after simple steps to reproduce the case,
> 
> Thanks for you advices,
> 
> Vincent Le Garrec

Hi Vincent,

Do you get a "suspended (tty output)" message?  If so, what you have is 
a background process trying to output on the terminal while the terminal 
has the "tostop" flag set.  A simpler case to reproduce it is:

$ gdb -batch -ex run --args /bin/echo salut &
[1] 28223
$
[1]  + 28223 suspended (tty output)  gdb -q -batch -ex run --args 
/bin/echo salut

I don't understand why though, because my terminal does not have 
"tostop" enabled:

$ stty
speed 38400 baud; line = 0;
-brkint -imaxbel iutf8

So something in the process probably sets that flag...  Anyway, one way 
to get around it is to change the terminal for the newly created 
inferiors, for example to /dev/null if you don't need them to do I/O on 
the terminal:

$ gdb -batch -ex "tty /dev/null" -ex run --args /bin/echo salut &
[1] 28276
$ [Inferior 1 (process 28285) exited normally]

[1]  + 28276 done       gdb -q -batch -ex "tty /dev/null" -ex run --args 
/bin/echo salut

One comment below:

> [1] : crash program
> main.c (in /tmp folder)
> 
> int main()
> {
>   int *t = 0xDEADBEEF;
>   *t = 1;
> }
> 
> Run it and it should crash.
> 
> [2] : multiple execution of gdb
> loopgdb.cpp
> 
> #include <thread>
> #include <future>
> #include <vector>
> #include <functional>
> #include <unistd.h>
> #include <iostream>
> #include <sys/types.h>
> #include <sys/wait.h>
> 
> void run_gdb()
> {
>   pid_t child_pid = fork();
>   std::cout << "run" << std::endl;
>   if (child_pid != 0)
>   {
>     pid_t wait_pid;
>     {
>       wait_pid = waitpid(child_pid, nullptr, WNOHANG);
>       sleep(1);
>     }
>     while(wait_pid == 0);

You are missing the "do" from your do/while, which results in an 
infinite loop here (if wait_pid is 0).

Simon

Re: Run multiple parallel instances of gdb

[LE GARREC Vincent]

I don't have a suspended message, it a stopped. But the behavior is the
same.

[1]+  Stopped                 ./loopgdb

My stty:
speed 38400 baud; line = 0;
erase = ^H;
-brkint -imaxbel iutf8

but your diagnostic is right. Adding the "-ex tty /dev/null" is perfect.
I will do some search about this "tostop" flag.

For the minimal testcase, I did it a bit fast but the "do" didn't missing
in my code.

Thanks for your advice, you really help me,
Best regards,
Vincent

2018-03-11 21:54 GMT+01:00 Simon Marchi <simon.marchi@polymtl.ca>:

> On 2018-03-11 16:15, LE GARREC Vincent wrote:
>
>> Hi everybody,
>>
>> I would like to sort crashes found by fuzzing. So I have around 1000 files
>> that make my application crashes. I made a small program to run gdb and to
>> extract backtraces to file. To increase speed, I run parallel instances.
>>
>> Problem, with parallel instances, my program stopped. It doesn't crashes,
>> it stops. I have to run "fg" from terminal to continue and it's happening
>> very often. So actually, I'm running with single thread.
>>
>> Is it normal ? Did I do something wrong ? If you need more information, I
>> can give you.
>>
>> Please find after simple steps to reproduce the case,
>>
>> Thanks for you advices,
>>
>> Vincent Le Garrec
>>
>
> Hi Vincent,
>
> Do you get a "suspended (tty output)" message?  If so, what you have is a
> background process trying to output on the terminal while the terminal has
> the "tostop" flag set.  A simpler case to reproduce it is:
>
> $ gdb -batch -ex run --args /bin/echo salut &
> [1] 28223
> $
> [1]  + 28223 suspended (tty output)  gdb -q -batch -ex run --args
> /bin/echo salut
>
> I don't understand why though, because my terminal does not have "tostop"
> enabled:
>
> $ stty
> speed 38400 baud; line = 0;
> -brkint -imaxbel iutf8
>
> So something in the process probably sets that flag...  Anyway, one way to
> get around it is to change the terminal for the newly created inferiors,
> for example to /dev/null if you don't need them to do I/O on the terminal:
>
> $ gdb -batch -ex "tty /dev/null" -ex run --args /bin/echo salut &
> [1] 28276
> $ [Inferior 1 (process 28285) exited normally]
>
> [1]  + 28276 done       gdb -q -batch -ex "tty /dev/null" -ex run --args
> /bin/echo salut
>
> One comment below:
>
> [1] : crash program
>> main.c (in /tmp folder)
>>
>> int main()
>> {
>>   int *t = 0xDEADBEEF;
>>   *t = 1;
>> }
>>
>> Run it and it should crash.
>>
>> [2] : multiple execution of gdb
>> loopgdb.cpp
>>
>> #include <thread>
>> #include <future>
>> #include <vector>
>> #include <functional>
>> #include <unistd.h>
>> #include <iostream>
>> #include <sys/types.h>
>> #include <sys/wait.h>
>>
>> void run_gdb()
>> {
>>   pid_t child_pid = fork();
>>   std::cout << "run" << std::endl;
>>   if (child_pid != 0)
>>   {
>>     pid_t wait_pid;
>>     {
>>       wait_pid = waitpid(child_pid, nullptr, WNOHANG);
>>       sleep(1);
>>     }
>>     while(wait_pid == 0);
>>
>
> You are missing the "do" from your do/while, which results in an infinite
> loop here (if wait_pid is 0).
>
> Simon
>

Re: Run multiple parallel instances of gdb

[Andreas Schwab]

On Mär 11 2018, Simon Marchi <simon.marchi@polymtl.ca> wrote:

> Do you get a "suspended (tty output)" message?  If so, what you have is a
> background process trying to output on the terminal while the terminal has
> the "tostop" flag set.  A simpler case to reproduce it is:
>
> $ gdb -batch -ex run --args /bin/echo salut &
> [1] 28223
> $
> [1]  + 28223 suspended (tty output)  gdb -q -batch -ex run --args
> /bin/echo salut
>
> I don't understand why though, because my terminal does not have "tostop"
> enabled:

tcsetattr ignores the tostop setting.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."