Problems debugging forked processes

Problems debugging forked processes

[Frank Schaefer]

[-- Attachment #1: Type: text/plain, Size: 1869 bytes --]

Hi there,

I've written a network daemon and client. The daemon forks to serve a
client request.
Everything runs fine outside of the debugger, but I want to trace the
code at least once, before I release it.

Here goes my problem:
# gdb daemon
GNU gdb 5.0
Copyright 2000 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i686-pc-linux-gnu"...
(gdb) b 13
Breakpoint 1 at 0x8048899: file daemon.c, line 13.
(gdb) show follow-fork-mode
Debugger response to a program call of fork or vfork is "parent".
(gdb) set follow-fork-mode child
(gdb) show follow-fork-mode
Debugger response to a program call of fork or vfork is "child".
(gdb) run
Starting program: /v_dsk/home/p10209/Tests/daemon 

Breakpoint 1, main () at syncboot.c:15
15      int                On = 1;
(gdb) n
...
(gdb) n
67   ChildPid = fork();
(gdb) n
69      sleep( 10 );
(gdb) print ChildPid
$1 = 3700

So it seems, I'm tracing the parent process - don't it? The same
happens, If I do:
	(gdb) set follow-fork-mode ask
instead of child. I'm not asked anything and remain debugging the
parent.

If I use the attach method from a second instance of gdb ( there's the
sleep() above for ), the child process exits on exceptions ( mostly
segmentation fault and sometimes invalid operation ) on different
(random???) lines in the code. This happens everytime in the fixup()
from ld.linux.so.

I'm using gdb 5.0, my compiler is gcc 2.95.3 on a glibc-2.2.4
linux-2.4.16 system. I have Binutils 2.11 installed.

Furtheron I've attached reduced versions of the code, for which the
fault still occurs.

Any suggestions?

Regards
Frank

[-- Attachment #2: daemon.c --]
[-- Type: text/x-c, Size: 2822 bytes --]

#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <netinet/in.h>
#include <netdb.h>

#define PORT 1023

const char MESSAGE[] = "Hi there!\n";

int main( void ) {
int                SrvSocket, ChildPid;
int                On = 1;
struct linger      Linger = { 0 };
char               Hostname[80];
struct hostent     *pHostEnt = NULL;
struct sockaddr_in ServerName = { 0 };

   SrvSocket = socket( PF_INET, SOCK_STREAM, IPPROTO_TCP );
   if( SrvSocket == -1 ) {
      perror( "Can't create socket!\n" );
      exit( 1 );
   }
   if( setsockopt( SrvSocket, SOL_SOCKET, SO_REUSEADDR, (const char*)&On, sizeof( On ) ) ) {
      perror( "Can't set reuse address on socket!\n" );
      exit( 1 );
   }
   Linger.l_onoff = 1;
   Linger.l_linger = 20;
   if( setsockopt( SrvSocket, SOL_SOCKET, SO_LINGER, (const char*)&Linger, sizeof( Linger ) ) ) {
      perror( "Can't set lingering for socket!\n" );
      exit( 1 );
   }
   if( gethostname( Hostname, sizeof( Hostname ) ) ) {
      perror( " Can't get my own hostname!\n" );
      exit( 1 );
   }
   pHostEnt = gethostbyname( Hostname );
   if( ! pHostEnt ) {
      perror( "Can't get hostentry!\n" );
      exit( 1 );
   }
   memset( &ServerName, 0, sizeof( ServerName ) );
   memcpy( &ServerName.sin_addr, pHostEnt->h_addr, pHostEnt->h_length );
   ServerName.sin_family = AF_INET;
   ServerName.sin_port = htons( PORT );
   if( bind( SrvSocket, (struct sockaddr*)&ServerName, sizeof( ServerName ) ) ) {
      perror( "Can't bind socket!\n" );
      exit( 1 );
   }
   if( listen(SrvSocket, BACKLOG ) ) {
      perror( "Can't listen on socket!\n" );
      exit( 1 );
   }
   while( 1 ) {
      struct sockaddr_in ClientName;
      int SlaveSocket, ClientLength = sizeof( ClientName );

      memset( &ClientName, 0, sizeof( ClientName ) );
      SlaveSocket = accept( SrvSocket, (struct sockaddr*)&ClientName, &ClientLength );
      if( SlaveSocket == -1 ) {
         perror( "Can't accept connection!\n" );
         exit( 1 );
      }
      ChildPid = fork();

sleep( 10 );

      switch( ChildPid ) {
         case -1:
            perror( "Can't fork!\n" );
            exit( 1 );
         case 0:
            close( SrvSocket );
            if( getpeername( SlaveSocket, (struct sockaddr*)&ClientName, &ClientLength )) {
               perror( "Can't get peername!\n" );
            } else {
               printf( "Connection request from %s\n", inet_ntoa( ClientName.sin_addr) );
            }
            write( SlaveSocket, MESSAGE, strlen( MESSAGE ) );
            close( SlaveSocket );
            exit( 0 );
         default:
            close( SlaveSocket );
      }
   }
   exit( 0 );
}

[-- Attachment #3: client.c --]
[-- Type: text/x-c, Size: 1452 bytes --]

#include "syncboot.h"
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <string.h>
#include <libgen.h>

int main( int argc, char* argv[] ) {
int			CltSocket, Status = 0;
char			*RemHost = NULL;
char			*ProgName = basename( argv[0] );
char			Buff[256] = "";
struct hostent		*pHostEnt;
struct sockaddr_in	SrvName;

   RemHost = argv[1];
   CltSocket = socket( PF_INET, SOCK_STREAM, IPPROTO_TCP );
   if( CltSocket == -1 ) {
      perror( "Can't create socket!\n" );
      exit( 1 );
   }
   pHostEnt = gethostbyname( RemHost );
   if( ! pHostEnt ) {
      pHostEnt = gethostbyaddr( RemHost, strlen( RemHost ), AF_INET );
   }
   if( ! pHostEnt ) {
      fprintf( stderr,"%s: Couldn't resolve remote host %s", ProgName, RemHost );
      exit( 1 );
   }
   SrvName.sin_family = AF_INET;
   SrvName.sin_port = htons( PORT );
   memcpy( &SrvName.sin_addr, pHostEnt->h_addr, pHostEnt->h_length );
   Status = connect( CltSocket, (struct sockaddr*)&SrvName, sizeof( SrvName ) );
   if( Status == -1 ) {
      perror( "Can't connect!\n" );
      exit( 1 );
   }
   while( ( Status = read( CltSocket, Buff, sizeof( Buff) -1 ) ) > 0 ) {
      printf( "%d : %s\n", Status, Buff );
   }
   if( Status == -1 ){
      perror( "Read error!\n" );
   }
   close( CltSocket );
   exit( 0 );
}

Re: Problems debugging forked processes

[Daniel Jacobowitz]

On Fri, Apr 19, 2002 at 09:45:49AM +0200, Frank Schaefer wrote:
> Hi there,
> 
> I've written a network daemon and client. The daemon forks to serve a
> client request.
> Everything runs fine outside of the debugger, but I want to trace the
> code at least once, before I release it.
> 
> Here goes my problem:
> # gdb daemon
> GNU gdb 5.0
> Copyright 2000 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> This GDB was configured as "i686-pc-linux-gnu"...
> (gdb) b 13
> Breakpoint 1 at 0x8048899: file daemon.c, line 13.
> (gdb) show follow-fork-mode
> Debugger response to a program call of fork or vfork is "parent".
> (gdb) set follow-fork-mode child
> (gdb) show follow-fork-mode
> Debugger response to a program call of fork or vfork is "child".

This variable doesn't actually affect GNU/Linux.  Yeah, I know, I know. 
We need a documentation patch to address this; please file a PR
(http://sources.redhat.com/gdb/, Bugs).

> If I use the attach method from a second instance of gdb ( there's the
> sleep() above for ), the child process exits on exceptions ( mostly
> segmentation fault and sometimes invalid operation ) on different
> (random???) lines in the code. This happens everytime in the fixup()
> from ld.linux.so.

Because the breakpoints which were inserted when the application forked
turn into traps.  There's no clear solution to this without actually
implementing fork-following - which should not be hard, but no one has
stepped up to do it.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: Problems debugging forked processes

[Eli Zaretskii]

On Fri, 19 Apr 2002, Daniel Jacobowitz wrote:

> > (gdb) show follow-fork-mode
> > Debugger response to a program call of fork or vfork is "parent".
> > (gdb) set follow-fork-mode child
> > (gdb) show follow-fork-mode
> > Debugger response to a program call of fork or vfork is "child".
> 
> This variable doesn't actually affect GNU/Linux.  Yeah, I know, I know. 
> We need a documentation patch to address this; please file a PR
> (http://sources.redhat.com/gdb/, Bugs).

If someone tells me what should be documented, I'll do that.  Now is the 
time, since v5.2 is in pretest.

Is the issue that follow-fork-mode is only supported on a small number of 
systems?  If so, what are those systems, and is it enough to have their 
list in the manual?

Re: Problems debugging forked processes

[Andrew Cagney]

> On Fri, 19 Apr 2002, Daniel Jacobowitz wrote:
> 
> 
>> > (gdb) show follow-fork-mode
>> > Debugger response to a program call of fork or vfork is "parent".
>> > (gdb) set follow-fork-mode child
>> > (gdb) show follow-fork-mode
>> > Debugger response to a program call of fork or vfork is "child".
> 
>> 
>> This variable doesn't actually affect GNU/Linux.  Yeah, I know, I know. 
>> We need a documentation patch to address this; please file a PR
>> (http://sources.redhat.com/gdb/, Bugs).
> 
> 
> If someone tells me what should be documented, I'll do that.  Now is the 
> time, since v5.2 is in pretest.
> 
> Is the issue that follow-fork-mode is only supported on a small number of 
> systems?  If so, what are those systems, and is it enough to have their 
> list in the manual?

It is only supported on HP/UX (So yes).

Andrew

Re: Problems debugging forked processes

[Daniel Jacobowitz]

On Sun, Apr 21, 2002 at 09:12:33AM +0300, Eli Zaretskii wrote:
> 
> On Fri, 19 Apr 2002, Daniel Jacobowitz wrote:
> 
> > > (gdb) show follow-fork-mode
> > > Debugger response to a program call of fork or vfork is "parent".
> > > (gdb) set follow-fork-mode child
> > > (gdb) show follow-fork-mode
> > > Debugger response to a program call of fork or vfork is "child".
> > 
> > This variable doesn't actually affect GNU/Linux.  Yeah, I know, I know. 
> > We need a documentation patch to address this; please file a PR
> > (http://sources.redhat.com/gdb/, Bugs).
> 
> If someone tells me what should be documented, I'll do that.  Now is the 
> time, since v5.2 is in pretest.
> 
> Is the issue that follow-fork-mode is only supported on a small number of 
> systems?  If so, what are those systems, and is it enough to have their 
> list in the manual?

I believe the only platform to implement this is actually HP/UX:

drow@nevyn:/opt/src/gdb/src/gdb% grep -r CHILD_HAS_FORKED .
./inftarg.c:#if !defined(CHILD_HAS_FORKED)
./config/pa/nm-hppah.h:#define CHILD_HAS_FORKED
./infttrace.c:#if defined(CHILD_HAS_FORKED)
./ChangeLog-1998:       (child_has_forked): Now enclosed by a CHILD_HAS_FORKED ifdef


There's a nice new debugging interface in the works for GNU/Linux that
will let us support this some day.  But for now, HP/UX only.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: Problems debugging forked processes

[Frank Schaefer]

On Fri, 2002-04-19 at 18:10, Daniel Jacobowitz wrote:
> On Fri, Apr 19, 2002 at 09:45:49AM +0200, Frank Schaefer wrote:
> > Hi there,
> > 
> > I've written a network daemon and client. The daemon forks to serve a
> > client request.
> > Everything runs fine outside of the debugger, but I want to trace the
> > code at least once, before I release it.
> > 
> > Here goes my problem:
> > # gdb daemon
> > GNU gdb 5.0
> > Copyright 2000 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and you
> > are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show warranty" for
> > details.
> > This GDB was configured as "i686-pc-linux-gnu"...
> > (gdb) b 13
> > Breakpoint 1 at 0x8048899: file daemon.c, line 13.
> > (gdb) show follow-fork-mode
> > Debugger response to a program call of fork or vfork is "parent".
> > (gdb) set follow-fork-mode child
> > (gdb) show follow-fork-mode
> > Debugger response to a program call of fork or vfork is "child".
> 
> This variable doesn't actually affect GNU/Linux.  Yeah, I know, I know. 
> We need a documentation patch to address this; please file a PR
> (http://sources.redhat.com/gdb/, Bugs).
> 
> > If I use the attach method from a second instance of gdb ( there's the
> > sleep() above for ), the child process exits on exceptions ( mostly
> > segmentation fault and sometimes invalid operation ) on different
> > (random???) lines in the code. This happens everytime in the fixup()
> > from ld.linux.so.
> 
> Because the breakpoints which were inserted when the application forked
> turn into traps.  There's no clear solution to this without actually
> implementing fork-following - which should not be hard, but no one has
> stepped up to do it.

O.K.,

I had a look at sources.redhat.com.
Maybe I'll have a look at the GDB-code too. Just now I don't have the
time. The fork related stuff should be in inftarg.c, fork-child.c and
infptrace.c for GNU/Linux right?

For now I use this workaround.

RetVal = fork();
if( RetVal ) printf( "%d\n", RetVal ); else sleep( 10 );
... further code

I start the daemon outside of gdb, and attach the child to a gdb session
after the PID is printed, having the first breakpoint somewhere in the
"further code". That's not very nice but at least it works for me.

Regards and thanks for the hints
Frank