From mboxrd@z Thu Jan  1 00:00:00 1970
From: "Glenn F. Maynard" <g_gdb@zewt.org>
To: gdb@sourceware.cygnus.com
Subject: Re: gdb: detection and/or fork+gethostbyname crash workaround?
Date: Tue, 24 Jul 2001 13:57:00 -0000
Message-id: <20010724165719.A3214@zewt.org>
References: <20010723210115.B1359@zewt.org> <1010724155219.ZM20181@ocotillo.lan> <20010724142248.A2923@zewt.org> <1010724202607.ZM20680@ocotillo.lan>
X-SW-Source: 2001-07/msg00350.html

To the other reply: well, I can already manually disable DNS
forking; I just don't want to have to do that.

On Tue, Jul 24, 2001 at 01:26:07PM -0700, Kevin Buettner wrote:
> > On Tue, Jul 24, 2001 at 08:52:19AM -0700, Kevin Buettner wrote:
> > > If it's one which uses ptrace(), the kernel usually prohibits two
> > > processes from invoking ptrace() on the same inferior.  So one
> > > strategy might be to have the program in question cause ptrace to be
> > > invoked on itself.  I don't think the process will be able to do this
> > > itself; I think it's likely that it would have to fork and let the
> > > child attempt this.  The return status from wait() or waitpid() could
> > > indicate whether the attempt to invoke ptrace() was successful or not.
> > 
> > That's what I tried (x86 Linux); a process can't ptrace itself.  I could
> > fork a process to test this, I suppose; I'll try that.  (Wouldn't it
> > be the return status from ptrace(), though?  I assume it'd return something
> > like EPERM if a process is already being debugged.)
> 
> Right; you'd pass back the results of the attempted ptrace() (and it
> looks to me like EPERM is the right thing to test for) through the
> child's exit status.  That's why I said you'd use wait() or waitpid()
> to determine it.

Got it.  In case anyone else wants it (or can point out an error, of course):

#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <assert.h>

#include <sys/ptrace.h>
#include <signal.h>

#include <sys/types.h>
#include <sys/wait.h>

bool in_debugger()
{
	int cpid = fork();
	assert(cpid != -1);

	if(cpid == 0) {
		signal(SIGCHLD, SIG_IGN);
		if(ptrace(PTRACE_ATTACH, getppid(), NULL, NULL) == -1) {
			/* Attach failed; it's probably already being debugged. */
			if(errno != EPERM)
				fprintf(stderr, "c ptrace: unexpected error: %s\n", strerror(errno));
			exit(1);
		}
		
		/* We just SIGSTOPped it; we need to CONT it now. */
		int ret;
		waitpid(getppid(), &ret, 0);

		if(ptrace(PTRACE_DETACH, getppid(), NULL, NULL) == -1)
			fprintf(stderr, "detach failed: %s\n", strerror(errno));

		kill(getppid(), SIGCONT);

		exit(0);
	}
	int ret;
	waitpid(cpid, &ret, 0);
	assert(!WIFSIGNALED(ret));

	return WEXITSTATUS(ret) != 0;

}

main()
{
	printf("%i\n", in_debugger());
}


>     1) Set up a signal handler in the child for SIGTRAP.  Have the
>        signal handler replace the trap instruction with a nop prior
>        to returning.

I tried that, with no luck.  I might have botched something silly, of course.

>     2) In the parent, call dlopen() to explicitly load the shared
>        libraries needed by gethostbyname().  That way, when
>        gethostbyname() is called, there won't be any unloaded
>        functions.

That'd mean searching out NSS libraries, and loading *all* of them ... and
there's no real guarantee that they're even in /lib.

-- 
Glenn Maynard