Re: Multiple threads in one corefile on Linux--gdb support?

Re: Multiple threads in one corefile on Linux--gdb support?

[Eric Paire]

> I've done a kernel patch for Linux which dumps multiple threads into different
> corefiles named core.##### where ##### is the PID.  I've been told that the
> ELF format'll support multiple register in a core file by having >1 PT_NOTE
> section.
> 
> If I patch the kernel to dump the multiple register sets into a single core,
> will GDB with LinuxThreads support recognize the corefile without
> modification?
> 
Two years ago, when I first developed the gdb extension for debugging
LinuxThreads applications, Philip Gladstone <philip@raptor.com> sent to me
patches for gdb-4.16 and Linux kernel in order to generate multithreaded
core dumps within the regular ELF format (exactly as you suggest).

Linus Torvalds has never included these patches into the kernel since they
require that the kernel kills all clones of the thread doing the core dump,
probably because it forces a given semantic to the cloned processes inside
the kernel.

Last year, I have suggested another philosophy for LinuxThread core dumps
which does not require any kernel modification, and which I have never had
enough spare time to implement. Everything is done in gdb and in the
LinuxThread library. Here is a rough description of it:

1) attach a signal handler to all signals (except SIGKILL and SIGSTOP
	obviously) which will be called with the sigcontext,
2) on signal (generating a core dump) reception, signal all other threads,
3) on each thread receiving the signal, store the context in a reserved
	thread-private part (usually into their own stack),
4) wait for the end of all threads but one (the manager one ?)
5) when only one thread remain, reset signal handler to default and resend
	original signal to itself, which may or may not generate a core dump,
	depending on the kernel implementation.

The kernel generates then a standard unithreaded core dump, which contains
	the current status of all threads.
6) gdb then retreives the status of all threads by retreiving internally the
	status of all threads at the place it has been stored since the
	memory is shared between all threads, including their private stack.

This schema has two advantages:
1) it does not require any kernel modification (which should please Linus
	Torvalds ;-)
2) it is completely core format and processor-independent (ELF format does not
	cover all processors).

The usual comment I have received is that the status of all threads will not
be the one at the time the original one received the signal, since thread
signaling is done in user mode and other threads may have run. This is true
for uniprocessor machines, but it is almost impossible to have the exact
status of all others threads at the time the thread died on a multiprocessor,
since other threads may run on different processors and during the time
signal is handled, the other threads may have run several instructions before
being stopped (which is a general problem of debugging MT applications).

-Eric
P.S. nevertheless, I have kept the original patches of Philip Gladstone which
	I can resend to interested people.
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ Eric PAIRE
Web  : http://www.ri.silicomp.com/~paire  | Groupe SILICOMP - Research Institute
Email: eric.paire@ri.silicomp.com         | 2, avenue de Vignate
Phone: +33 (0) 476 63 48 71               | F-38610 Gieres
Fax  : +33 (0) 476 51 05 32               | FRANCE

Re: Multiple threads in one corefile on Linux--gdb support?

[George T. Talbot]

Eric Paire wrote:
> 
> > I've done a kernel patch for Linux which dumps multiple threads into different
> > corefiles named core.##### where ##### is the PID.  I've been told that the
> > ELF format'll support multiple register in a core file by having >1 PT_NOTE
> > section.
> >
> > If I patch the kernel to dump the multiple register sets into a single core,
> > will GDB with LinuxThreads support recognize the corefile without
> > modification?
> >
> Two years ago, when I first developed the gdb extension for debugging
> LinuxThreads applications, Philip Gladstone <philip@raptor.com> sent to me
> patches for gdb-4.16 and Linux kernel in order to generate multithreaded
> core dumps within the regular ELF format (exactly as you suggest).
> 
> Linus Torvalds has never included these patches into the kernel since they
> require that the kernel kills all clones of the thread doing the core dump,
> probably because it forces a given semantic to the cloned processes inside
> the kernel.

Why kill all the cloned processes?  Just pause them during the core dump...

In any case, do you guys save what Linus said?  I.E. his actual objections?

> Last year, I have suggested another philosophy for LinuxThread core dumps
> which does not require any kernel modification, and which I have never had
> enough spare time to implement. Everything is done in gdb and in the
> LinuxThread library. Here is a rough description of it:
> 
> 1) attach a signal handler to all signals (except SIGKILL and SIGSTOP
>         obviously) which will be called with the sigcontext,
> 2) on signal (generating a core dump) reception, signal all other threads,
> 3) on each thread receiving the signal, store the context in a reserved
>         thread-private part (usually into their own stack),
> 4) wait for the end of all threads but one (the manager one ?)
> 5) when only one thread remain, reset signal handler to default and resend
>         original signal to itself, which may or may not generate a core dump,
>         depending on the kernel implementation.
> 
> The kernel generates then a standard unithreaded core dump, which contains
>         the current status of all threads.
> 6) gdb then retreives the status of all threads by retreiving internally the
>         status of all threads at the place it has been stored since the
>         memory is shared between all threads, including their private stack.
> 
> This schema has two advantages:
> 1) it does not require any kernel modification (which should please Linus
>         Torvalds ;-)
> 2) it is completely core format and processor-independent (ELF format does not
>         cover all processors).
> 
> The usual comment I have received is that the status of all threads will not
> be the one at the time the original one received the signal, since thread
> signaling is done in user mode and other threads may have run. This is true
> for uniprocessor machines, but it is almost impossible to have the exact
> status of all others threads at the time the thread died on a multiprocessor,
> since other threads may run on different processors and during the time
> signal is handled, the other threads may have run several instructions before
> being stopped (which is a general problem of debugging MT applications).
> 
> -Eric
> P.S. nevertheless, I have kept the original patches of Philip Gladstone which
>         I can resend to interested people.

Would it be possible to do as you suggest in user-space, and also write the
core file with multiple register sections from user space so as to be
compatible with debuggers ported from other platforms?

Just thinking out loud...
--
George T. Talbot
<george at moberg dot com>
From paire@ri.silicomp.fr Thu May 11 02:13:00 2000
From: Eric Paire <paire@ri.silicomp.fr>
To: "George T. Talbot" <george@moberg.com>
Cc: gdb@sourceware.cygnus.com
Subject: Re: Multiple threads in one corefile on Linux--gdb support? 
Date: Thu, 11 May 2000 02:13:00 -0000
Message-id: <200005110913.LAA10037@mailhost.ri.silicomp.fr>
References: <391962CA.6430446@moberg.com>
X-SW-Source: 2000-05/msg00050.html
Content-length: 5010

George T. Talbot wrote:
> 
> Eric Paire wrote:
> > 
> > > I've done a kernel patch for Linux which dumps multiple threads into different
> > > corefiles named core.##### where ##### is the PID.  I've been told that the
> > > ELF format'll support multiple register in a core file by having >1 PT_NOTE
> > > section.
> > >
> > > If I patch the kernel to dump the multiple register sets into a single core,
> > > will GDB with LinuxThreads support recognize the corefile without
> > > modification?
> > >
> > Two years ago, when I first developed the gdb extension for debugging
> > LinuxThreads applications, Philip Gladstone <philip@raptor.com> sent to me
> > patches for gdb-4.16 and Linux kernel in order to generate multithreaded
> > core dumps within the regular ELF format (exactly as you suggest).
> > 
> > Linus Torvalds has never included these patches into the kernel since they
> > require that the kernel kills all clones of the thread doing the core dump,
> > probably because it forces a given semantic to the cloned processes inside
> > the kernel.
> 
> Why kill all the cloned processes?  Just pause them during the core dump...
> 
AFAIK, this is something almost impossible to achieve in the current Linux
kernel, since the process must be stopped in order to get its register values,
otherwise, you will get the ones as stored the last time I entered into the
kernel. And there is no such synchronization, since unneeded for now.

> In any case, do you guys save what Linus said?  I.E. his actual objections?
> 
No, Linus has never officially sent to me any objection; you should ask
Philip Gladstone as well.

> > Last year, I have suggested another philosophy for LinuxThread core dumps
> > which does not require any kernel modification, and which I have never had
> > enough spare time to implement. Everything is done in gdb and in the
> > LinuxThread library. Here is a rough description of it:
> > 
> > 1) attach a signal handler to all signals (except SIGKILL and SIGSTOP
> >         obviously) which will be called with the sigcontext,
> > 2) on signal (generating a core dump) reception, signal all other threads,
> > 3) on each thread receiving the signal, store the context in a reserved
> >         thread-private part (usually into their own stack),
> > 4) wait for the end of all threads but one (the manager one ?)
> > 5) when only one thread remain, reset signal handler to default and resend
> >         original signal to itself, which may or may not generate a core dump,
> >         depending on the kernel implementation.
> > 
> > The kernel generates then a standard unithreaded core dump, which contains
> >         the current status of all threads.
> > 6) gdb then retreives the status of all threads by retreiving internally the
> >         status of all threads at the place it has been stored since the
> >         memory is shared between all threads, including their private stack.
> > 
> > This schema has two advantages:
> > 1) it does not require any kernel modification (which should please Linus
> >         Torvalds ;-)
> > 2) it is completely core format and processor-independent (ELF format does not
> >         cover all processors).
> > 
> > The usual comment I have received is that the status of all threads will not
> > be the one at the time the original one received the signal, since thread
> > signaling is done in user mode and other threads may have run. This is true
> > for uniprocessor machines, but it is almost impossible to have the exact
> > status of all others threads at the time the thread died on a multiprocessor,
> > since other threads may run on different processors and during the time
> > signal is handled, the other threads may have run several instructions before
> > being stopped (which is a general problem of debugging MT applications).
> > 
> > -Eric
> > P.S. nevertheless, I have kept the original patches of Philip Gladstone which
> >         I can resend to interested people.
> 
> Would it be possible to do as you suggest in user-space, and also write the
> core file with multiple register sections from user space so as to be
> compatible with debuggers ported from other platforms?
> 
This means that the LinuxThread library has to be aware of all the core
formats supported by all versions of Linux, i.e read all core sections,
insert needed information and rewrite all core sections so that the format
is coherent with the information added. I don't think that this is the
right direction for supporting multi-threaded core dumps.

The LinuxThread has now an associated library for debug interface. It is just
a matter to add new entries to support core dump management in this library.

-Eric
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ Eric PAIRE
Web  : http://www.ri.silicomp.com/~paire  | Groupe SILICOMP - Research Institute
Email: eric.paire@ri.silicomp.com         | 2, avenue de Vignate
Phone: +33 (0) 476 63 48 71               | F-38610 Gieres
Fax  : +33 (0) 476 51 05 32               | FRANCE