Re: Making "info thread" sane

Re: Making "info thread" sane

[Jim Houston]

George Anzinger wrote:
> The problem is that, for most threaded apps and for the kernel which
> treats each task as a thread, the "info thread" command gives a list
> of threads all stopped in the context switch code.  What is desired is
> to do one or more "up" commands and report info on this location.

Hi Everyone,

I just posted a couple patches (on gdb-patches) which I did to
support using kgdb on x86-64 linux.

The one titled "thread info - automatic skip frames" addresses
the problem George described above.  

My patch lets you list functions which should be automatically
skipped when choosing a frame to display as part of the 
"info threads" command.  e.g.:

	set skip-frame thread_return,schedule_timeout
	
Jim Houston - Concurrent Computer Corp.

Re: Making "info thread" sane

[George Anzinger]

Change the topic to make more sense and include the list...

Daniel Jacobowitz wrote:
> On Tue, Mar 02, 2004 at 02:14:33PM -0800, George Anzinger wrote:
> 
>>Daniel Jacobowitz wrote:
>>
>>>On Tue, Mar 02, 2004 at 01:38:38PM -0800, George Anzinger wrote:
>>>
>>>
>>>>A new "set thread_level" command that would take the "bt" level to
>>>>use on the thread display. A new "set thread_limits command that
>>>>would take two expressions that would reduce to two memory addresses.
>>>>
>>>>Which ever of these is entered last will be active and used by "info 
>>>>thread" as follows:
>>>>
>>>>if thread_level is active gdb will do the indicated number of  "up" 
>>>>operations and display the result on the info thread line for that thread 
>>>>(note there is other info on this line that will not be changed).
>>>>
>>>>if thread_limits is active gdb will do 0 or more "up" commands until the 
>>>>resultant PC is NOT between the given limits.
>>>>
>>>>The kernel, at this time, has defined symbols for the thread_limits 
>>>>command (it is used in the kernel for its internal display of threads).  
>>>>I would expect that the thread_level version would be the answer for 
>>>>theaded application programs.
>>>>
>>>>Daniel, how does this sound?
>>>
>>>
>>>I really don't have time to work on this sort of thing, unfortunately. 
>>
>>I was thinking of doing the code....
>>
>>
>>>But I think it sounds terribly awkward :) What you want is a generic,
>>>target specific way to specify the default current frame after the
>>>target stops.  The right way to do that may be with kgdb-specific
>>>extensions to GDB, or not.  Someone on the GDB list might have
>>>comments.
>>
>>I would think that the problem of all threads being in the context switch 
>>code would be much more generic than just kgdb.
>>
>>As to how to set this up, I would like to see a way for stubs to query gdb 
>>as gdb querys stubs (although the "try this, did it fail" method is a bit 
>>limited in what it can do).  Possibly a gdb request of the stub such as 
>>"tell me about your self" which is replied with by a set of couplets 
>>indicating what commands the sub can do and other useful info.  This 
>>becomes a bit more of a problem when there is no stub, but I suppose gdb is 
>>configured to handle this sort of thing.
> 
> 
> None of this should be specific to the remote protocol at all.  This is
> the same thing we do in C++; you say "catch throw", and what you really
> want to see is one frame up from where we hit the utility function.  In
> that sense, it needs to be more general.  And context-sensitive.
> 
> But the question of where to make this decision - I don't think the
> stub should be involved at all.

Well my origional suggestion was to set it up using set commands, i.e. most 
likely in the users .gdbinit file.  But I really do think the stub has a more 
detailed knowledge of what is up.  For example, if the stub is kgdb, i.e. the 
kernel, or a gdb stub in a target machines user space, doesn't gdb see the same 
thing?  And these two environments should really be treated differently WRT how 
far back to take the thead.

Another thought is to provide entry points in the target that define the area 
that needs to be backed out of.  Gdb already knows about the alloc entry point 
so this is not too far fetched.  Something like:
gdb_thread_switch_begin and gdb_thread_switch_end

-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml

Re: Making "info thread" sane

[Andrew Cagney]

Um, can you explain the problem?

Andrew

Re: Making "info thread" sane

[George Anzinger]

Andrew Cagney wrote:
> Um, can you explain the problem?

The problem is that, for most threaded apps and for the kernel which treats each 
task as a thread, the "info thread" command gives a list of threads all stopped 
in the context switch code.  What is desired is to do one or more "up" commands 
and report info on this location.

The proposal I made was to define a set command to specify an address range that 
"info thread" would "back" out of (using internal "up" commands), or to provide 
a fixed number of "up"s, depending on which made the most sense for the application.

For the kernel, the range is what is currently used when it reports thread 
information (internal kernel code, not kgdb or any such).

In the kgdb I currently have in the -mm kernel, I back this stuff up in the kgdb 
stub, but it really doesn't have enough info to always do it right (but I have 
not seen it fail yet :).  Gdb, on the other hand, has the full frame debug info 
and can do it correctly.

Please understand, if we then to a thread X command, we want to start in the 
switch code, it is only for the thread info that we want the back out.

I have proposed writing this code.  Daniel suggested that we discuss it a bit 
more to make sure it all makes sense.  I think his concern was with how we tell 
gdb what the right thing to do is.  I don't know how things like this are passed 
to gdb at this time, but I have been wondering about a mech. to communicate with 
stubs.  Such an interface could pass back the bounds to use, for example.  I 
suppose we could bury this info in the thread info preamble record...

On the other hand, you probably don't want to let me loose in gdb.  No telling 
what I might add :)


-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml

Re: Making "info thread" sane

[Eli Zaretskii]

> Date: Tue, 02 Mar 2004 16:14:35 -0800
> From: George Anzinger <george@mvista.com>
> 
> Andrew Cagney wrote:
> > Um, can you explain the problem?
> 
> The problem is that, for most threaded apps and for the kernel which treats each 
> task as a thread, the "info thread" command gives a list of threads all stopped 
> in the context switch code.  What is desired is to do one or more "up" commands 
> and report info on this location.

Can you explain why GDB should know about this?  The user could
always "up" manually or via the GDB's scripting language, right?

As I see it, the situation is analogous to when you, e.g., attach GDB
to a running process, and the backtrace shows that it is stuck in
some uninteresting system call.  The very next thing to do is either
"up" or step the program until it winds up in some application code
that _is_ interesting.  We don't request GDB to show the application
code automagically, do we?

Re: Making "info thread" sane

[Daniel Jacobowitz]

On Wed, Mar 03, 2004 at 08:01:58AM +0200, Eli Zaretskii wrote:
> > Date: Tue, 02 Mar 2004 16:14:35 -0800
> > From: George Anzinger <george@mvista.com>
> > 
> > Andrew Cagney wrote:
> > > Um, can you explain the problem?
> > 
> > The problem is that, for most threaded apps and for the kernel which treats each 
> > task as a thread, the "info thread" command gives a list of threads all stopped 
> > in the context switch code.  What is desired is to do one or more "up" commands 
> > and report info on this location.
> 
> Can you explain why GDB should know about this?  The user could
> always "up" manually or via the GDB's scripting language, right?
> 
> As I see it, the situation is analogous to when you, e.g., attach GDB
> to a running process, and the backtrace shows that it is stuck in
> some uninteresting system call.  The very next thing to do is either
> "up" or step the program until it winds up in some application code
> that _is_ interesting.  We don't request GDB to show the application
> code automagically, do we?

The interesting thing about George's situation is that there's a lot of
threads (basically, all but one of them) that we know in advance will
be stuck in context switching code.  One of the nice things about info
threads is that it shows you the current frame for all your threads;
but in this case, that's not really very interesting information.

If we could find out where those threads were _before_ they switched
out, now, that would make for an interesting overview.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Making "info thread" sane

[Andrew Cagney]

> On Wed, Mar 03, 2004 at 08:01:58AM +0200, Eli Zaretskii wrote:
> 
>>>> > Date: Tue, 02 Mar 2004 16:14:35 -0800
>>>> > From: George Anzinger <george@mvista.com>
>>>> > 
>>>> > Andrew Cagney wrote:
>>>
>>>>> > > Um, can you explain the problem?
>>>
>>>> > 
>>>> > The problem is that, for most threaded apps and for the kernel which treats each 
>>>> > task as a thread, the "info thread" command gives a list of threads all stopped 
>>>> > in the context switch code.  What is desired is to do one or more "up" commands 
>>>> > and report info on this location.

>>> Can you explain why GDB should know about this?  The user could
>>> always "up" manually or via the GDB's scripting language, right?
>>> 
>>> As I see it, the situation is analogous to when you, e.g., attach GDB
>>> to a running process, and the backtrace shows that it is stuck in
>>> some uninteresting system call.  The very next thing to do is either
>>> "up" or step the program until it winds up in some application code
>>> that _is_ interesting.  We don't request GDB to show the application
>>> code automagically, do we?

Right.  Commands like:

info threads
thread 1
info frame

should all give consistent output.  As they say, don't lie to the user.

> The interesting thing about George's situation is that there's a lot of
> threads (basically, all but one of them) that we know in advance will
> be stuck in context switching code.  One of the nice things about info
> threads is that it shows you the current frame for all your threads;
> but in this case, that's not really very interesting information.

> If we could find out where those threads were _before_ they switched
> out, now, that would make for an interesting overview.

It should be possible to script this (I suspect it isn't and hence part 
of the problem).   For instance something like:

(gdb) define kthread-info
   thread apply all try
     set $skip 0
     if $frame.func == &func you want to ignore
       set $skip = $skip + 1
     end
     silent up $skip
     thread info
     silent down $skip
   end
end

There are patches for "try" and "$func" lurking, silent is easy (if it 
makes sense).  Wonder if the parser allows "thread apply all try ...end"

I know other kernels have, over time, accumulated a bunch of scripts for 
dumping out internal structures in human readable form.  I think this 
goes into that category.

enjoy,
Andrew

Re: Making "info thread" sane

[George Anzinger]

Daniel Jacobowitz wrote:
> On Wed, Mar 03, 2004 at 08:01:58AM +0200, Eli Zaretskii wrote:
> 
>>>Date: Tue, 02 Mar 2004 16:14:35 -0800
>>>From: George Anzinger <george@mvista.com>
>>>
>>>Andrew Cagney wrote:
>>>
>>>>Um, can you explain the problem?
>>>
>>>The problem is that, for most threaded apps and for the kernel which treats each 
>>>task as a thread, the "info thread" command gives a list of threads all stopped 
>>>in the context switch code.  What is desired is to do one or more "up" commands 
>>>and report info on this location.
>>
>>Can you explain why GDB should know about this?  The user could
>>always "up" manually or via the GDB's scripting language, right?
>>
>>As I see it, the situation is analogous to when you, e.g., attach GDB
>>to a running process, and the backtrace shows that it is stuck in
>>some uninteresting system call.  The very next thing to do is either
>>"up" or step the program until it winds up in some application code
>>that _is_ interesting.  We don't request GDB to show the application
>>code automagically, do we?
> 
> 
> The interesting thing about George's situation is that there's a lot of
> threads (basically, all but one of them) that we know in advance will
> be stuck in context switching code.  One of the nice things about info
> threads is that it shows you the current frame for all your threads;
> but in this case, that's not really very interesting information.
> 
> If we could find out where those threads were _before_ they switched
> out, now, that would make for an interesting overview.

Also, I am not apposed to a macro solution.  But, as far as I know the macros 
are a bit weak.  For example, the info thread command lists the thread number, 
pid, and an info field (which in my case is the task name from the kernel task 
struct).  So how would a macro keep this info intact and display it along with 
the "up" result on the same line?  In my thoughts on this I have considered a 
maintaince request to the host which would return the pid and the info field, ....

I also wonder how to determine, when stepping through the threads, that that was 
the last one.  As far as I know the top thread number is not available as a $var 
(but that would be nice for other macros as well).


> 

-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml

Re: Making "info thread" sane

[Andrew Cagney]

>> The interesting thing about George's situation is that there's a lot of
>> threads (basically, all but one of them) that we know in advance will
>> be stuck in context switching code.  One of the nice things about info
>> threads is that it shows you the current frame for all your threads;
>> but in this case, that's not really very interesting information.
>>
>> If we could find out where those threads were _before_ they switched
>> out, now, that would make for an interesting overview.
> 
> 
> Also, I am not apposed to a macro solution.  But, as far as I know the macros are a bit weak.  For example, the info thread command lists the thread number, pid, and an info field (which in my case is the task name from the kernel task struct).  So how would a macro keep this info intact and display it along with the "up" result on the same line?  In my thoughts on this I have considered a maintaince request to the host which would return the pid and the info field, ....

> I also wonder how to determine, when stepping through the threads, that that was the last one.  As far as I know the top thread number is not available as a $var (but that would be nice for other macros as well). 

Sorry, I'm lost here.  Can you perhaphs sketch out how you'd expect GDB, 
the user, and the target to interact?

Andrew

Re: Making "info thread" sane

[George Anzinger]

Andrew Cagney wrote:
> 
>>> The interesting thing about George's situation is that there's a lot of
>>> threads (basically, all but one of them) that we know in advance will
>>> be stuck in context switching code.  One of the nice things about info
>>> threads is that it shows you the current frame for all your threads;
>>> but in this case, that's not really very interesting information.
>>>
>>> If we could find out where those threads were _before_ they switched
>>> out, now, that would make for an interesting overview.
>>
>>
>>
>> Also, I am not apposed to a macro solution.  But, as far as I know the 
>> macros are a bit weak.  For example, the info thread command lists the 
>> thread number, pid, and an info field (which in my case is the task 
>> name from the kernel task struct).  So how would a macro keep this 
>> info intact and display it along with the "up" result on the same 
>> line?  In my thoughts on this I have considered a maintaince request 
>> to the host which would return the pid and the info field, ....
> 
> 
>> I also wonder how to determine, when stepping through the threads, 
>> that that was the last one.  As far as I know the top thread number is 
>> not available as a $var (but that would be nice for other macros as 
>> well). 
> 
> 
> Sorry, I'm lost here.  Can you perhaphs sketch out how you'd expect GDB, 
> the user, and the target to interact?

First, the objective is to get something like what "info thread" does but with a 
frame that is outside of the switch code (which may mean  several frames up the 
stack).  I was considering a macro that would do a silent info thread followed 
by a loop on each discovered thread.  In your message yesterday you suggested 
something like:

thread apply all try... end

Well, I don't find "try" but the apply all seems to accept a macro as a command 
so I think this will do the right thing.  And up-silent does a silent up.

So, this would be my macro set:

define do_threads
   thread apply all do_th_lines
end

define do_th_lines
   while ($pc > $low_sched) && ($pc < $high_sched)
     up-silent
   end
   do-silent
   up
end

What is missing are: 1) I would like to not have the newline after the
"Thread 1 (Thread 1):" (a minor point, but with 100 threads it adds up) and 2) I 
would like to have the result of the "ThreadExtraInfo" on the same line (as the 
info thread command does).

Nice, would be the ability to print the final up result without doing the down 
first.  In fact this is really needed if it turns out that we are at the first 
frame which would be the case for the current thread.  Is that a command I missed?

I suspect that 2) can be handled by "info remote-process" with changes to the 
stub AND I would like this to NOT put in a linefeed.

It would appear that this has unwound into a couple of rather simple things:
a) No new line capability on the "thread apply all"
b) No new line on "info remote-process"
c) Ability to do the up/down report without moving to a new frame.




-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml

Re: Making "info thread" sane

[Andrew Cagney]

> First, the objective is to get something like what "info thread" does but with a frame that is outside of the switch code (which may mean  several frames up the stack).  I was considering a macro that would do a silent info thread followed by a loop on each discovered thread.  In your message yesterday you suggested something like:
> 
> thread apply all try... end
> 
> Well, I don't find "try" but the apply all seems to accept a macro as a command so I think this will do the right thing.  And up-silent does a silent up.

"try" isn't seen cos the patch is sitting in the bug database :-(

I mentioned "try" as without it the command will abort on the first 
error (e.g., corrupt stack for bad memory access).

> So, this would be my macro set:
> 
> define do_threads
>   thread apply all do_th_lines
> end
> 
> define do_th_lines
>   while ($pc > $low_sched) && ($pc < $high_sched)
>     up-silent
>   end
>   do-silent
>   up
> end
> 
> What is missing are: 1) I would like to not have the newline after the
> "Thread 1 (Thread 1):" (a minor point, but with 100 threads it adds up) and
 > 2) I would like to have the result of the "ThreadExtraInfo" on the 
same line (as the info thread command does).

Can you post the output so we can see exactly what you mean here?

> Nice, would be the ability to print the final up result without doing the down first.  In fact this is really needed if it turns out that we are at the first frame which would be the case for the current thread.  Is that a command I missed?

You mean an abbreviated "info frame"?  Yes, using "up" to get that 
output isn't right.

> I suspect that 2) can be handled by "info remote-process" with changes to the stub AND I would like this to NOT put in a linefeed.

What "info remote-process" command?

> It would appear that this has unwound into a couple of rather simple things:
> a) No new line capability on the "thread apply all"
> b) No new line on "info remote-process"
> c) Ability to do the up/down report without moving to a new frame. 

Andrew

Re: Making "info thread" sane

[George Anzinger]

Andrew Cagney wrote:
>> First, the objective is to get something like what "info thread" does 
>> but with a frame that is outside of the switch code (which may mean  
>> several frames up the stack).  I was considering a macro that would do 
>> a silent info thread followed by a loop on each discovered thread.  In 
>> your message yesterday you suggested something like:
>>
>> thread apply all try... end
>>
>> Well, I don't find "try" but the apply all seems to accept a macro as 
>> a command so I think this will do the right thing.  And up-silent does 
>> a silent up.
> 
> 
> "try" isn't seen cos the patch is sitting in the bug database :-(
> 
> I mentioned "try" as without it the command will abort on the first 
> error (e.g., corrupt stack for bad memory access).
> 
>> So, this would be my macro set:
>>
>> define do_threads
>>   thread apply all do_th_lines
>> end
>>
>> define do_th_lines
>>   while ($pc > $low_sched) && ($pc < $high_sched)
>>     up-silent
>>   end
>>   do-silent
>>   up
>> end
>>
>> What is missing are: 1) I would like to not have the newline after the
>> "Thread 1 (Thread 1):" (a minor point, but with 100 threads it adds 
>> up) and
> 
>  > 2) I would like to have the result of the "ThreadExtraInfo" on the 
> same line (as the info thread command does).
> 
> Can you post the output so we can see exactly what you mean here?

I haven't got it all working as yet, but the "thread apply all"
puts out something like:
  Thread 1 (Thread 1):
for each thread.  An option I would like is for this to not have new line so I 
can add to it, something like:
Thread 1 (Thread 1): (init) 0xff00....
and keep it all on one line.
> 
>> Nice, would be the ability to print the final up result without doing 
>> the down first.  In fact this is really needed if it turns out that we 
>> are at the first frame which would be the case for the current 
>> thread.  Is that a command I missed?
> 
> 
> You mean an abbreviated "info frame"?  Yes, using "up" to get that 
> output isn't right.

Not only isn't it right, it does not work if at the bottom of the stack.  Right 
now this is what errors out my macro set.
> 
>> I suspect that 2) can be handled by "info remote-process" with changes 
>> to the stub AND I would like this to NOT put in a linefeed.
> 
> 
> What "info remote-process" command?
To quote the source:

* This query allows the target stub to return an arbitrary string
  * (or strings) giving arbitrary information about the target process.
  * This is optional; the target stub isn't required to implement it.
  *
  * Syntax: qfProcessInfo        request first string
  *         qsProcessInfo        request subsequent string
  * reply:  'O'<hex-encoded-string>
  *         'l'                  last reply (empty)
  */

I have not, as yet, coded this into my stub so I don't know what gdb does with 
it, but, again, I want a way to suppress the new line that I suspect it attaches.

In the case of kgdb, what I want to return here is the task name and put it in 
the threads report much as the extra thread info is put in the info threads report.
> 
>> It would appear that this has unwound into a couple of rather simple 
>> things:
>> a) No new line capability on the "thread apply all"
>> b) No new line on "info remote-process"
>> c) Ability to do the up/down report without moving to a new frame. 
> 
> 
Actually with out c) the whole thing is a non-starter.  Is there some trick to 
make this happen today?

I think an option on the info stack command would be just the thing.  I.e. info 
stack <frame> <limit>  where limit, if present says how many frames to display 
prior to <frame>.  So info stack 3 0 would display frame 3 only while
info stack 3 2 would display frames 1,2,and 3.

-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml

Re: Making "info thread" sane

[Andrew Cagney]

>>
>>  > 2) I would like to have the result of the "ThreadExtraInfo" on the same line (as the info thread command does).
>>
>> Can you post the output so we can see exactly what you mean here?
> 
> 
> I haven't got it all working as yet, but the "thread apply all"
> puts out something like:
>  Thread 1 (Thread 1):
> for each thread.  An option I would like is for this to not have new line so I can add to it, something like:
> Thread 1 (Thread 1): (init) 0xff00....
> and keep it all on one line. 

Ah, ulgh.  I don't think it should output anything.  Something like:

foreach thread $t
end

?

Andrew

Re: Making "info thread" sane

[George Anzinger]

Andrew Cagney wrote:
>>>
>>>  > 2) I would like to have the result of the "ThreadExtraInfo" on the 
>>> same line (as the info thread command does).
>>>
>>> Can you post the output so we can see exactly what you mean here?
>>
>>
>>
>> I haven't got it all working as yet, but the "thread apply all"
>> puts out something like:
>>  Thread 1 (Thread 1):
>> for each thread.  An option I would like is for this to not have new 
>> line so I can add to it, something like:
>> Thread 1 (Thread 1): (init) 0xff00....
>> and keep it all on one line. 
> 
> 
> Ah, ulgh.  I don't think it should output anything.  Something like:
> 
> foreach thread $t
> end
> 
> 
But we are trying to build a replacement for info thread and do need the two 
numbers.  If they were available as "$" vars that would work too.  Then I could 
change the second "Thread" to "pid" as it really is.

-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml

Re: Making "info thread" sane

[George Anzinger]

Some time ago I started working on dwarf code for the linux kernel to allow gdb 
(via kgdb) to back trace through interrupts and traps.  The main problem here is 
that the back trace needs to terminate if the trap or interrupt comes from user 
space.  If it came from the kernel we want to keep on trucking.

At one time, the indication in gdb that the stack was exhausted was getting back 
a P register of 0.  Someone (don't know who) said "NOWAY", and, as I recall, gdb 
was changed to no longer do this.  The alternative proposed was to return a 
frame address of zero.

I would like to question this.  There are two problems here:

1) With a frame address of zero, none of the other registers are available to 
gdb (since they would be at location 0 or there about).  While we may be at the 
BOS, it is still possible to have meaningful register content.

2) In order to do the dwarf CFA / FDE for these frames an expression needs to be 
used.  If we use an expression for the CFA address, each FDE needs to have this 
expression, while if we use an expression for the P register, only the CFI need 
have the expression.  In the x86 entry.S code, traps will usually have a couple 
of additional things pushed on the stack prior to the call.  These are popped 
when the call returns, but the FDE for these frames, rather than being a simple 
relocation of the CFA has to do the whole expression thing.

As to using a P register of 0, as gdb currently stands, it always subtracts 1 
from the P register (which is typed as an unsigned long).  This means that it 
will be using 0xffffffff to look for a frame.  On all the machines I am aware 
of, this is a violation of the address space constraints, i.e. 0xffffffff is 
NEVER going to be in the same space as 0 and no valid frame will ever be found 
with this address.

I would like to change gdb to recognize P = 0 as a special case which indicated 
that there is no further unwind information.
-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml