Re: The gdb x86 function prologue parser

Mark Kettenis
2005-06-12T07:07:49Z

From: Jason Molenda <jmolenda@apple.com>
Date: Tue, 7 Jun 2005 22:51:36 -0700

Hi Jason,

Sorry I didn't reply before; I still was on vacation when you sent out
this mail.

As was announced yesterday, we'll be transitioning from PPC to x86
over the next couple of years. Over the last few months I had a
delightful little introduction to the x86 architecture (read: read
the IA32 PDFs until I couldn't see straight any more) and worked to
get our x86 gdb support up to snuff for this software release.

Seems like a step backwards to me, but hey who am I to judge Apple.
Watched the QuickTime video where your big boss announces the thingy.
Seems he's firmly in bed with Intel and not AMD. Still makes me
wonder if Apple is going to provide a 64-bit version of MacOS X for
the new platform.

Another question that I have is what calling convention MacOS X will
use. Is it something like the System V ABI where the caller is
supposed to clean up the stack after a function call? I certainly
hope so since the Microsoft way of doing things poses a major headache
for prologue scanning.

One of the biggest problems we found was the x86 function prologue
parser is remarkably weak. We have a very mature and featureful
prologue parser on our ppc side and an amazing number of bugs were
directed my way as we had people pounding on the x86 side. We aren't
using DWARF yet, so CFI can't save our bacon -- the prologue parser
has to work or our gdb fails.

I don't know to what extent your version of gdb is synched with the
FSF tree, but if it is anything close to gdb 6.x, then yes, you're
pretty much hosed if the prologue scanner fails. It's no surprise to
me though that the prologue scanner appears a bit weak to you though.
It started out as a prologue scanner for the origional System V
compiler with some additions to older GCC versions when I took over.
At that point, GCC still had a fixed prologue. When GCC 3.x started
scheduling prologue instructions, it also started generating usable
DWARF CFI, so whe took the conscious decision to rely on that and only
improve the prologue scanner on an as-needed basis. Since GCC 3.x
targets for all major free OS'es all use DWARF by default, this means
that the prologue scanner really only handles some bits of
hand-optimized assembler.

There are a couple classes of changes I made, and I spent today
trying to massage them into some kind of presentable form. This is
not perfect -- well, to be honest, this is just a first sketch -- but
this is a HUGE improvement over the existing facilities. I wrote the
code while under immense deadline pressure so I'm not particularly
interested in how I implemented any of it. But changes akin to these
are necessary if you want to debug programs with optimized code on
the stack and without CFI. I'll guarantee it.

It'd be great if we could improve the prologue scanner. This reminds
me that I really should start testing -gstabs.

Roughly speaking, here are the class of changes included in this patch.

1. i386_match_insn() bug fixes. It wouldn't work for an instruction
pattern of 1 byte, and it would never check anything beyond the 2nd
byte (notice where the final "return insn" is located). I've added
patterns that can match prologue instructions, so exceptions had to
be added for the big two (push %ebp, mov %esp, %ebp) and the
equivalent ones used in the first frame (_start()).

Duh. I must have been on drugs when I wrote that code.

2. i386_frame_setup_skip_insns table expansion. Because you can't
skip over an unknown instruction on x86 without knowing its length,
this was of paramount importance. Initially I waited for users to
tell me of prologues that gdb was failing on, but this was taking too
long and there were too many instructions scheduled into prologues
for me to hear of them in time. So I wrote a little maintenance
command (not included in the patch to keep things simple) which would
tell you if gdb could parse through the prologue of a given
function. Then with a couple of shell scripts, I could have gdb try
to analyze the prologues of every function in every library on my
MacOS X system and show me the ones it failed on. I'd add them to
this list. I also made a little testsuite generator where the input
looks like

# SOURCE: RedHat FedoraCore2 /lib/ld-2.3.3.so _dl_reloc_bad_type
# PROLOGUE
push %ebp
shl $0x5, %ecx # [ 0xc1 0xe1 0x05 ]
mov %esp, %ebp
sub $0x8, %esp
# EPILOGUE
add $0x8, %esp
pop %ebp
ret

and a script that transforms the patterns into a test program and a
Dejagnu expect script. So you can ensure that you don't regress the
prologue parser. This was the lesson we learned in writing our PPC
parser -- we have this wonderfully ornate parser with lots of
exceptions and known tricks, but no testsuite for it. So whenever we
change it we're cringing because the gdb testsuite has nothing useful
in it. (you need optimized, no debug info test cases to be sure it's
still working right). The testsuite stuff isn't included in this
patch, but I'll put that together soon and send it along if anyone's
interested.

Certainly, if we change the prologue scanner to deal with new patterns
we should test these patterns in the testsuite.

3. relatively minor changes to i386_analyze_frame_setup(). It had to
have the push %ebp as the very first instruction or it would give
up. That's really bad -- the compiler can (and does) schedule all
sorts of stuff before that instruction.

I believe you. I'm wondering though if the current way the prologue
scanner is built up makes sense for this new world of completely
scheduled prologues.

4. new function, i386_find_esp_adjustments(). This is used in a
frameless leaf function where the compiler may create space on the
stack for local variables and stuff, but doesn't call anything so it
doesn't save the caller's frame pointer. And it allows -fomit-leaf-
frame-pointer codegen to be debugged. -fomit-frame-pointer is a
whole lot more complicated, but this wasn't so bad. (we didn't end
up enabling -fomit-leaf-frame-pointer in this release because of the
schedule time constraints, but that's why I wrote it)

Being able to debug -fomit-frame-pointer code without CFI probably
means that instead of scanning the prologue, we'll have to scan the
complete function up to the current instruction pointer. I really
wonder if that's the way we should go.

5. Huge i386_frame_cache() changes. There's no way around it, this
function is just not right. It doesn't handle frameless functions
correctly at all. It's written without a clear understanding of the
different classes of functions it needs to handle and works primarily
by luck. And for goodness sakes, if we can't figure out anything
about a function that's not at the top of the stack, don't you think
it'd be reasonable to assume that the function has set up a stack
frame and saved the caller's EBP? Sure seems like a reasonable
assumption to me. Why can't this function do something even that
basic? This function really cheesed my mellow.

Well, it handles most of the frameless functions encountered on a
GNU/Linux system with GCC 3.2 fine. And no, assuming that a function
has set up a stack frame isn't right; it makes gdb silently skip
function calls in backtraces. That can be very confusing. As I've
stated before, I'd rather have a backtrace that's obviously wrong than
one that silently omits things. But it's a trade-off. Maybe
improving the prologue scanner can shift the balance far enough that
the assumption that a stack frame has been set up makes more sense
again.

Assuming that a function saves the previous value of %ebp is demanded
by the System V ABI, but GCC might violate the ABI for static
functions where it knows the caller has already saved %ebp.

Mark, I want to say that I'm not directing any of these criticisms
towards you -- I've been looking over the changes you've made and
they're definite improvements over the existing code. The existing
code bites, though. I can't even begin to imagine how annoyed
developers using the FSF gdb on x86 must be. The changes I'm sending
here are not a panacea/beautiful/perfect, but *functionally* they're
a huge improvement. Now that our release is out there I'll be more
than happy to revisit the decisions/implementation that I came up
with on little sleep. :-)

Great. I haven't looked at your patch in detail yet. But it sounds
like some of the improvements can be made right away, so let's get
working on this ;-).

Oh, and I ran my "find all prologues gdb can't parse" on a FedoraCore
2 system I have handy here at the office today and added the patterns
of the biggest offenders. There are still a few patterns I need to
add to get 100% parsing success but I want to go home :-) so that'll
be for another day.

We're in the middle of an all-week Apple developer's conference, so
my replies may not be very speedy (I'm off-line 10-12 hours a day; I
slipped away to get this patch together today) until the weekend.
But I'll try to stay on top of any questions or comments and address
them promptly.

No problem; I was having a vacation anyway ;-).

Mark