stabs vs. dwarf-2 for C programs

stabs vs. dwarf-2 for C programs

[J.T. Conklin]

In general, are there any advantages for using dwarf-2 over 
stabs debugging symbols for C (not C++) programs?

I did a quick test of rebuilding our system with dwarf-2 debug
symbols, and found that the image file grew from 35MB to 167MB 
and link times nearly quadrupled, so dwarf-2 isn't looking so 
good so far.  If I had to guess, it looks like duplicate debug
info (from headers, etc.) isn't being eliminated as is done 
for stabs.

        --jtc

-- 
J.T. Conklin
RedBack Networks

Re: stabs vs. dwarf-2 for C programs

[Christopher Faylor]

On Thu, Apr 12, 2001 at 07:13:26PM -0700, J.T. Conklin wrote:
>In general, are there any advantages for using dwarf-2 over 
>stabs debugging symbols for C (not C++) programs?
>
>I did a quick test of rebuilding our system with dwarf-2 debug
>symbols, and found that the image file grew from 35MB to 167MB 
>and link times nearly quadrupled, so dwarf-2 isn't looking so 
>good so far.  If I had to guess, it looks like duplicate debug
>info (from headers, etc.) isn't being eliminated as is done 
>for stabs.

Daniel Berlin contributed some duplicate debug elimination code to gcc
for Dwarf-2.  It should be in the gcc 3.0 branch, if you are
adventurous.

There is an option to turn it on "-feliminate-dwarf2-dups".  I don't
know if this is the default or not.

cgf

Re: stabs vs. dwarf-2 for C programs

[J.T. Conklin]

>>>>> "cgf" == Christopher Faylor <cgf@redhat.com> writes:
>> In general, are there any advantages for using dwarf-2 over 
>> stabs debugging symbols for C (not C++) programs?
>> 
>> I did a quick test of rebuilding our system with dwarf-2 debug
>> symbols, and found that the image file grew from 35MB to 167MB 
>> and link times nearly quadrupled, so dwarf-2 isn't looking so 
>> good so far.  If I had to guess, it looks like duplicate debug
>> info (from headers, etc.) isn't being eliminated as is done 
>> for stabs.

cgf> Daniel Berlin contributed some duplicate debug elimination code
cgf> to gcc for Dwarf-2.  It should be in the gcc 3.0 branch, if you
cgf> are adventurous.

Thanks, I'll check that out.

Assuming that Dan's changes make that a non-issue, are there any other
reasons one might prefer dwarf-2 over stabs for C programs?

        --jtc

-- 
J.T. Conklin
RedBack Networks

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

jtc@redback.com (J.T. Conklin) writes:

> >>>>> "cgf" == Christopher Faylor <cgf@redhat.com> writes:
> >> In general, are there any advantages for using dwarf-2 over 
> >> stabs debugging symbols for C (not C++) programs?
> >> 
> >> I did a quick test of rebuilding our system with dwarf-2 debug
> >> symbols, and found that the image file grew from 35MB to 167MB 
> >> and link times nearly quadrupled, so dwarf-2 isn't looking so 
> >> good so far.  If I had to guess, it looks like duplicate debug
> >> info (from headers, etc.) isn't being eliminated as is done 
> >> for stabs.
> 
> cgf> Daniel Berlin contributed some duplicate debug elimination code
> cgf> to gcc for Dwarf-2.  It should be in the gcc 3.0 branch, if you
> cgf> are adventurous.
> 
> Thanks, I'll check that out.
> 
> Assuming that Dan's changes make that a non-issue, are there any other
> reasons one might prefer dwarf-2 over stabs for C programs?

Like I just posted, they probably won't make it a non-issue for C.

The pre-linker pass is on my list of things to do, after all the
optimized code debugging support and finishing the work on the new
register allocator for gcc. 

If you are using C, right now, and don't need optimized code
debugging, for at least the next 5 or 6 months, i'd stick with STABS.


> 
>         --jtc
> 
> -- 
> J.T. Conklin
> RedBack Networks

-- 
The brain is a wonderful organ; it starts working the moment you
get up in the morning and doesn't stop until you get to work.

Re: stabs vs. dwarf-2 for C programs

[John Haller]

Daniel Berlin wrote:

> The pre-linker pass is on my list of things to do, after all the
> optimized code debugging support and finishing the work on the new
> register allocator for gcc.

The native Sun Solaris linker takes an interesting approach
to eliminating duplicate debugging information.  It does
not copy much debug information from the original object
module to the final linked product, just enough to identify
which original object contains the debugging information.
The debugger then needs to locate the original object module
when setting breakpoints or looking up symbol information.
The only downside is that this prevents much symbolic debugging
if the original built objects aren't around.  It can also
speed the link not only by not having to rewrite symbolic
debug information, but also by not having to read it in the
original objects.  It also speeds up starting the debugger,
as the debugger no longer has to read the entire symbol table
at startup.

I'm glad to see other people looking at this, because in
my spare time it could take years to work on this myself.
-- 
John Haller jhaller@lucent.com

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

John Haller <jhaller@lucent.com> writes:

> Daniel Berlin wrote:
> 
> > The pre-linker pass is on my list of things to do, after all the
> > optimized code debugging support and finishing the work on the new
> > register allocator for gcc.
> 
> The native Sun Solaris linker takes an interesting approach
> to eliminating duplicate debugging information.  It does
> not copy much debug information from the original object
> module to the final linked product, just enough to identify
> which original object contains the debugging information.
> The debugger then needs to locate the original object module
> when setting breakpoints or looking up symbol information.
> The only downside is that this prevents much symbolic debugging
> if the original built objects aren't around.  It can also
> speed the link not only by not having to rewrite symbolic
> debug information, but also by not having to read it in the
> original objects.  It also speeds up starting the debugger,
> as the debugger no longer has to read the entire symbol table
> at startup.

We already don't do this anyway, without needing to do this kind of
hairy trick.

DWARF2 was made to be able to be quickly scanned, and not entirely
loaded all at once.

> 
> I'm glad to see other people looking at this, because in
> my spare time it could take years to work on this myself.
> -- 
> John Haller jhaller@lucent.com

-- 
One night a jet flew a little bit too close to my house.  I was
walking from the living room to the kitchen, and the stewardess
told me to sit down.

Re: stabs vs. dwarf-2 for C programs

[Ian Lance Taylor]

Daniel Berlin <dan@cgsoftware.com> writes:

> > The native Sun Solaris linker takes an interesting approach
> > to eliminating duplicate debugging information.  It does
> > not copy much debug information from the original object
> > module to the final linked product, just enough to identify
> > which original object contains the debugging information.
> > The debugger then needs to locate the original object module
> > when setting breakpoints or looking up symbol information.
> > The only downside is that this prevents much symbolic debugging
> > if the original built objects aren't around.  It can also
> > speed the link not only by not having to rewrite symbolic
> > debug information, but also by not having to read it in the
> > original objects.  It also speeds up starting the debugger,
> > as the debugger no longer has to read the entire symbol table
> > at startup.
> 
> We already don't do this anyway, without needing to do this kind of
> hairy trick.
> 
> DWARF2 was made to be able to be quickly scanned, and not entirely
> loaded all at once.

That's fine for the debugger, but not for the linker.  The GNU linker
still has to actually read the bytes from the object files and write
them into the executable.  Using DWARF, the debugging information can
easily be much larger than the actual executable code.  Skipping
reading and writing all the data can speed up the link step
significantly--by at least a factor of two.

Speeding up link times by a factor of two while retaining the ability
to do full debugging can significantly speed up the compile/link/debug
cycle.

Ian

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

Ian Lance Taylor <ian@zembu.com> writes:

> Daniel Berlin <dan@cgsoftware.com> writes:
> 
> > > The native Sun Solaris linker takes an interesting approach
> > > to eliminating duplicate debugging information.  It does
> > > not copy much debug information from the original object
> > > module to the final linked product, just enough to identify
> > > which original object contains the debugging information.
> > > The debugger then needs to locate the original object module
> > > when setting breakpoints or looking up symbol information.
> > > The only downside is that this prevents much symbolic debugging
> > > if the original built objects aren't around.  It can also
> > > speed the link not only by not having to rewrite symbolic
> > > debug information, but also by not having to read it in the
> > > original objects.  It also speeds up starting the debugger,
> > > as the debugger no longer has to read the entire symbol table
> > > at startup.
> > 
> > We already don't do this anyway, without needing to do this kind of
> > hairy trick.
> > 
> > DWARF2 was made to be able to be quickly scanned, and not entirely
> > loaded all at once.
> 
> That's fine for the debugger, but not for the linker.  The GNU linker
> still has to actually read the bytes from the object files and write
> them into the executable.  Using DWARF, the debugging information can
> easily be much larger than the actual executable code.  Skipping
> reading and writing all the data can speed up the link step
> significantly--by at least a factor of two.
> 
> Speeding up link times by a factor of two while retaining the ability
> to do full debugging can significantly speed up the compile/link/debug
> cycle.

Effectively, you are saying "Linking without linking some of the info,
makes it faster".
Of course it does.

And if you just had the linker merge all the debug info, and throw it
in a file, but not link the objects, it'd be faster, too.
However, it's not linking.
With things like dwarf2 dupe elimination implemented in compilers
using linkonce sections, and relocations on debug info in object
files, leaving the debugging info  in the object files, is simply
impossible.  It's a hack. Nothing more or less. 

Why don't we try actually optimizing the reading/writing of files in
ld?

Other linkers are tons faster on the same size executables, without
doing what he proposes.

I suspect there is a reason for this.
> 
> Ian

-- 
I was in the grocery store.  I saw a sign that said "pet
supplies."  So I did.  Then I went outside and saw a sign that
said "compact cars"...

Re: stabs vs. dwarf-2 for C programs

[Ian Lance Taylor]

Daniel Berlin <dan@cgsoftware.com> writes:

> With things like dwarf2 dupe elimination implemented in compilers
> using linkonce sections, and relocations on debug info in object
> files, leaving the debugging info  in the object files, is simply
> impossible.  It's a hack. Nothing more or less. 

Huh?  Leaving the debugging information in object files is clearly not
impossible.  You need a minimal amount of debugging information in the
executable: enough for it to determine which object file to check for
information about a particular address range.  That is not impossible.
The debugger needs to process relocations at debug time.  That is not
impossible.

Do you have a valid reason to reject this approach?  I honestly don't
know why you are so critical.  This is doable, and it would help.

> Why don't we try actually optimizing the reading/writing of files in
> ld?
> 
> Other linkers are tons faster on the same size executables, without
> doing what he proposes.
> 
> I suspect there is a reason for this.

Yes, if that's true, I'm sure there are reasons, and as the single
person most responsible for the present GNU linker code, it's quite
possible that any such reasons can be traced directly to me.

But I hope you don't think that nobody has ever tried to optimize the
linker.  In fact, I've spent months on it.  At one time, around 1995
or 1996, the speed of the linker was within 50% of the speed of cat
(on SunOS, for a.out).  However, the addition of shared library
support has definitely slowed the linker down, as in the current
regime ELF relocations are processed twice--once to build the PLT and
GOT, and once to actually perform the relocations.  At the time I
implemented it, I couldn't figure out an alternate approach.  Perhaps
we can do so, with much more knowledge and familiarity with the
requirements.

Actually, as I've said before, I think the correct approach right now
is to write an ELF specific linker.  The current linker is very
flexible, which is good, but that flexibility slows it down for the
most common uses.

Ian

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

Ian Lance Taylor <ian@zembu.com> writes:

> Daniel Berlin <dan@cgsoftware.com> writes:
> 
> > With things like dwarf2 dupe elimination implemented in compilers
> > using linkonce sections, and relocations on debug info in object
> > files, leaving the debugging info  in the object files, is simply
> > impossible.  It's a hack. Nothing more or less. 
> 
> Huh?  Leaving the debugging information in object files is clearly not
> impossible.  

Of course not, unless you perform elimination of debug info, in which
case, it can be impossible without having to recompile possibly every
object, every time (see below).


> You need a minimal amount of debugging information in the
> executable: enough for it to determine which object file to check for
> information about a particular address range.  That is not impossible.
> The debugger needs to process relocations at debug time.  That is not
> impossible.
> 
> Do you have a valid reason to reject this approach?

Yes. It requires the debugger to become a linker.
You haven't removed the time necessary, you've simply moved it to
another place, excluding disk write time from the end of link pass.

But mainly, it means you have to
do dwarf2/stabs elimination across object files, which makes those object files
potentially useless on the next compile.
What happens if one of the changes to one of the source files causes a
class to suddenly not appear in that object file (which can happen due
to deferred output and whatnot.  You don't instantiate a template
class anymore, by removing a usage of std::string, or something) , and
the elimination had chosen to eliminate all instances of std::string's
debug info, except that one, on the previous link?

Now yer screwed.

This is because DWARF2 elimination isn't based on include file
elimination, it's based on the equivalence of die's (though this could
be changed, if necessary, at the cost of making object files bigger,
because it would mean not deferring output of debug info, as gcc does
now, instead outputting everything we see, so that including a file
gives you the same debug info no matter what code you write)

And what about constructor sections, or destructor sections?
Or sections that only get merged by the linker?

You need to do the same amount of linking, you've just spread the time
around, and removed the time necessary to write (probably at the cost
of memory) the final executable.

In fact, that's all this approach saves. The cost of writing. We still
eat the cost of reading, and the cpu usage, and we have to perform the
linking/relocation somewhere, so we eat as much memory, as you would disk.

Unless you only use C, of course, in which case this is a perfectly
valid approach.


> How
>   I honestly don't
> know why you are so critical. 

Because I don't want GDB to turn into a linker. 

>  This is doable, and it would help.
> 
> > Why don't we try actually optimizing the reading/writing of files in
> > ld?
> > 
> > Other linkers are tons faster on the same size executables, without
> > doing what he proposes.
> > 
> > I suspect there is a reason for this.
> 
> Yes, if that's true, I'm sure there are reasons, and as the single
> person most responsible for the present GNU linker code, it's quite
> possible that any such reasons can be traced directly to me.
> 
> But I hope you don't think that nobody has ever tried to optimize the
> linker.
Of course not. I'm sure they have.
But times change, disk speeds change, memory speeds change, and
bottlenecks change.  

>   In fact, I've spent months on it.  At one time, around 1995
> or 1996, the speed of the linker was within 50% of the speed of cat
> (on SunOS, for a.out).  However, the addition of shared library
> support has definitely slowed the linker down, as in the current
> regime ELF relocations are processed twice--once to build the PLT and
> GOT, and once to actually perform the relocations.  At the time I
> implemented it, I couldn't figure out an alternate approach.  Perhaps
> we can do so, with much more knowledge and familiarity with the
> requirements.

> Actually, as I've said before, I think the correct approach right now
> is to write an ELF specific linker.

I believe Ulrich is doing this.
>   The current linker is very
> flexible, which is good, but that flexibility slows it down for the
> most common uses.

Correct.

It also doesn't support a lot of things that are nice for disk i/o
dominated programs like linking. These days, they are dominated by
seek time, and we do *tons* of seeks in bfd.  We'd be better off using
mmap if possible (file windows aren't the answer, just map the file
when open, unmap when close, and if a flag in the bfd structure says
it's mapped, use the right memory functions).

I'm pretty sure this alone will cut linking time by a large factor,
based on the number of seeks and the amount of time they are taking,
in a few tests i just did.

> 
> Ian

-- 
I went to the cinema, and the prices were:  Adults $5.00,
children $2.50.  So I said, "Give me two boys and a girl."

Re: stabs vs. dwarf-2 for C programs

[Ian Lance Taylor]

Daniel Berlin <dan@cgsoftware.com> writes:

> > > With things like dwarf2 dupe elimination implemented in compilers
> > > using linkonce sections, and relocations on debug info in object
> > > files, leaving the debugging info  in the object files, is simply
> > > impossible.  It's a hack. Nothing more or less. 
> > 
> > Huh?  Leaving the debugging information in object files is clearly not
> > impossible.  
> 
> Of course not, unless you perform elimination of debug info, in which
> case, it can be impossible without having to recompile possibly every
> object, every time (see below).

If you are leaving debug info in object files, I don't understand why
you would do elimination of debug info.  The goal of leaving debug
info in object files is to speed up the linker at the expense of the
debugger.  The usual purpose of eliminating debug info is to shrink
the size of the executable (which does not apply) and to speed up the
debugger at the expense of the linker (which does not apply).

Is there some other reason to eliminate debug info?

> > You need a minimal amount of debugging information in the
> > executable: enough for it to determine which object file to check for
> > information about a particular address range.  That is not impossible.
> > The debugger needs to process relocations at debug time.  That is not
> > impossible.
> > 
> > Do you have a valid reason to reject this approach?
> 
> Yes. It requires the debugger to become a linker.
> You haven't removed the time necessary, you've simply moved it to
> another place, excluding disk write time from the end of link pass.

Well, yes.  That's the whole point.  The linker is run more often than
the debugger.  Quite a bit more often, in fact.  It can be a good idea
to speed up the linker at the expense of the debugger.

It doesn't require the debugger to become a full-fledged linker, of
course.  It only requires the debugger to process the simple
relocations which can appear in debugging sections.  This is fairly
easy, and in fact BFD provides a function which does it.

[ I omit further discussions about debug info elimination, since I
  don't understand why you would do it when leaving debug info in
  object files. ]

> And what about constructor sections, or destructor sections?
> Or sections that only get merged by the linker?

I give up.  What about them?

> You need to do the same amount of linking, you've just spread the time
> around, and removed the time necessary to write (probably at the cost
> of memory) the final executable.
> 
> In fact, that's all this approach saves. The cost of writing. We still
> eat the cost of reading, and the cpu usage, and we have to perform the
> linking/relocation somewhere, so we eat as much memory, as you would disk.

No.  The approach saves the cost of reading the debug information from
the files, it saves the cost of processing the debug information for
duplicate information, it saves the cost of processing debug
information relocations, and, as you say, it saves the cost of writing
the debug information.

> It also doesn't support a lot of things that are nice for disk i/o
> dominated programs like linking. These days, they are dominated by
> seek time, and we do *tons* of seeks in bfd.  We'd be better off using
> mmap if possible (file windows aren't the answer, just map the file
> when open, unmap when close, and if a flag in the bfd structure says
> it's mapped, use the right memory functions).
> 
> I'm pretty sure this alone will cut linking time by a large factor,
> based on the number of seeks and the amount of time they are taking,
> in a few tests i just did.

It's possible.  Around 1997 or so I tried using mmap as you suggest.
It actually made the linker slower on the systems I tested it on,
including Solaris and on the Linux kernel, so I dropped the effort.
But, as you say, times change, and maybe now it would indeed be
faster.  It's an easy optimization in BFD; why don't you try it?

Ian

Re: stabs vs. dwarf-2 for C programs

[DJ Delorie]

> > I'm pretty sure this alone will cut linking time by a large factor,
> > based on the number of seeks and the amount of time they are taking,
> > in a few tests i just did.
> 
> It's possible.  Around 1997 or so I tried using mmap as you suggest.
> It actually made the linker slower on the systems I tested it on,
> including Solaris and on the Linux kernel, so I dropped the effort.
> But, as you say, times change, and maybe now it would indeed be
> faster.  It's an easy optimization in BFD; why don't you try it?

One of the optimizations in DJGPP's runtime library is specifically
because the linker does a lot of seeks (other programs, like gcc, tend
to stream data instead).  We changed the buffer size dynamically to
adapt to random vs linear accesses.  Of course, this assumes your
default buffer size is bigger than a sector anyway.  Linux seems to
have a 4k buffer; try setting the buffer size to 512 for all bfds in
the linker and see if it improves performance.

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

Ian Lance Taylor <ian@zembu.com> writes:

> Daniel Berlin <dan@cgsoftware.com> writes:
> 
> > > > With things like dwarf2 dupe elimination implemented in compilers
> > > > using linkonce sections, and relocations on debug info in object
> > > > files, leaving the debugging info  in the object files, is simply
> > > > impossible.  It's a hack. Nothing more or less. 
> > > 
> > > Huh?  Leaving the debugging information in object files is clearly not
> > > impossible.  
> > 
> > Of course not, unless you perform elimination of debug info, in which
> > case, it can be impossible without having to recompile possibly every
> > object, every time (see below).
> 
> If you are leaving debug info in object files, I don't understand why
> you would do elimination of debug info. 

Errr, because the memory usage in the debugger will jump otherwise.

Not just by a small amount, either.   Assuming you have a lot of
common headers, which most programs do, at least for the average C++
programs (where 95% of the debug info comes from headers, rather than
code), you'll effectively multiply the memory usage of the debugger 
by the number of object files you have.
This is not a good idea at all.

However, even with C, it's still a killer.

JT just mentioned without dwarf2, his debug executable builds are 67 meg, and
with dwarf2, 167 meg.

this is dwarf2 without elimination, and before linking, dwarf2 object
files are usually 70-80% the size of the stabs objects (and dominate
the size of an object file). So while he
could probably debug in about 30-200 meg of memory before (varying on
what part he's debugging), he now debugs in 130-400 megs.
Not a pretty sight.

Unless you suggest we do the elimination in the debugger?
I tried this, and it worked fine on the memory issue, but people just
complained about the executable size anyway. :)

>  The goal of leaving debug
> info in object files is to speed up the linker at the expense of the
> debugger.  The usual purpose of eliminating debug info is to shrink
> the size of the executable (which does not apply) and to speed up the
> debugger at the expense of the linker (which does not apply).
It does if you are talking about going from using 200 meg of memory in
gdb to using 600 meg of memory in GDB, it does apply.

Remember, link times are only long on large projects anyway, so we
aren't talking about a small amount of debug info.

> 
> Is there some other reason to eliminate debug info?

Yes, to reduce debugger memory usage.
> 
> > > You need a minimal amount of debugging information in the
> > > executable: enough for it to determine which object file to check for
> > > information about a particular address range.  That is not impossible.
> > > The debugger needs to process relocations at debug time.  That is not
> > > impossible.
> > > 
> > > Do you have a valid reason to reject this approach?
> > 
> > Yes. It requires the debugger to become a linker.
> > You haven't removed the time necessary, you've simply moved it to
> > another place, excluding disk write time from the end of link pass.
> 
> Well, yes.  That's the whole point.  The linker is run more often than
> the debugger. 
Depends, I guess.
I find i run the debugger as much as the linker, but i deal with
either gcc, gdb, or hairy multithreaded programs that seem to not do
the locking they need to :)
>  Quite a bit more often, in fact.  It can be a good idea
> to speed up the linker at the expense of the debugger.

Yes, to a certain degree. But I think you'll end up raping the
debugger :).

If you don't, i'm all for it.

> 
> It doesn't require the debugger to become a full-fledged linker, of
> course.  It only requires the debugger to process the simple
> relocations which can appear in debugging sections.  This is fairly
> easy, and in fact BFD provides a function which does it.

Sure.
> 
> [ I omit further discussions about debug info elimination, since I
>   don't understand why you would do it when leaving debug info in
>   object files. ]

See the above. 

Think about, for instance, the mozilla guys having to need 50x as much
memory to debug programs, though it links 10x faster.

It's probably not a good tradeoff in this case.
I can't think of many cases where it *is* a good tradeoff, unless you
almost never debug, in which case, why the heck are you doing debug
builds anyway?


> 
> 
> > You need to do the same amount of linking, you've just spread the time
> > around, and removed the time necessary to write (probably at the cost
> > of memory) the final executable.
> > 
> > In fact, that's all this approach saves. The cost of writing. We still
> > eat the cost of reading, and the cpu usage, and we have to perform the
> > linking/relocation somewhere, so we eat as much memory, as you would disk.
> 
> No.  The approach saves the cost of reading the debug information from
> the files, it saves the cost of processing the debug information for
> duplicate information, it saves the cost of processing debug
> information relocations, and, as you say, it saves the cost of writing
> the debug information.

This has to be much lower than the cost of reading and writing other
sections. There is nothing much to process. I would imagine
that debug info sections probably take the least amount of time to
process of all the sections, and that section time processing is
dominated by a few section types that require more than just "read
huge chunk->process huge chunk->write huge chunk whenever we flush for
some reason".  I could be completely off the mark, of course.

I just don't see the debug info as the big eater of disk i/o, when
disk i/o is dominated by seeks.  We can easily read 16 meg a second
sequentially, so even a debug info section 16 times larger (which
wouldn't be uncommon) than one requiring what amounts mostly to
non-sequential i/o probably goes at the same speed.

> 
> > It also doesn't support a lot of things that are nice for disk i/o
> > dominated programs like linking. These days, they are dominated by
> > seek time, and we do *tons* of seeks in bfd.  We'd be better off using
> > mmap if possible (file windows aren't the answer, just map the file
> > when open, unmap when close, and if a flag in the bfd structure says
> > it's mapped, use the right memory functions).
> > 
> > I'm pretty sure this alone will cut linking time by a large factor,
> > based on the number of seeks and the amount of time they are taking,
> > in a few tests i just did.
> 
> It's possible.  Around 1997 or so I tried using mmap as you suggest.
> It actually made the linker slower on the systems I tested it on,
> including Solaris and on the Linux kernel, so I dropped the effort.

This isn't surprising from 1997, mmap in Linux then was horribly bad,
and  reports i read say disk's have increased in performance (but
not seek time) at about 60% a year since 1995.  At some point between
1995 and 2001, disk throughput became irrelevant, and seek time
started to dominate.   PRML was just starting to be generally used in
hard drives in 1997, too, and MR heads were in somewhat short supply
(WD didn't introduce a drive using MR heads until the end of 1997),
and both of these increased throughput a large amount (though the main
reason for doing it was the increased density, of course).


> But, as you say, times change, and maybe now it would indeed be
> faster.  It's an easy optimization in BFD; why don't you try it?
See the next post.

I'll probably do some more comprehensive tests tomorrow.

> 
> Ian

-- 
I like to fill my tub up with water, then turn the shower on and
act like I'm in a submarine that's been hit.

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

Daniel Berlin <dan@cgsoftware.com> writes:


> > 
> > No.  The approach saves the cost of reading the debug information from
> > the files, it saves the cost of processing the debug information for
> > duplicate information, it saves the cost of processing debug
> > information relocations, and, as you say, it saves the cost of writing
> > the debug information.
> 
> This has to be much lower than the cost of reading and writing other
> sections. There is nothing much to process. I would imagine
> that debug info sections probably take the least amount of time to
> process of all the sections, and that section time processing is
> dominated by a few section types that require more than just "read
> huge chunk->process huge chunk->write huge chunk whenever we flush for
> some reason".  I could be completely off the mark, of course.
> 
Of course, the above is someone incoherent now that i look at it
again.

I mean that it *should* be lower than the cost of reading and writing
other sections, because it should be mostly sequential i/o, with no
seeks.

The testing I did shows this currently isn't the case, which is
probably why we have link time increasing at the same rate as file
size.

In other words, I think a large increase in the debug info causing a
large increase in link time is a symptom of the disk i/o being a problem, and not the
cause of the problem itself.

If we can make it so large increases in debug info doesn't equal large
increase in linking time, i don't see a need to leave debug info in
the object files in most cases.  If it doesn't, and we can do it
without making the debugger unusable (i'm happy to implement
optimizations in the debugger to do this, if necessary), i'm all for
it.
As I said, more comprehensive testing tomorrow.

-- 
The other day, I was walking my dog around my building...  on
the ledge.  Some people are afraid of heights.  Not me, I'm
afraid of widths.

Re: stabs vs. dwarf-2 for C programs

[Ian Lance Taylor]

Daniel Berlin <dan@cgsoftware.com> writes:

> > If you are leaving debug info in object files, I don't understand why
> > you would do elimination of debug info. 
> 
> Errr, because the memory usage in the debugger will jump otherwise.

You would certainly have to read more information from the object
files, but I would assume that the debugger could easily discard
information it already had.  I would not expect the debugger memory
usage to increase by more than 30%.

Anyhow, I freely concede that the debugger will be slower and will
require more memory.

> Unless you suggest we do the elimination in the debugger?
> I tried this, and it worked fine on the memory issue, but people just
> complained about the executable size anyway. :)

Well, yes, but in the case we are discussing they would hardly do
that.

> > > > You need a minimal amount of debugging information in the
> > > > executable: enough for it to determine which object file to check for
> > > > information about a particular address range.  That is not impossible.
> > > > The debugger needs to process relocations at debug time.  That is not
> > > > impossible.
> > > > 
> > > > Do you have a valid reason to reject this approach?
> > > 
> > > Yes. It requires the debugger to become a linker.
> > > You haven't removed the time necessary, you've simply moved it to
> > > another place, excluding disk write time from the end of link pass.
> > 
> > Well, yes.  That's the whole point.  The linker is run more often than
> > the debugger. 
> Depends, I guess.
> I find i run the debugger as much as the linker, but i deal with
> either gcc, gdb, or hairy multithreaded programs that seem to not do
> the locking they need to :)

Obviously this optimization would be optional.  Different people would
have different choices.

Even if you run the debugger exactly as often as you run the linker,
if you only debug a small part of the program, the debugger should
only have to read and parse a small portion of the debugging
information.  You might still win overall with the improvement in link
time.

> I can't think of many cases where it *is* a good tradeoff, unless you
> almost never debug, in which case, why the heck are you doing debug
> builds anyway?

Clearly we have very different approaches to development.

I run the debugger relatively rarely during development.  I normally
debug by examining the program behaviour and examining the program
code.  I normally use the debugger for examinations of core files, and
when I am stuck.

I run the linker far more often than the debugger, but that doesn't
mean that I don't want to be able to run the debugger when I need it.
I don't want to have to go do a link merely in order to debug.

> > No.  The approach saves the cost of reading the debug information from
> > the files, it saves the cost of processing the debug information for
> > duplicate information, it saves the cost of processing debug
> > information relocations, and, as you say, it saves the cost of writing
> > the debug information.
> 
> This has to be much lower than the cost of reading and writing other
> sections. There is nothing much to process.

The only real processing the linker does is symbol table handling and
relocation processing.  DWARF debugging sections don't have many
symbols, but they have plenty of relocations to process.

> I would imagine
> that debug info sections probably take the least amount of time to
> process of all the sections, and that section time processing is
> dominated by a few section types that require more than just "read
> huge chunk->process huge chunk->write huge chunk whenever we flush for
> some reason".  I could be completely off the mark, of course.

I think you are.  There aren't really any section types which require
significant special processing, and they certainly don't dominate the
link time.

Ian

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

Daniel Berlin <dan@cgsoftware.com> writes:

> Ian Lance Taylor <ian@zembu.com> writes:

> Correct.
> 
> It also doesn't support a lot of things that are nice for disk i/o
> dominated programs like linking. These days, they are dominated by
> seek time, and we do *tons* of seeks in bfd.  We'd be better off using
> mmap if possible (file windows aren't the answer, just map the file
> when open, unmap when close, and if a flag in the bfd structure says
> it's mapped, use the right memory functions).
> 
> I'm pretty sure this alone will cut linking time by a large factor,
> based on the number of seeks and the amount of time they are taking,
> in a few tests i just did.

Oh, and i know about the BFD_IN_MEMORY stuff, it just doesn't seem to
work well compared to just mmapping the same files. Probably because
mmap is a lot more optimized than just reading the whole file into
memory at once.

Actually, it's also because we do so many out of memory writes, reads, and
seeks, that it doesn't help us as much as it could.

For instance, linking gdb, which is one archive, 8 objects, requires
5000 out of memory seeks (ie fseeks), 7532 out of memory reads of
random sizes, at random offsets, and 28000 out of memory writes.

Of course, linking gdb is barely disk i/o bound, the numbers get much worse
worse with large programs.
The worst part about the reads is we do tons of small reads (almost
all of the reads are 40 bytes or less), 80% being followed (IE we do 8
40 byte reads, and a seek) by tons of seeks to positions probably just
outside the buffering range, and then reads.

After a lot of that, we start seeking back and forth to various
places, doing 3-4 reads of decreasing order (50k, 7k, 2k, 30 bytes),
each of the reads followed by a seek to some random place.

Then we start doing
read
seek
write
seek
For the majority of the reads and writes (going by total amount of
reading/writing/seeking), usually with matched reads and write sizes.
Each of these 2 seek/1 read/1 write combos could be replaced
with a single memcpy using an mmap'd implementation (at best, this is
what your buffering will give you anyway, with more overhead).

Eventually, we get to writing the generated parts of output file.
The worst part about the writes is that we are doing tons of very
small writes, interspersed at random points with seeks (for alignment
reasons, i guess).

Spending an hour hacking up a horrid implementation of "always use mmap"
in bfd (IE not submittable), on large programs, i decreased the link
time by a lot, and you can hear the difference.  :)


> 
> > 
> > Ian
> 
> -- 
> I went to the cinema, and the prices were:  Adults $5.00,
> children $2.50.  So I said, "Give me two boys and a girl."

-- 
I had to stop driving my car for a while...  The tires got
dizzy.

Re: stabs vs. dwarf-2 for C programs

[Ian Lance Taylor]

Daniel Berlin <dan@cgsoftware.com> writes:

> Oh, and i know about the BFD_IN_MEMORY stuff, it just doesn't seem to
> work well compared to just mmapping the same files. Probably because
> mmap is a lot more optimized than just reading the whole file into
> memory at once.

BFD_IN_MEMORY was a wretched hack I threw in there to support OSF/1
3.2 archive files, in which the native ar program would compress
object files stored in archives (as I recall, I disassembled the
native ar program to work out the compression scheme).  Using
BFD_IN_MEMORY was simpler than using a temporary file and much simpler
than decompressing on demand.

Since then DJ has beefed up BFD_IN_MEMORY to work for some other
cases.  But in any case it was never intended to be a poor man's
mmap.

> For instance, linking gdb, which is one archive, 8 objects, requires
> 5000 out of memory seeks (ie fseeks), 7532 out of memory reads of
> random sizes, at random offsets, and 28000 out of memory writes.
> 
> Of course, linking gdb is barely disk i/o bound, the numbers get much worse
> worse with large programs.
> The worst part about the reads is we do tons of small reads (almost
> all of the reads are 40 bytes or less), 80% being followed (IE we do 8
> 40 byte reads, and a seek) by tons of seeks to positions probably just
> outside the buffering range, and then reads.

I'm surprised that the reads are so small.  I would have thought that
most of the reads when linking ELF are for complete ELF sections, and
I wouldn't expect most ELF sections to be 40 bytes or less.

I wonder if this is in part caused by the overall trend to more,
smaller, sections?  The linker is written to handle one section at a
time, which would cause worse behaviour if there are many small
sections.

Ian

Re: stabs vs. dwarf-2 for C programs

[Todd Whitesel]

> Yes. It requires the debugger to become a linker.
> You haven't removed the time necessary, you've simply moved it to
> another place, excluding disk write time from the end of link pass.
...
> Because I don't want GDB to turn into a linker. 

Um, doesn't full LKM debugging require this, or something that is so close
as to be indistinguishable?

Todd Whitesel
toddpw @ best.com

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

jtc@redback.com (J.T. Conklin) writes:

> In general, are there any advantages for using dwarf-2 over 
> stabs debugging symbols for C (not C++) programs?

Unless you want optimized code debugging, only space savings
(theoretical, of course, right now. For C, anyway).

> 
> I did a quick test of rebuilding our system with dwarf-2 debug
> symbols, and found that the image file grew from 35MB to 167MB 
> and link times nearly quadrupled, 

Sounds right, link time is dominated by disk i/o. The file got 4 times
bigger, the link takes 4 times longer. :)
> so dwarf-2 isn't looking so 
> good so far.  If I had to guess, it looks like duplicate debug
> info (from headers, etc.) isn't being eliminated as is done 
> for stabs.

Correct.
I added support for this to gcc 3.0. It's not currently the default
because gdb, whose dwarf2 reader, being based on
the dwarf1 reader, was wholly unprepared to handle it (It assumes to
only have to process one CU at a time, and only see things inside that
CU. So when you do elimination, and end up with references into other
CU's, it barfs).  I've rewritten large parts of the dwarf2 reader to
support it, and hopefully, will submit the work sometime next week,
after i finish adding a few features.

If you want to get an idea of space savings, use -feliminate-dwarf2-dups.

However, it's meant for C++, mainly, i doubt it'll do much for C, if
anything at all.

It's mainly handling the case of duplication caused by real code in
header files, which can't occur in C.

Doing more space saving than that requires implementing the elimination in the
linker, or using a lot of space for labels and whatnot.  I tried this
once, but because of how LD works, it's impossible (we don't have
access to the labels offsets and whatnot we need. STABS, has support built
into the format for this type of thing. We don't, and thus, can't
leave these things lying around in the object files, unless we are
guaranteed GNU LD.)  It would need to be done as a pre-linker pass run
by ld or something. 

> 
>         --jtc
> 
> -- 
> J.T. Conklin
> RedBack Networks

-- 
Under my bed I have shoe box full of telephone rings.  Whenever
I get lonely I open it up just a bit and I get a call.  One time
I dropped the box all over the floor and the phone wouldn't stop
ringing, so I had it disconnected.  I bought a new phone though.
I didn't have much money so I had to buy an irregular phone --
it had no number 5 on it.  I saw a close friend of mine the
other day...  He said, "Steven, why haven't you called me?"  I
said, "I can't call everyone I want.  My new phone has no five
on it."  He said, "How long have you had it?"  I said, "I don't
know...  My calendar has no sevens on it."

Re: stabs vs. dwarf-2 for C programs

[Eli Zaretskii]

> From: Daniel Berlin <dan@cgsoftware.com>
> Date: 12 Apr 2001 22:54:44 -0400
> 
> jtc@redback.com (J.T. Conklin) writes:
> 
> > In general, are there any advantages for using dwarf-2 over 
> > stabs debugging symbols for C (not C++) programs?
> 
> Unless you want optimized code debugging, only space savings
> (theoretical, of course, right now. For C, anyway).

I _always_ debug optimized code (I think doing otherwise is not wise,
to put it gently).  What are the advantages of dwarf2 for debugging
optimized C code?

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

Eli Zaretskii <eliz@delorie.com> writes:

> > From: Daniel Berlin <dan@cgsoftware.com>
> > Date: 12 Apr 2001 22:54:44 -0400
> > 
> > jtc@redback.com (J.T. Conklin) writes:
> > 
> > > In general, are there any advantages for using dwarf-2 over 
> > > stabs debugging symbols for C (not C++) programs?
> > 
> > Unless you want optimized code debugging, only space savings
> > (theoretical, of course, right now. For C, anyway).
> 
> I _always_ debug optimized code (I think doing otherwise is not wise,
> to put it gently).  What are the advantages of dwarf2 for debugging
> optimized C code?

Any variable, not optimized out (excluding interprocedural
optimizations that cause very weird things, like a variable to be
split into two different object file sections, also known as
discontiguous scopes, which won't be
supported until DWARF 2.1), can be viewed just fine.

When I get done teaching GDB a little more, you'll not be able to
notice the difference between inline functions, and non-inline
ones. We can present them to the user as if they weren't inlined (IE
having their own frame, being able to breakpoint in them, etc), without any trouble.

Lots of other stuff too.

I know of a few compilers/debuggers that use DWARF2 to it's fullest
abilities to describe optimized code, and you can't tell the
difference between debugging a program compiled with optimization, and
without.

-- 
I love to go shopping.  I love to freak out salespeople.  They
ask me if they can help me, and I say, "Have you got anything
I'd like?"  Then they ask me what size I need, and I say, "Extra
medium."

Re: stabs vs. dwarf-2 for C programs

[Geoff Keating]

> From: jtc@redback.com (J.T. Conklin)
> Date: 12 Apr 2001 19:13:26 -0700

> In general, are there any advantages for using dwarf-2 over 
> stabs debugging symbols for C (not C++) programs?

In general, dwarf-2 is much more expressive and can deal better
with some of the more complex optimisations that gcc can do.
(I don't know if this actually works yet.)

> I did a quick test of rebuilding our system with dwarf-2 debug
> symbols, and found that the image file grew from 35MB to 167MB 
> and link times nearly quadrupled, so dwarf-2 isn't looking so 
> good so far.  If I had to guess, it looks like duplicate debug
> info (from headers, etc.) isn't being eliminated as is done 
> for stabs.

Yes, no-one has yet taught the linker how to eliminate all the
duplicates.

-- 
- Geoffrey Keating <geoffk@geoffk.org>

Re: stabs vs. dwarf-2 for C programs

[Daniel Berlin]

Geoff Keating <geoffk@geoffk.org> writes:

> > From: jtc@redback.com (J.T. Conklin)
> > Date: 12 Apr 2001 19:13:26 -0700
> 
> > In general, are there any advantages for using dwarf-2 over 
> > stabs debugging symbols for C (not C++) programs?
> 
> In general, dwarf-2 is much more expressive and can deal better
> with some of the more complex optimisations that gcc can do.
> (I don't know if this actually works yet.)

I just got it, building on the location list patch i submitted
yesterday, to track the splitting of iv's that occurs during loop
unrolling. Works great.  It turns out most of the live_range stuff in
the fsf tree that were from the Cygnus live range implementation,
isn't needed. (We really only need the start/end of the range, and a
mapping of old->new registers.  It used to have all kinds of info in
there to pass to the register allocator, since live range splitting
was a seperate pass.)
The new register allocator also needs the location lists because it
does all kinds of live range splitting, rematerialization, etc. 

Unfortunately, even the live range hacks to stabs can't support this
kind of thing (they were limited to simple splitting of registers,
etc), but that's life. STABS never could support optimized 
code debugging at all anyway. I'll probably just implement a simple
linear scan allocator for unoptimized code generation.


> 
> > I did a quick test of rebuilding our system with dwarf-2 debug
> > symbols, and found that the image file grew from 35MB to 167MB 
> > and link times nearly quadrupled, so dwarf-2 isn't looking so 
> > good so far.  If I had to guess, it looks like duplicate debug
> > info (from headers, etc.) isn't being eliminated as is done 
> > for stabs.
> 
> Yes, no-one has yet taught the linker how to eliminate all the
> duplicates.

And me being lazy, i taught the compiler how to do it instead, since
it was easier, and i didn't feel like the 3 days i spent doing it were
as wasted as the 3 weeks i spent trying to do it in LD.
Sometime this summer, i'll probably spend the 7 days to do it as a
linker prepass, called by ld.



> 
> -- 
> - Geoffrey Keating <geoffk@geoffk.org>

-- 
There was a power outage at a department store yesterday.
Twenty people were trapped on the escalators.