GDB memory usage with compressed debug info

GDB memory usage with compressed debug info

[Mike Gulick via Gdb]

Hi,

I'm observing that GDB memory usage is much higher when I have debug 
info compressed with 'objcopy --compress-debug-sections'.  In a large 
C++ application, I see the instance with uncompressed debug info use 
46GB VIRTUAL memory and 11GB RSS, and the instance with compressed debug 
info is using 46GB VIRTUAL memory and 42GB RSS.  In case it matters, the 
debug info is separated from the original binary into its own file.

It seems like GDB must load the full uncompressed debug info into memory 
when the underlying files are compressed?

I'm currently using GDB 9.2.  I checked the 10.1 NEWS and didn't see 
anything that looked like it would have changed this result, but I'm 
happy to give it a try if you think it might help.

Is there any chance this could be improved with a patch to GDB, or is 
this just the nature of compressed debug data?

Also, FYI, the NEWS link for the GDB 10.1 release on the GDB home page 
points to the NEWS file for the 9.1 release.

Thanks!

-Mike

Re: GDB memory usage with compressed debug info

[Mike Gulick via Gdb]

On 3/16/21 2:40 PM, Mike Gulick wrote:
> Hi,
> 
> I'm observing that GDB memory usage is much higher when I have debug 
> info compressed with 'objcopy --compress-debug-sections'.  In a large 
> C++ application, I see the instance with uncompressed debug info use 
> 46GB VIRTUAL memory and 11GB RSS, and the instance with compressed debug 
> info is using 46GB VIRTUAL memory and 42GB RSS.  In case it matters, the 
> debug info is separated from the original binary into its own file.
> 
> It seems like GDB must load the full uncompressed debug info into memory 
> when the underlying files are compressed?
> 
> I'm currently using GDB 9.2.  I checked the 10.1 NEWS and didn't see 
> anything that looked like it would have changed this result, but I'm 
> happy to give it a try if you think it might help.
> 
> Is there any chance this could be improved with a patch to GDB, or is 
> this just the nature of compressed debug data?
> 
> Also, FYI, the NEWS link for the GDB 10.1 release on the GDB home page 
> points to the NEWS file for the 9.1 release.
> 
> Thanks!
> 
> -Mike

I should add that this high memory usage only occurs when there is a 
FILE:LINENUM breakpoint set (or pending).  I can run the application 
being debugged, observe that GDB's memory usage is around 11 or 12 GB, 
then run 'b foobar.cpp:123', and the GDB memory usage will climb up to 
42 GB.  Interesting that symbolic breakpoints don't trigger this high 
memory usage, but line-based breakpoints do.  Does this seem expected?

Thanks,
Mike

Re: GDB memory usage with compressed debug info

[Christian Biesinger via Gdb]

On Thu, Mar 18, 2021 at 7:38 PM Mike Gulick via Gdb <gdb@sourceware.org> wrote:
>
> On 3/16/21 2:40 PM, Mike Gulick wrote:
> > Hi,
> >
> > I'm observing that GDB memory usage is much higher when I have debug
> > info compressed with 'objcopy --compress-debug-sections'.  In a large
> > C++ application, I see the instance with uncompressed debug info use
> > 46GB VIRTUAL memory and 11GB RSS, and the instance with compressed debug
> > info is using 46GB VIRTUAL memory and 42GB RSS.  In case it matters, the
> > debug info is separated from the original binary into its own file.
> >
> > It seems like GDB must load the full uncompressed debug info into memory
> > when the underlying files are compressed?
> >
> > I'm currently using GDB 9.2.  I checked the 10.1 NEWS and didn't see
> > anything that looked like it would have changed this result, but I'm
> > happy to give it a try if you think it might help.
> >
> > Is there any chance this could be improved with a patch to GDB, or is
> > this just the nature of compressed debug data?
> >
> > Also, FYI, the NEWS link for the GDB 10.1 release on the GDB home page
> > points to the NEWS file for the 9.1 release.
> >
> > Thanks!
> >
> > -Mike
>
> I should add that this high memory usage only occurs when there is a
> FILE:LINENUM breakpoint set (or pending).  I can run the application
> being debugged, observe that GDB's memory usage is around 11 or 12 GB,
> then run 'b foobar.cpp:123', and the GDB memory usage will climb up to
> 42 GB.  Interesting that symbolic breakpoints don't trigger this high
> memory usage, but line-based breakpoints do.  Does this seem expected?

I'm guessing that this is due to parsing full debug info ("expanding
psymtabs") which is not necessary for symbolic breakpoints, which can
rely on minsyms or maybe psymtabs. Have you tried using a heap
profiler to see where the memory usage comes from?

Christian

Re: GDB memory usage with compressed debug info

[Matt Rice via Gdb]

On Mon, Mar 22, 2021 at 6:16 PM Christian Biesinger via Gdb <
gdb@sourceware.org> wrote:

> On Thu, Mar 18, 2021 at 7:38 PM Mike Gulick via Gdb <gdb@sourceware.org>
> wrote:
> >
> > On 3/16/21 2:40 PM, Mike Gulick wrote:
> > > Hi,
> > >
> > > I'm observing that GDB memory usage is much higher when I have debug
> > > info compressed with 'objcopy --compress-debug-sections'.  In a large
> > > C++ application, I see the instance with uncompressed debug info use
> > > 46GB VIRTUAL memory and 11GB RSS, and the instance with compressed
> debug
> > > info is using 46GB VIRTUAL memory and 42GB RSS.  In case it matters,
> the
> > > debug info is separated from the original binary into its own file.
> > >
> > > It seems like GDB must load the full uncompressed debug info into
> memory
> > > when the underlying files are compressed?
> > >
> > > I'm currently using GDB 9.2.  I checked the 10.1 NEWS and didn't see
> > > anything that looked like it would have changed this result, but I'm
> > > happy to give it a try if you think it might help.
> > >
> > > Is there any chance this could be improved with a patch to GDB, or is
> > > this just the nature of compressed debug data?
> > >
> > > Also, FYI, the NEWS link for the GDB 10.1 release on the GDB home page
> > > points to the NEWS file for the 9.1 release.
> > >
> > > Thanks!
> > >
> > > -Mike
> >
> > I should add that this high memory usage only occurs when there is a
> > FILE:LINENUM breakpoint set (or pending).  I can run the application
> > being debugged, observe that GDB's memory usage is around 11 or 12 GB,
> > then run 'b foobar.cpp:123', and the GDB memory usage will climb up to
> > 42 GB.  Interesting that symbolic breakpoints don't trigger this high
> > memory usage, but line-based breakpoints do.  Does this seem expected?
>
> I'm guessing that this is due to parsing full debug info ("expanding
> psymtabs") which is not necessary for symbolic breakpoints, which can
> rely on minsyms or maybe psymtabs. Have you tried using a heap
> profiler to see where the memory usage comes from?
>

It could possibly narrow it down if the behavior of FILE:label breakpoints
differ, though I'm not sure what to expect exactly.

Re: GDB memory usage with compressed debug info

[Mike Gulick via Gdb]

On 3/24/21 3:59 PM, Matt Rice wrote:
> On Mon, Mar 22, 2021 at 6:16 PM Christian Biesinger via Gdb <gdb@sourceware.org> wrote:
> 
>> On Thu, Mar 18, 2021 at 7:38 PM Mike Gulick via Gdb <gdb@sourceware.org>
>> wrote:
>>>
>>> On 3/16/21 2:40 PM, Mike Gulick wrote:
>>>> Hi,
>>>>
>>>> I'm observing that GDB memory usage is much higher when I have debug
>>>> info compressed with 'objcopy --compress-debug-sections'.  In a large
>>>> C++ application, I see the instance with uncompressed debug info use
>>>> 46GB VIRTUAL memory and 11GB RSS, and the instance with compressed debug
>>>> info is using 46GB VIRTUAL memory and 42GB RSS.  In case it matters, the
>>>> debug info is separated from the original binary into its own file.
>>>>
>>>> It seems like GDB must load the full uncompressed debug info into memory
>>>> when the underlying files are compressed?
>>>>
>>>> I'm currently using GDB 9.2.  I checked the 10.1 NEWS and didn't see
>>>> anything that looked like it would have changed this result, but I'm
>>>> happy to give it a try if you think it might help.
>>>>
>>>> Is there any chance this could be improved with a patch to GDB, or is
>>>> this just the nature of compressed debug data?
>>>>
>>>> Also, FYI, the NEWS link for the GDB 10.1 release on the GDB home page
>>>> points to the NEWS file for the 9.1 release.
>>>>
>>>> Thanks!
>>>>
>>>> -Mike
>>>
>>> I should add that this high memory usage only occurs when there is a
>>> FILE:LINENUM breakpoint set (or pending).  I can run the application
>>> being debugged, observe that GDB's memory usage is around 11 or 12 GB,
>>> then run 'b foobar.cpp:123', and the GDB memory usage will climb up to
>>> 42 GB.  Interesting that symbolic breakpoints don't trigger this high
>>> memory usage, but line-based breakpoints do.  Does this seem expected?
>>
>> I'm guessing that this is due to parsing full debug info ("expanding
>> psymtabs") which is not necessary for symbolic breakpoints, which can
>> rely on minsyms or maybe psymtabs. Have you tried using a heap
>> profiler to see where the memory usage comes from?
>>
> 
> It could possibly narrow it down if the behavior of FILE:label breakpoints
> differ, though I'm not sure what to expect exactly.
> 

I've never used a heap profiler before, but I tried running a debug 
build of GDB 10.1 with heap profiling via an LD_PRELOAD of the tcmalloc 
library (https://gperftools.github.io/gperftools/heapprofile.html). 
This showed that the stack which was doing the vast majority of the 
allocations was this:

bfd_malloc /gdb-10.1/bfd/libbfd.c:275
bfd_get_full_section_contents /gdb-10.1/bfd/compress.c:320
gdb_bfd_map_section /gdb-10.1/gdb/gdb_bfd.c:724
dwarf2_section_info::read /gdb-10.1/gdb/dwarf2/section.c:161
cutu_reader::cutu_reader /gdb-10.1/gdb/dwarf2/read.c:7339
dw2_get_file_names /gdb-10.1/gdb/dwarf2/read.c:3287
dw2_map_symtabs_matching_filename /gdb-10.1/gdb/dwarf2/read.c:3399
iterate_over_symtabs /gdb-10.1/gdb/symtab.c:558
collect_symtabs_from_filename /gdb-10.1/gdb/linespec.c:3797
symtabs_from_filename /gdb-10.1/gdb/linespec.c:3817
parse_linespec /gdb-10.1/gdb/linespec.c:2613
event_location_to_sals /gdb-10.1/gdb/linespec.c:3150
decode_line_full /gdb-10.1/gdb/linespec.c:3230
parse_breakpoint_sals /gdb-10.1/gdb/breakpoint.c:9037
create_sals_from_location_default /gdb-10.1/gdb/breakpoint.c:13733
bkpt_create_sals_from_location /gdb-10.1/gdb/breakpoint.c:12534
create_breakpoint /gdb-10.1/gdb/breakpoint.c:9253
break_command_1 /gdb-10.1/gdb/breakpoint.c:9411
break_command /gdb-10.1/gdb/breakpoint.c:9482
do_const_cfunc /gdb-10.1/gdb/cli/cli-decode.c:95
cmd_func /gdb-10.1/gdb/cli/cli-decode.c:2181
execute_command /gdb-10.1/gdb/top.c:668
catch_command_errors /gdb-10.1/gdb/main.c:457
captured_main_1 /gdb-10.1/gdb/main.c:1218
captured_main /gdb-10.1/gdb/main.c:1243

I also tried setting a breakpoint using FILE:function style, and the 
memory usage was pretty much equivalent to FILE:LINENUM.  This would 
imply (as would the above stack trace) that matching the file name 
causes the full debug info to be read in.

Thanks,
Mike