Re: Does gdb debuginfod download libc etc.?

["Arsen Arsenović via Gdb" <gdb@sourceware.org>]

[-- Attachment #1: Type: text/plain, Size: 8274 bytes --]

Hi Paul,

Paul Smith <psmith@gnu.org> writes:

> I'm trying to start using debuginfod support in my environment.  I've
> started my own debuginfod server and I've added lots of historical
> binaries and shared libraries that I get core files from.
>
> In that past I've asked users to package up all their system libraries
> such as libc.so, libm.so, libpthread.so, the runtime linker ld-linux.so
> etc. and send them along with their core file.  Then I run GDB like:
>   gdb -iex 'set sysroot xxx' -c core exe
>
> What I was hoping to do is to run "gdb -c core" and have debuginfod get
> everything I need without having to ask users to send anything
> (assuming I've already got the libraries, which I usually do).  But, it
> doesn't seem to be working.
>
> I've used debuginfod-find to verify that all the libraries referenced
> by the core file do exist on the server (via their build ID).
>
> When I run "gdb -c core" it DOES correctly download the executable
> file, so I know that GDB's debuginfod configuration is working.  It
> also appears to download linux-vdso.so.1.
>
> However, none of my other libraries are being loaded, even if I use bt
> or whatever to look at the backtrace.  Without libc for example, none
> of the threads which were in the C runtime when the core was taken can
> be traced and I just get "?? ()" for every frame.  If I use "info
> sharedlib" I see all the .so paths are the local system versions of
> these libraries, and they have a "Syms Read" value of "No".

This is strange - I've tried replicating it.

For context, I run a debuginfod instance populated with packages I build
for my own Gentoo binhost.  To test this, I got a dump from a few days
ago, and put it in a blank Arch Linux container (meaning it had access
to none of my installed libraries) with gdb 17.1, and told it to look up
debug info on my debuginfod server.  It pulled all the libraries:

  [root@bstg /]# gdb -c /host/core
  GNU gdb (GDB) 17.1
  Copyright (C) 2025 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.
  Type "show copying" and "show warranty" for details.
  This GDB was configured as "x86_64-pc-linux-gnu".
  Type "show configuration" for configuration details.
  For bug reporting instructions, please see:
  <https://www.gnu.org/software/gdb/bugs/>.
  Find the GDB manual and other documentation resources online at:
      <http://www.gnu.org/software/gdb/documentation/>.

  For help, type "help".
  Type "apropos word" to search for commands related to "word".

  This GDB supports auto-downloading debuginfo from the following URLs:
    <http://localhost/debuginfod>
  Enable debuginfod for this session? (y or [n]) y
  Debuginfod has been enabled.
  To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
  Downloading 559.69 K file /usr/bin/foot
  Downloading 846.47 K file /usr/lib64/libpcre2-8.so.0.15.0
  Downloading 182.12 K file /usr/lib64/libgraphite2.so.3.2.1
  Downloading 1.43 M file /usr/lib64/libglib-2.0.so.0.8400.4
  Downloading 302.20 K file /usr/lib64/libpng16.so.16.55.0
  Downloading 97.14 K file /usr/lib64/libbz2.so.1.0.8
  Downloading 122.19 K file /usr/lib64/libz.so.1.3.2
  Downloading 1.40 M file /usr/lib64/libharfbuzz.so.0.61232.0
  Downloading 210.12 K file /usr/lib64/libexpat.so.1.11.2
  Downloading 1.13 M file /usr/lib64/libfreetype.so.6.20.4
  Downloading 66.27 K file /usr/lib64/libffi.so.8.2.0
  Downloading 2.29 M file /usr/lib64/libc.so.6
  Downloading 146.17 K file /usr/lib64/libfcft.so.4.3.2
  Downloading 350.05 K file /usr/lib64/libutf8proc.so.3.2.3
  Downloading 398.13 K file /usr/lib64/libfontconfig.so.1.16.0
  Downloading 438.34 K file /usr/lib64/libxkbcommon.so.0.13.1
  Downloading 38.64 K file /usr/lib64/libwayland-cursor.so.0.24.0
  Downloading 62.91 K file /usr/lib64/libwayland-client.so.0.24.0
  Downloading 1.17 M file /usr/lib64/libpixman-1.so.0.46.4
  Downloading 1.15 M file /usr/lib64/libm.so.6
  Downloading 3.19 M file /usr/lib64/ld-linux-x86-64.so.2
  [New LWP 44937]
  [New LWP 44936]
  [New LWP 44927]
  [New LWP 44935]
  [New LWP 44933]
  [New LWP 44940]
  [New LWP 44938]
  [New LWP 44939]
  [New LWP 44934]
  [New LWP 44930]
  [New LWP 44932]
  [New LWP 44931]
  [New LWP 44929]
  [New LWP 44928]
  [New LWP 44925]
  [New LWP 44926]
  [New LWP 44900]
  Reading symbols from /root/.cache/debuginfod_client/abbc57ebe52352cc4ffa93ac153b7c0fe2a58249/executable...
  Downloading 1.15 M separate debug info for /root/.cache/debuginfod_client/abbc57ebe52352cc4ffa93ac153b7c0fe2a58249/executable
  Reading symbols from /root/.cache/debuginfod_client/abbc57ebe52352cc4ffa93ac153b7c0fe2a58249/debuginfo...
  Downloading 2.28 M separate debug info for /root/.cache/debuginfod_client/38175f895390f3e40aa5bb2706b7647f2bac973d/executable
  Downloading 2.26 M separate debug info for /root/.cache/debuginfod_client/7c3be6dccd26bb757832f7771fad5fb212389ffb/executable
  Downloading 143.06 K separate debug info for /root/.cache/debuginfod_client/38601e6dd2b36f0ca50de7f91428cb04c3f5daf0/executable
  Downloading 75.24 K separate debug info for /root/.cache/debuginfod_client/e25df18e5a3dc25042cf8416296e719060108659/executable
  Downloading 587.47 K separate debug info for /root/.cache/debuginfod_client/70848bd835b74ec3fb95052083ddc6f1373f39d8/executable
  Downloading 711.10 K separate debug info for /root/.cache/debuginfod_client/e90348e8ad4f35454f86a262ec0b0ccab6b8b801/executable
  Downloading 34.14 K separate debug info for /root/.cache/debuginfod_client/887865c3d0a8c001dd6b69951e2580ab1828e0ad/executable
  Downloading 244.07 K separate debug info for /root/.cache/debuginfod_client/70cb794006d13437bb56195bd7994b1bf71ab23c/executable
  Downloading 6.05 M separate debug info for /root/.cache/debuginfod_client/22ca58d9cfb2479c0f72b964e33727b4ee98425d/executable
  Downloading 118.19 K separate debug info for /root/.cache/debuginfod_client/cc25dd3b84efdaab8afc4773179518016873eb53/executable
  Downloading 1.78 M separate debug info for /root/.cache/debuginfod_client/6691a92dfa6857e5f9447ae05c02cfc3f567560f/executable
  Downloading 272.71 K separate debug info for /root/.cache/debuginfod_client/ecbc8c047da9ad5284444c510628c7124e03f0fe/executable
  Downloading 13.00 M separate debug info for /root/.cache/debuginfod_client/6caf7bc7ac5a7afaa9270090608a5d695600bd20/executable
  Downloading 176.43 K separate debug info for /root/.cache/debuginfod_client/e3c6680e6a61d29aa8b4b3e84d7001468d1cb39a/executable
  Downloading 101.69 K separate debug info for /root/.cache/debuginfod_client/c62bf82cb11187867183bf2644ee2042e08c4d08/executable
  Downloading 525.26 K separate debug info for /root/.cache/debuginfod_client/c86efb7898aec6ad91a7a537375bf0edeaea0d05/executable
  Downloading 2.38 M separate debug info for /root/.cache/debuginfod_client/41b432ccf719b8a5a62b00472107d8d2798cf02a/executable
  Downloading 629.27 K separate debug info for /root/.cache/debuginfod_client/ed7f3f1ce1d5d639459b5f61a063d550d9b5571e/executable
  Downloading 1.07 M separate debug info for /root/.cache/debuginfod_client/b6705f4c7b443400e9f3136658df3ea38e7551c0/executable
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/usr/lib/libthread_db.so.1".
  Core was generated by `/usr/bin/foot'.
  Program terminated with signal SIGSEGV, Segmentation fault.
  Downloading 181.82 K source file /usr/src/debug/gui-apps/foot-1.25.0/foot-1.25.0-build/../foot-1.25.0/render.c
  #0  0x0000564ed601552c in render_worker_thread (_ctx=<optimized out>) at ../foot-1.25.0/render.c:2259
  2259	                    pixman_image_composite32(PIXMAN_OP_SRC,
  [Current thread is 1 (Thread 0x7f87bd7fa6c0 (LWP 44937))]

Do the binaries and libraries the program that dumped core have build
IDs?  More information is probably needed to be able to tell what
happened.

You can probably try debugging this by setting:

  set debug separate-debug-file on

... or such, to see whether gdb knows about the build IDs.
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 418 bytes --]