Sorry for the long email below, This is mostly GDB logs with comments, promise. :-) With my latest change in the tree, we're now *not* discarding and refetching the list of shared libraries when we attach/detach to/from processes. The list is now only fetched on connection, or when the user does "info sharedlibraries". This prevents the error reported on issue #3571 from triggering on the common attach/detach case. Let me go through what is was again: Issue #3571's error is triggered due to breakpoint_re_set being called on every object file loaded. Since "attach" got rid of the previous list of SOs, attach 1 &"attach 1\n" ~"Attached to process 1\n" =thread-group-created,id="1" =thread-created,id="1",group-id="1" ^running *running,thread-id="1" &"Error while mapping shared library sections:\n" &"ProcessSupport.LM_AXG425.elf.R7T01.i686: No such file or directory.\n" ~"Reading symbols from /local/home/lmckhou/TSP/CodeSourcery/STUB/src/dicos_stub/DicosKernel_LM/bin/DicosKernel.LM_AXG561.elf.R7S01_ECHO.i686..." ~"done.\n" &"Error in re-setting breakpoint 1: No source file named /local/home/lmckhou/TSP/TADE/Test_OU/src/test.cc.\n" ... and this breakpoint was set at in an SO whose symbols are only loaded afterwards, below, we get this error above, on the first SO loaded. ~"Loaded symbols for /local/home/lmckhou/TSP/CodeSourcery/STUB/src/dicos_stub/DicosKernel_LM/bin/DicosKernel.LM_AXG561.elf.R7S01_ECHO.i686\n" ~"Symbol file not found for ProcessSupport.LM_AXG425.elf.R7T01.i686\n" ~"Reading symbols from /local/home/lmckhou/TSP/TADE/Test_LM/bin/Test.LM_Test1.elf.R1A01.i686..." ~"done.\n" ~"Loaded symbols for /local/home/lmckhou/TSP/TADE/Test_LM/bin/Test.LM_Test1.elf.R1A01.i686\n" (gdb) Now that we don't get rid of the solist so often, we don't trip on this as much. We do still trip on it on disconnect/reconnect: (gdb) b test.cc:123 Breakpoint 2 at 0xe3f6468: file /scratch/pedro/dicos/build/Test_OU/src/test.cc, line 123. (gdb) info sharedlibrary From To Syms Read Shared Object Library 0x0916b730 0x0a2f3d23 Yes /net/daisy/scratch/pedro/dicos/build/DicosKernel_LM/bin/DicosKernel.LM_AXG561.elf.R7S01_ECHO.i686 No ProcessSupport.LM_AXG425.elf.R7T01.i686 0x0e267944 0x0e3f68bf Yes /net/daisy/scratch/pedro/dicos/build/Test_LM/bin/Test.LM_Test1.elf.R1A01.i686 (gdb) info breakpoints Num Type Disp Enb Address What 2 breakpoint keep y 0x0e3f6468 in aBlockingCapsule() at /scratch/pedro/dicos/build/Test_OU/src/test.cc:123 (gdb) disconnect Ending remote debugging. (gdb) info sharedlibrary No shared libraries loaded at this time. (gdb) info breakpoints Num Type Disp Enb Address What 2 breakpoint keep y 0x0e3f6468 /scratch/pedro/dicos/build/Test_OU/src/test.cc:123 (gdb) disable 2 (gdb) enable 2 (gdb) info breakpoints Num Type Disp Enb Address What 2 breakpoint keep y 0x0e3f6468 /scratch/pedro/dicos/build/Test_OU/src/test.cc:123 (gdb) tar extended-remote :10019 Remote debugging using :10019 Error while mapping shared library sections: ProcessSupport.LM_AXG425.elf.R7T01.i686: No such file or directory. Reading symbols from /net/daisy/scratch/pedro/dicos/build/DicosKernel_LM/bin/DicosKernel.LM_AXG561.elf.R7S01_ECHO.i686...done. Error in re-setting breakpoint 2: No source file named test.cc. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Loaded symbols for /net/daisy/scratch/pedro/dicos/build/DicosKernel_LM/bin/DicosKernel.LM_AXG561.elf.R7S01_ECHO.i686 Symbol file not found for ProcessSupport.LM_AXG425.elf.R7T01.i686 Reading symbols from /net/daisy/scratch/pedro/dicos/build/Test_LM/bin/Test.LM_Test1.elf.R1A01.i686...done. Loaded symbols for /net/daisy/scratch/pedro/dicos/build/Test_LM/bin/Test.LM_Test1.elf.R1A01.i686 ^^ test.cc is found in this library. (gdb) info breakpoints Num Type Disp Enb Address What 2 breakpoint keep n 0x0e3f6468 in aBlockingCapsule() at /scratch/pedro/dicos/build/Test_OU/src/test.cc:123 ^^^ ^^^ breakpoint is now disabled. The error is almost OK to ignore, if it weren't for the fact, that when enough symbols to resolve the breakpoint are loaded, the breakpoint is left disabled, when it was enabled before disconnect. I've spent a bit trying to change this behaviour, but I couldn't get it 100% solved. Here's what I've done: Check this sequence, showing how to reproduce this easilly on linux: Setting a breakpoint, when we have symbols: >gdb ./gdb (top-gdb) b main Breakpoint 3 at 0x44f1a7: file ../../src/gdb/gdb.c, line 28. Running to it: (top-gdb) r Starting program: /home/pedro/gdb/multi_process/build/gdb/gdb [Thread debugging using libthread_db enabled] [New Thread 0x7fbdca1fb6e0 (LWP 26651)] [Switching to Thread 0x7fbdca1fb6e0 (LWP 26651)] Breakpoint 3, main (argc=1, argv=0x7fffd2220688) at ../../src/gdb/gdb.c:28 28 memset (&args, 0, sizeof args); Setting another breakpoint: (top-gdb) b gdb_main Breakpoint 4 at 0x450288: file ../../src/gdb/main.c, line 839. Let's take a look at the breakpoints state: (top-gdb) info breakpoints Num Type Disp Enb Address What 1 breakpoint keep y 0x0000000000459ca6 in internal_error at ../../src/gdb/utils.c:893 2 breakpoint keep y 0x000000000049f6d2 in info_command at ../../src/gdb/cli/cli-cmds.c:199 silent return 3 breakpoint keep y 0x000000000044f1a7 in main at ../../src/gdb/gdb.c:28 breakpoint already hit 1 time 4 breakpoint keep y 0x0000000000450288 in gdb_main at ../../src/gdb/main.c:839 Let's now discard the loaded symbols: (top-gdb) file A program is being debugged already. Are you sure you want to change the file? (y or n) y No executable file now. Discard symbol table from `/home/pedro/gdb/multi_process/build/gdb/gdb'? (y or n) y Error in re-setting breakpoint 1: No symbol table is loaded. Use the "file" command. Error in re-setting breakpoint 2: No symbol table is loaded. Use the "file" command. Error in re-setting breakpoint 3: No symbol table is loaded. Use the "file" command. Error in re-setting breakpoint 4: No symbol table is loaded. Use the "file" command. No symbol file now. Hmmm, a bunch of weird looking errors. What happened to the breakpoints? (top-gdb) info breakpoints Num Type Disp Enb Address What 1 breakpoint keep n 0x00459ca6 ../../src/gdb/utils.c:893 2 breakpoint keep n 0x0049f6d2 ../../src/gdb/cli/cli-cmds.c:199 silent return 3 breakpoint keep n 0x0044f1a7 ../../src/gdb/gdb.c:28 breakpoint already hit 1 time 4 breakpoint keep n 0x00450288 ../../src/gdb/main.c:839 Oh, right, no symbol table is loaded, so the breakpoints are gone. And, they are marked as not enabled. Ok, let's fix what GDB suspected was the reason --- let's load a symbol table: (top-gdb) file gdb A program is being debugged already. Are you sure you want to change the file? (y or n) y Reading symbols from /home/pedro/gdb/multi_process/build/gdb/gdb...done. So, at this point, breakpoint should have been enabled, shouldn't they?: (top-gdb) info breakpoints Num Type Disp Enb Address What 1 breakpoint keep n 0x0000000000459ca6 in internal_error at ../../src/gdb/utils.c:893 2 breakpoint keep n 0x000000000049f6d2 in info_command at ../../src/gdb/cli/cli-cmds.c:199 silent return 3 breakpoint keep n 0x000000000044f1a7 in main at ../../src/gdb/gdb.c:28 breakpoint already hit 1 time 4 breakpoint keep n 0x0000000000450288 in gdb_main at ../../src/gdb/main.c:839 (top-gdb) Bummer, they're still disabled... ------------------------- Patch 001 (submitted upstream a few months back already), in combination with patch 002, changes GDB's behaviour so instead of throwing an error when reparsing a breakpoint failed, we set it to pending. So now, you'd get: >./gdb --quiet --args ./gdb --quiet Setting up the environment for debugging gdb. Breakpoint 1 at 0x459ce6: file ../../src/gdb/utils.c, line 893. Breakpoint 2 at 0x49f756: file ../../src/gdb/cli/cli-cmds.c, line 199. (top-gdb) b main Breakpoint 3 at 0x44f1e7: file ../../src/gdb/gdb.c, line 28. (top-gdb) r Starting program: /home/pedro/gdb/multi_process/build/gdb/gdb --quiet [Thread debugging using libthread_db enabled] Breakpoint 3, main (argc=2, argv=0x7fffffffe448) at ../../src/gdb/gdb.c:28 28 memset (&args, 0, sizeof args); (top-gdb) b gdb_main Breakpoint 4 at 0x4502c8: file ../../src/gdb/main.c, line 839. (top-gdb) file A program is being debugged already. Are you sure you want to change the file? (y or n) y No executable file now. Discard symbol table from `/home/pedro/gdb/multi_process/build/gdb/gdb'? (y or n) y warning: Leaving breakpoint #1 pending warning: Leaving breakpoint #2 pending warning: Leaving breakpoint #3 pending warning: Leaving breakpoint #4 pending No symbol file now. (top-gdb) info breakpoints Num Type Disp Enb Address What 1 breakpoint keep y ../../src/gdb/utils.c:893 2 breakpoint keep y ../../src/gdb/cli/cli-cmds.c:199 silent return 3 breakpoint keep y ../../src/gdb/gdb.c:28 breakpoint already hit 1 time 4 breakpoint keep y ../../src/gdb/main.c:839 (top-gdb) file gdb A program is being debugged already. Are you sure you want to change the file? (y or n) y Reading symbols from /home/pedro/gdb/multi_process/build/gdb/gdb...done. (top-gdb) info breakpoints Num Type Disp Enb Address What 1 breakpoint keep y 0x0000000000459ce6 in internal_error at ../../src/gdb/utils.c:893 2 breakpoint keep y 0x000000000049f756 in info_command at ../../src/gdb/cli/cli-cmds.c:199 silent return 3 breakpoint keep y 0x000000000044f1e7 in main at ../../src/gdb/gdb.c:28 breakpoint already hit 1 time 4 breakpoint keep y 0x00000000004502c8 in gdb_main at ../../src/gdb/main.c:839 (top-gdb) Ahh, now they're left enabled. This looks promising, but when shared libraries are involved, things don't look so great. I've added a hack so that if we're fetching the solist from the inferior/target, breakpoint resetting is blocked until after loading the whole list. That's patch 003. Unfortunatelly, on linux, behaviour still isn't the best: (gdb) b main Breakpoint 1 at 0x40066c: file ../../../src/gdb/testsuite/gdb.base/solib-symbol-main.c, line 25. (gdb) r Starting program: /home/pedro/gdb/multi_process/build/gdb/testsuite/gdb.base/solib-symbol-main Breakpoint 1, main () at ../../../src/gdb/testsuite/gdb.base/solib-symbol-main.c:25 25 printf ("in main\n"); (gdb) b foo Breakpoint 2 at 0x7ffff7bdf570: file ../../../src/gdb/testsuite/gdb.base/solib-symbol-lib.c, line 22. (gdb) file A program is being debugged already. Are you sure you want to change the file? (y or n) y No executable file now. Discard symbol table from `/home/pedro/gdb/multi_process/build/gdb/testsuite/gdb.base/solib-symbol-main'? (y or n) y warning: Leaving breakpoint #1 pending warning: Leaving breakpoint #2 pending No symbol file now. (gdb) info breakpoints Num Type Disp Enb Address What 1 breakpoint keep y ../../../src/gdb/testsuite/gdb.base/solib-symbol-main.c:25 breakpoint already hit 1 time 2 breakpoint keep y ../../../src/gdb/testsuite/gdb.base/solib-symbol-lib.c:22 (gdb) file /home/pedro/gdb/multi_process/build/gdb/testsuite/gdb.base/solib-symbol-main A program is being debugged already. Are you sure you want to change the file? (y or n) y Reading symbols from /home/pedro/gdb/multi_process/build/gdb/testsuite/gdb.base/solib-symbol-main...done. (gdb) info breakpoints Num Type Disp Enb Address What 1 breakpoint keep y 0x000000000040066c in main at ../../../src/gdb/testsuite/gdb.base/solib-symbol-main.c:25 breakpoint already hit 1 time 2 breakpoint keep y 0x00000000004005a0 ^^^^^^^ (gdb) Oh, the pending breakpoint at foo is now resolved to the plt in the main binary... This happens because there are still calls to breakpoint_re_set in several places, before we get to load the symbols of the SO where foo is defined in. Specifically in this case, this happens when we load the symbols for the main executable, the expression finds the foo@plt symbol and is happy with it. So subsequent loads of SO's symbols don't change the breakpoint anymore. At the point in the code, where the symbols for the main executable are loaded, it isn't possible to know if the user or the target is going to add more symbols afterwards, which means, that we can't remove the breakpoint_re_set call... I'm much inclined to punt on this at this point, as the reported "error" will only trigger across a disconnect/reconnect, and that is not the use case Marc was reporting. What do you think? -- Pedro Alves