Re: Remote mutli-threaded debugging with large number of threads

m.sikorski0
2007-07-16T11:38:33Z

After looking further into it and trying to narrow down issue cause I found out that probably not number of threads does matter here, but amount of debug information or binary size.
The improper case operated on binary of ~17MB (with debug symbols) - after removing symbols from all app threads except of one it seems to be working fine (binary size reduced to ~7MB).

Does anyone have any comments/suggestions? Is there any configuration change to support large binaries or is this simply a bug in GDB?
Regards
Maciej Sikorski

m.sikorski0@poczta.onet.pl napisaÂ³:
>Here are logs from this case (and for good one as comparison below).
>(gdb) info threads
>Sending packet: $mf933d58,4#a3...Ack
>Packet received: 9421ffa0
>Sending packet: $mf933d5c,4#ce...Ack
>Packet received: 7c0802a6
>Sending packet: $mf933d60,4#9c...Ack
>Packet received: 480182bd
>Sending packet: $mf933d64,4#a0...Ack
>Packet received: 93010040
>Sending packet: $mf933d64,4#a0...Ack
>Packet received: 93010040
>Sending packet: $mf933d68,4#a4...Ack
>Packet received: 7c982378
>Sending packet: $mf933d6c,4#cf...Ack
>Packet received: 93a10054
>Sending packet: $mf933d70,4#9d...Ack
>Packet received: 3ba00000
>Sending packet: $mf933d74,4#a1...Ack
>Packet received: 93c10058
>Sending packet: $mf933d78,4#a5...Ack
>Packet received: 7fc802a6
>Sending packet: $mf933d7c,4#d0...Ack
>Packet received: 93e1005c
>Sending packet: $mf933d80,4#9e...Ack
>Packet received: 7c7f1b78
>Sending packet: $mf933d84,4#a2...Ack
>Packet received: 90010064
>Sending packet: $mf933d88,4#a6...Ack
>Packet received: 38000001
>Sending packet: $mf933d8c,4#d1...Ack
>Packet received: 92e1003c
>Sending packet: $mf933d90,4#9f...Ack
>Packet received: 93210044
>Sending packet: $mf933d94,4#a3...Ack
>Packet received: 93410048
>Sending packet: $mf933d98,4#a7...Ack
>Packet received: 9361004c
>Sending packet: $mf933d9c,4#d2...Ack
>Packet received: 93810050
>Sending packet: $mf933da0,4#c7...Ack
>Packet received: 7d201828
>Sending packet: $m7ffff360,4#35...Ack
>Packet received: 7ffff3c0
>Sending packet: $qfThreadInfo#bb...Ack
>Packet received: m5138
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5288
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5287
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5286
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5285
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5284
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5283
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5282
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5281
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5203
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m516d
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m516c
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m516b
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m516a
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5169
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5168
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5167
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5166
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5165
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5164
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5163
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5162
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5161
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5160
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m515f
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m515e
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m515d
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m515c
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m515b
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m515a
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5159
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5158
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5157
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5156
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5155
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5154
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5153
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5152
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5151
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5150
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m514f
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m514e
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m514d
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m514c
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m514b
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m514a
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5149
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5147
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5146
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5144
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5143
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5142
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5141
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5140
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m513f
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m513e
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m513d
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m513c
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m513b
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m513a
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m5139
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: l
>Sending packet: $qThreadExtraInfo,5139#57...Ack
>Packet received:
>Sending packet: $qP0000001f0000000000005139#8a...Ack
>Packet received:
>Sending packet: $Hg5139#81...Ack
>Packet received: OK
>Sending packet: $p40#d4...Ack
>Packet received:
>Sending packet: $g#67...Ack
>Packet received: 000000b0308281803082ff8000000004308282500000000000000008000000000002d0000000000030828b780000000048044422105724bc39c0629839c0627839c062580ff05c5c300287580ffef8c00fe968040f94c2380ff05c900ff05c7c0ff05c780ff05c740ff05c680ff05c6400000002308282500ff0c71c30828250000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f0fa5180002d000580444420f0fa62c000000000000000000000000
>Sending packet: $m0,4#fd...Ack
>Packet received: E01
>Sending packet: $m0,4#fd...Ack
>Packet received: E01
>Sending packet: $mfffffffc,4#fa...Ack
>Packet received: E01
>Sending packet: $mfffffffc,4#fa...Ack
>Packet received: E01
>Sending packet: $m0,4#fd...Ack
>Packet received: E01
>Sending packet: $m0,4#fd...Ack
>Packet received: E01
>Sending packet: $mfffffffc,4#fa...Ack
>Packet received: E01
>Sending packet: $mfffffffc,4#fa...Ack
>Packet received: E01
> 61 Thread 20793 0x00000000 in ?? ()
>Sending packet: $qP0000001f000000000000513a#b2...Ack
>Packet received:
>Sending packet: $Hg513a#a9...Ack
>Packet received: OK
>Sending packet: $g#67...Ack
>Packet received: 000000dd311442f03114bf80000000040000000000000000000000000f936aec0002d0000000000031144b7831144f88d4410000105724bc39c0629839c0627839c0625810b53048300287580000000010b53b083114449010b531d40000000000000002000000020000000010b531b010b52cc0000000020f94c02010b531b4000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f93670c0002d000380444820f9366f0000000002000000000000000
>Sending packet: $m0,4#fd...Ack
>Packet received: E01
>Sending packet: $m0,4#fd...Ack
>Packet received: E01
>Sending packet: $mfffffffc,4#fa...Ack
>Packet received: E01
>Sending packet: $mfffffffc,4#fa...Ack
>Packet received: E01
>Sending packet: $m0,4#fd...Ack
>Packet received: E01
>Sending packet: $m0,4#fd...Ack
>Packet received: E01
>Sending packet: $mfffffffc,4#fa...Ack
>Packet received: E01
>Sending packet: $mfffffffc,4#fa...Ack
>Packet received: E01
> 60 Thread 20794 0x00000000 in ?? ()
>
>
>good case:
>(gdb) info threads
>Sending packet: $mf933d58,4#a3...Ack
>Packet received: 9421ffa0
>Sending packet: $mf933d5c,4#ce...Ack
>Packet received: 7c0802a6
>Sending packet: $mf933d60,4#9c...Ack
>Packet received: 480182bd
>Sending packet: $mf933d64,4#a0...Ack
>Packet received: 93010040
>Sending packet: $mf933d64,4#a0...Ack
>Packet received: 93010040
>Sending packet: $mf933d68,4#a4...Ack
>Packet received: 7c982378
>Sending packet: $mf933d6c,4#cf...Ack
>Packet received: 93a10054
>Sending packet: $mf933d70,4#9d...Ack
>Packet received: 3ba00000
>Sending packet: $mf933d74,4#a1...Ack
>Packet received: 93c10058
>Sending packet: $mf933d78,4#a5...Ack
>Packet received: 7fc802a6
>Sending packet: $mf933d7c,4#d0...Ack
>Packet received: 93e1005c
>Sending packet: $mf933d80,4#9e...Ack
>Packet received: 7c7f1b78
>Sending packet: $mf933d84,4#a2...Ack
>Packet received: 90010064
>Sending packet: $mf933d88,4#a6...Ack
>Packet received: 38000001
>Sending packet: $mf933d8c,4#d1...Ack
>Packet received: 92e1003c
>Sending packet: $mf933d90,4#9f...Ack
>Packet received: 93210044
>Sending packet: $mf933d94,4#a3...Ack
>Packet received: 93410048
>Sending packet: $mf933d98,4#a7...Ack
>Packet received: 9361004c
>Sending packet: $mf933d9c,4#d2...Ack
>Packet received: 93810050
>Sending packet: $mf933da0,4#c7...Ack
>Packet received: 7d201828
>Sending packet: $m7ffff350,4#34...Ack
>Packet received: 7ffff3b0
>Sending packet: $qfThreadInfo#bb...Ack
>Packet received: m2a02
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a50
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a4f
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a11
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a10
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a0e
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a0d
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a0c
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a0b
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a0a
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a09
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a08
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a07
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a06
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a05
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a04
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: m2a03
>Sending packet: $qsThreadInfo#c8...Ack
>Packet received: l
>Sending packet: $qThreadExtraInfo,2a03#7b...Ack
>Packet received:
>Sending packet: $qP0000001f0000000000002a03#ae...Ack
>Packet received:
>Sending packet: $Hg2a03#a5...Ack
>Packet received: OK
>Sending packet: $p40#d4...Ack
>Packet received:
>Sending packet: $g#67...Ack
>Packet received: 000000b0308281803082ff8000000004308282500000000000000008000000000002d0000000000030828b7800000000480444221004eff839c0629839c0627839c062580ff05c5c300287580ffef8c00fe968040f94c2380ff05c900ff05c7c0ff05c780ff05c740ff05c680ff05c6400000002308282500ff0c71c30828250bff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f2115180002d000580444220f21162c000000000000000000000000
>Sending packet: $mf211518,4#65...Ack
>Packet received: 7c000026
>Sending packet: $mf211514,4#61...Ack
>Packet received: 44000002
>Sending packet: $mf211518,4#65...Ack
>Packet received: 7c000026
>Sending packet: $mf211514,4#61...Ack
>Packet received: 44000002
> 17 Thread 10755 0x0f211518 in do_sigwaitinfo () from /tftpboot/nfs/lib/tls/libc.so.6
>Sending packet: $qP0000001f0000000000002a04#af...Ack
>Packet received:
>Sending packet: $Hg2a04#a6...Ack
>Packet received: OK
>Sending packet: $g#67...Ack
>Packet received: 000000dd311442f03114bf80000000040000000000000000000000000f936aec0002d0000000000031144b7831144f88d0a400001004eff839c0629839c0627839c0625810061e603002875800000000100623f83114449010061fec0000000000000002000000020000000010061fc810061ad8000000020f94c02010061fccbff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f93670c0002d000380444220f9366f0000000002000000000000000
>Sending packet: $mf93670c,4#9f...Ack
>Packet received: 7c000026
>Sending packet: $mf936708,4#74...Ack
>Packet received: 44000002
>Sending packet: $mf93670c,4#9f...Ack
>Packet received: 7c000026
>Sending packet: $mf936708,4#74...Ack
>Packet received: 44000002
> 16 Thread 10756 0x0f93670c in sem_wait@GLIBC_2.0 () from /tftpboot/nfs/lib/tls/libpthread.so.0
>Sending packet: $qP0000001f0000000000002a05#b0...Ack
>
>UÂ¿ytkownik Jim Blandy <jimb@codesourcery.com> napisaÂ³:
>>m.sikorski0@poczta.onet.pl writes:
>>> Hello,
>>> I faced recently strange issue using remote gdb debugging (cross i386-ppc). With ~60 threaded application debugging does not work well.
>>>
>>> Info threads returns incomplete information:
>>> (gdb) info threads
>>> 60 Thread 2150 0x00000000 in ?? ()
>>> 59 Thread 2151 0x00000000 in ?? ()
>>> 58 Thread 2152 0x00000000 in ?? ()
>>> 57 Thread 2153 0x00000000 in ?? ()
>>> 56 Thread 2155 0x00000000 in ?? ()
>>> 55 Thread 2156 0x00000000 in ?? ()
>>> 54 Thread 2157 0x00000000 in ?? ()
>>> ...
>>
>>That is disappointing. Can you type 'set debug remote 1' and see what
>>information the stub is returning to GDB for these threads?
>>
>