Re: [PATCH] nptl: Move stack list variables into _rtld_global

Re: [PATCH] nptl: Move stack list variables into _rtld_global

[Simon Marchi via Gdb-patches]

On 2021-03-05 12:15 p.m., Florian Weimer wrote:> * Simon Marchi via Libc-alpha:
> 
>> On 2020-11-13 10:10 a.m., Florian Weimer wrote:
>>> Now __thread_gscope_wait (the function behind THREAD_GSCOPE_WAIT,
>>> formerly __wait_lookup_done) can be implemented directly in ld.so,
>>> eliminating the unprotected GL (dl_wait_lookup_done) function
>>> pointer.
>>
>> Hi Florian,
>>
>> Presumably starting with this commit (I don't really know how to build a
>> glibc and test against it), GDB fails to attach to a threaded process
>> because libthread_db fails to initialize.  See:
>>
>>   https://sourceware.org/bugzilla/show_bug.cgi?id=27526
>>
>> The difference in behavior as seen from GDB is that libthread_db now
>> asks to look up a symbol "_dl_stack_user" in module NULL.  GDB can't
>> find this symbol, which fails the initialization.
>>
>> Can you shed some light on this?  Is this request expected, and where is
>> GDB expected to find this symbol?
> 
> It is not expected.  This is the fallback path if _rtld_global cannot be
> located.  The actual failure is that __td_ta_rtld_global does not
> succeed.

[adding gdb-patches]

Ok, thanks for that tip.  Indeed I see that GDB returns PS_NOSYM for
_rtld_global.  If I now log what GDB returns:

    LOOKUP nptl_version in libpthread.so.0
    Found 0x7fdb02713037
    LOOKUP _rtld_global in ld-linux-x86-64.so.2
    Not found
    LOOKUP _dl_stack_user in (null)
    Not found

So this lookup of _rtld_global is new too.  And I think I see the
problem, it looks like an ordering issue: libthread_db is loaded when
GDB notices the program has libpthread loaded in it.  When attaching,
GDB walks the shared library list.  In that list, libpthread comes
before ld-linux.  So at the time we try to load libthread_db, GDB hasn't
yet noticed that the program has ld-linux loaded in it, hasn't ingested
its symbols, so doesn't find _rtld_global.

For comparison, glibc 2.31 (on Ubuntu 20.04) only requested symbols in
libpthread itself, so there wasn't this ordering issue:

    LOOKUP nptl_version in libpthread.so.0
    Found 0x7f38538e9037
    LOOKUP __stack_user in libpthread.so.0
    Found 0x7f38538f3350
    LOOKUP _thread_db_list_t_next in libpthread.so.0
    Found 0x7f38538e93b0
    LOOKUP _thread_db_const_thread_area in libpthread.so.0
    Found 0x7f38538e92b4
    LOOKUP _thread_db_sizeof_pthread in libpthread.so.0
    Found 0x7f38538e92cc
    LOOKUP _thread_db_pthread_specific in libpthread.so.0
    Found 0x7f38538e9400
    LOOKUP _thread_db_pthread_schedpolicy in libpthread.so.0
    Found 0x7f38538e9420
    LOOKUP _thread_db_pthread_schedparam_sched_priority in libpthread.so.0
    Found 0x7f38538e9410
    LOOKUP _thread_db_pthread_tid in libpthread.so.0
    Found 0x7f38538e9450
    LOOKUP _thread_db_pthread_cancelhandling in libpthread.so.0
    Found 0x7f38538e9430
    LOOKUP _thread_db_pthread_report_events in libpthread.so.0
    Found 0x7f38538e9460
    LOOKUP _thread_db_pthread_start_routine in libpthread.so.0
    Found 0x7f38538e9440
    LOOKUP _thread_db_pthread_eventbuf_eventmask_event_bits in libpthread.so.0
    Found 0x7f38538e93d0

If we have to deal with this, I guess that GDB should now do things in a
different order: go through the whole library list and load their
symbols.  And then if one of those libraries were libpthread, try to
initialize libthread_db.

Simon

Re: [PATCH] nptl: Move stack list variables into _rtld_global

[Florian Weimer via Gdb-patches]

* Simon Marchi:

> On 2021-03-05 12:15 p.m., Florian Weimer wrote:> * Simon Marchi via Libc-alpha:
>> 
>>> On 2020-11-13 10:10 a.m., Florian Weimer wrote:
>>>> Now __thread_gscope_wait (the function behind THREAD_GSCOPE_WAIT,
>>>> formerly __wait_lookup_done) can be implemented directly in ld.so,
>>>> eliminating the unprotected GL (dl_wait_lookup_done) function
>>>> pointer.
>>>
>>> Hi Florian,
>>>
>>> Presumably starting with this commit (I don't really know how to build a
>>> glibc and test against it), GDB fails to attach to a threaded process
>>> because libthread_db fails to initialize.  See:
>>>
>>>   https://sourceware.org/bugzilla/show_bug.cgi?id=27526
>>>
>>> The difference in behavior as seen from GDB is that libthread_db now
>>> asks to look up a symbol "_dl_stack_user" in module NULL.  GDB can't
>>> find this symbol, which fails the initialization.
>>>
>>> Can you shed some light on this?  Is this request expected, and where is
>>> GDB expected to find this symbol?
>> 
>> It is not expected.  This is the fallback path if _rtld_global cannot be
>> located.  The actual failure is that __td_ta_rtld_global does not
>> succeed.
>
> [adding gdb-patches]
>
> Ok, thanks for that tip.  Indeed I see that GDB returns PS_NOSYM for
> _rtld_global.  If I now log what GDB returns:
>
>     LOOKUP nptl_version in libpthread.so.0
>     Found 0x7fdb02713037
>     LOOKUP _rtld_global in ld-linux-x86-64.so.2
>     Not found
>     LOOKUP _dl_stack_user in (null)
>     Not found
>
> So this lookup of _rtld_global is new too.  And I think I see the
> problem, it looks like an ordering issue: libthread_db is loaded when
> GDB notices the program has libpthread loaded in it.  When attaching,
> GDB walks the shared library list.  In that list, libpthread comes
> before ld-linux.  So at the time we try to load libthread_db, GDB hasn't
> yet noticed that the program has ld-linux loaded in it, hasn't ingested
> its symbols, so doesn't find _rtld_global.

Oh, that reads like a plausible explanation.  And I assume the
non-attaching case, where GDB starts the process, is very different, and
this ordering issue does not appear?

> If we have to deal with this, I guess that GDB should now do things in a
> different order: go through the whole library list and load their
> symbols.  And then if one of those libraries were libpthread, try to
> initialize libthread_db.

Initialization of libthread_db should be unconditional.  Programs use
TLS data without linking against libpthread.  And glibc 2.34 might not
have a separate libpthread at all.

Thanks,
Florian

Re: [PATCH] nptl: Move stack list variables into _rtld_global

[Simon Marchi via Gdb-patches]

On 2021-03-05 1:03 p.m., Florian Weimer wrote:
> Oh, that reads like a plausible explanation.  And I assume the
> non-attaching case, where GDB starts the process, is very different, and
> this ordering issue does not appear?

Indeed, in that case ld-linux-x86-64.so.2 is loaded before
libpthread.so.0 (ld-linux is necessarily loaded before the others, I
guess, since it's the one loading the others).  So the symbol is found:

    LOOKUP _rtld_global in ld-linux-x86-64.so.2
    Found 0x7ffff7ffd000

>> If we have to deal with this, I guess that GDB should now do things in a
>> different order: go through the whole library list and load their
>> symbols.  And then if one of those libraries were libpthread, try to
>> initialize libthread_db.
> 
> Initialization of libthread_db should be unconditional.  Programs use
> TLS data without linking against libpthread.  And glibc 2.34 might not
> have a separate libpthread at all.

Ok, currently GDB attempts to load libthread_db when noticing the main
objfile / program (I guess it is needed if the program is statically
linked to libpthread?) or when seeing a library named libpthread*.

I'm not sure how to fix this, other than making GDB attempt to load
libthread_db on every new shared library it notices, since that new
shared library may "finally" make it work.  The current code
specifically exists to avoid trying to load libthread_db for every new
shared library we notice, since that was considered wasteful.  Here's
the original thread about it:

  https://sourceware.org/pipermail/gdb-patches/2011-October/085781.html
  https://pi.simark.ca/gdb-patches/20111005182705.D744E2461D1@ruffy.mtv.corp.google.com/

About the hypothetical scenario for glibc 2.34: do you mean that the
pthread infrastructure will directly be in libc.so?  If so, our current
strategy of attempting to load libthread_db only for the main program
or a libpthread* library will indeed not work.  And I suppose that will
also require trying to load libthread_db on every new shared lib...

Simon

Re: [PATCH] nptl: Move stack list variables into _rtld_global

[Florian Weimer via Gdb-patches]

* Simon Marchi:

>>> If we have to deal with this, I guess that GDB should now do things in a
>>> different order: go through the whole library list and load their
>>> symbols.  And then if one of those libraries were libpthread, try to
>>> initialize libthread_db.
>> 
>> Initialization of libthread_db should be unconditional.  Programs use
>> TLS data without linking against libpthread.  And glibc 2.34 might not
>> have a separate libpthread at all.
>
> Ok, currently GDB attempts to load libthread_db when noticing the main
> objfile / program (I guess it is needed if the program is statically
> linked to libpthread?) or when seeing a library named libpthread*.

Would it be possible to load libthread_db unconditionally after loading
all shared objects?  Then it is loaded only once.

> About the hypothetical scenario for glibc 2.34: do you mean that the
> pthread infrastructure will directly be in libc.so?  If so, our current
> strategy of attempting to load libthread_db only for the main program
> or a libpthread* library will indeed not work.  And I suppose that will
> also require trying to load libthread_db on every new shared lib...

I think one attempt loading is enough, after all shared objects are
available.  In both the attaching and starting case, libpthread will be
seen by libthread_db if it is there.  I do not think it is necessary to
try loading libpthread_db again for each dlopen.  Maybe you could
restrict that to trigger on libpthread, but then dlopen of libpthread
does not really work today.

Thanks,
Florian

Re: [PATCH] nptl: Move stack list variables into _rtld_global

[Florian Weimer]

* Florian Weimer via Gdb-patches:

> * Simon Marchi:
>
>>>> If we have to deal with this, I guess that GDB should now do things in a
>>>> different order: go through the whole library list and load their
>>>> symbols.  And then if one of those libraries were libpthread, try to
>>>> initialize libthread_db.
>>> 
>>> Initialization of libthread_db should be unconditional.  Programs use
>>> TLS data without linking against libpthread.  And glibc 2.34 might not
>>> have a separate libpthread at all.
>>
>> Ok, currently GDB attempts to load libthread_db when noticing the main
>> objfile / program (I guess it is needed if the program is statically
>> linked to libpthread?) or when seeing a library named libpthread*.
>
> Would it be possible to load libthread_db unconditionally after loading
> all shared objects?  Then it is loaded only once.
>
>> About the hypothetical scenario for glibc 2.34: do you mean that the
>> pthread infrastructure will directly be in libc.so?  If so, our current
>> strategy of attempting to load libthread_db only for the main program
>> or a libpthread* library will indeed not work.  And I suppose that will
>> also require trying to load libthread_db on every new shared lib...
>
> I think one attempt loading is enough, after all shared objects are
> available.  In both the attaching and starting case, libpthread will be
> seen by libthread_db if it is there.  I do not think it is necessary to
> try loading libpthread_db again for each dlopen.  Maybe you could
> restrict that to trigger on libpthread, but then dlopen of libpthread
> does not really work today.

I would appreciate if we could make some progress on this issue.
Please let me know if you need glibc test builds or something in that
area.  Thanks.

Re: [PATCH] nptl: Move stack list variables into _rtld_global

[Simon Marchi via Gdb-patches]

On 2021-03-29 4:26 a.m., Florian Weimer wrote:> * Florian Weimer via Gdb-patches:
> 
>> * Simon Marchi:
>>
>>>>> If we have to deal with this, I guess that GDB should now do things in a
>>>>> different order: go through the whole library list and load their
>>>>> symbols.  And then if one of those libraries were libpthread, try to
>>>>> initialize libthread_db.
>>>>
>>>> Initialization of libthread_db should be unconditional.  Programs use
>>>> TLS data without linking against libpthread.  And glibc 2.34 might not
>>>> have a separate libpthread at all.
>>>
>>> Ok, currently GDB attempts to load libthread_db when noticing the main
>>> objfile / program (I guess it is needed if the program is statically
>>> linked to libpthread?) or when seeing a library named libpthread*.
>>
>> Would it be possible to load libthread_db unconditionally after loading
>> all shared objects?  Then it is loaded only once.
>>
>>> About the hypothetical scenario for glibc 2.34: do you mean that the
>>> pthread infrastructure will directly be in libc.so?  If so, our current
>>> strategy of attempting to load libthread_db only for the main program
>>> or a libpthread* library will indeed not work.  And I suppose that will
>>> also require trying to load libthread_db on every new shared lib...
>>
>> I think one attempt loading is enough, after all shared objects are
>> available.  In both the attaching and starting case, libpthread will be
>> seen by libthread_db if it is there.  I do not think it is necessary to
>> try loading libpthread_db again for each dlopen.  Maybe you could
>> restrict that to trigger on libpthread, but then dlopen of libpthread
>> does not really work today.
> 
> I would appreciate if we could make some progress on this issue.
> Please let me know if you need glibc test builds or something in that
> area.  Thanks.

Hi Florian,

I'll try to look into it, but I can't promise anything as I have nearly
zero free / personal time for GDB these days.

Simon