From mboxrd@z Thu Jan 1 00:00:00 1970 From: shehabyomn@gmail.com (Shehab Elsayed) Date: Fri, 26 Apr 2019 16:44:16 -0400 Subject: [lttng-dev] Tracepoints for pthreads conditional variables In-Reply-To: References: <20180501143953.GA26829@joraj-alpa> Message-ID: I came across something some time ago that I think might solve this problem. It goes back to making sure that the version of the synchronization function called from the wrapper is the same that would be called without the wrapper. To specify the version the following instruction can be used: ....... = dlvsym(RTLD_NEXT, "function_name", "GLIBC_2.3.2") To know which version is normally used, you can use the following: objdump -t | grep where is the name of the program to be run, and is the name of the function to be called from the wrapper. For more information, check this link: http://blog.fesnel.com/blog/2009/08/25/preloading-with-multiple-symbol-versions/ I hope this helps. On Thu, Jul 5, 2018 at 5:36 PM Shehab Elsayed wrote: > Any suggestions where I should investigate to fix this bug? Also, please > let me know whether the git repo I shared is sufficient. > > Thanks, > Shehab > > Shehab Y. Elsayed, MSc. > PhD Student > The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering > University of Toronto > E-mail: shehabyomn at gmail.com > > > On Tue, May 29, 2018 at 10:36 AM, Shehab Elsayed > wrote: > >> I was wondering if you had a chance to look over this issue. Thanks! >> >> Shehab Y. Elsayed, MSc. >> PhD Student >> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >> University of Toronto >> E-mail: shehabyomn at gmail.com >> >> >> On Thu, May 3, 2018 at 4:37 PM, Shehab Elsayed >> wrote: >> >>> Thanks for the reply. Here is a link to the repo >>> https://github.com/ShehabElsayed/LTTng_debugging.git >>> >>> It contains the LTTng source code I am using and one of the benchmarks >>> that is causing problems. >>> >>> Shehab Y. Elsayed, MSc. >>> PhD Student >>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering >>> University of Toronto >>> E-mail: shehabyomn at gmail.com >>> >>> >>> On Tue, May 1, 2018 at 10:39 AM, Jonathan Rajotte-Julien < >>> jonathan.rajotte-julien at efficios.com> wrote: >>> >>>> Hi Shehab, >>>> >>>> Please provide a link to a lttng-ust git tree or an actual patch (git >>>> format-patch). >>>> This will help any person who might be interested in helping you >>>> including >>>> myself. >>>> >>>> Cheers >>>> >>>> On Tue, May 01, 2018 at 09:31:52AM -0400, Shehab Elsayed wrote: >>>> > Hi, >>>> > >>>> > I am trying to create wrappers for pthreads conditional variables >>>> functions >>>> > (wait, signal and broadcast) to insert tracepoints. I followed the >>>> same >>>> > approach done for the mutex functions already provided with lttng, >>>> however >>>> > I am running into some problems. Mainly the applications seem to get >>>> stuck >>>> > when the conditional variable wrappers are enabled. >>>> > >>>> > Here is my implementation for the wrapper functions: >>>> > int pthread_cond_wait(pthread_cond_t *condition, pthread_mutex_t >>>> > *mutex) >>>> > >>>> > { >>>> > >>>> > static int (*cond_wait)(pthread_cond_t *, pthread_mutex_t >>>> > *); >>>> > int retval; >>>> > >>>> > >>>> > >>>> > if (!cond_wait) >>>> > { >>>> > >>>> > cond_wait = dlsym(RTLD_NEXT, "pthread_cond_wait"); >>>> > >>>> > if (!cond_wait) >>>> > { >>>> > >>>> > if (thread_in_trace) >>>> > { >>>> > >>>> > >>>> > abort(); >>>> > >>>> > >>>> > } >>>> > >>>> > fprintf(stderr, "unable to initialize pthread wrapper >>>> > library.\n"); >>>> > return EINVAL; >>>> > >>>> > >>>> > } >>>> > >>>> > >>>> > } >>>> > >>>> > if (thread_in_trace) >>>> > { >>>> > >>>> > return cond_wait(condition, mutex); >>>> > >>>> > >>>> > } >>>> > >>>> > >>>> > >>>> > thread_in_trace = >>>> > 1; >>>> > >>>> > tracepoint(lttng_ust_pthread, pthread_cond_wait_begin, condition, >>>> > mutex, >>>> > >>>> > LTTNG_UST_CALLER_IP()); >>>> > >>>> > retval = cond_wait(condition, mutex); >>>> > >>>> > tracepoint(lttng_ust_pthread, pthread_cond_wait_end, condition, >>>> > mutex, >>>> > >>>> > LTTNG_UST_CALLER_IP()); >>>> > >>>> > thread_in_trace = >>>> > 0; >>>> > >>>> > return retval; >>>> > >>>> > } >>>> > >>>> > >>>> > >>>> > int pthread_cond_signal(pthread_cond_t >>>> > *condition) >>>> > >>>> > { >>>> > >>>> > static int (*cond_signal)(pthread_cond_t >>>> > *); >>>> > >>>> > int retval; >>>> > >>>> > >>>> > >>>> > if (!cond_signal) >>>> > { >>>> > >>>> > cond_signal = dlsym(RTLD_NEXT, "pthread_cond_signal"); >>>> > >>>> > if (!cond_signal) >>>> > { >>>> > >>>> > if (thread_in_trace) >>>> > { >>>> > >>>> > >>>> > abort(); >>>> > >>>> > >>>> > } >>>> > >>>> > fprintf(stderr, "unable to initialize pthread wrapper >>>> > library.\n"); >>>> > return EINVAL; >>>> > >>>> > >>>> > } >>>> > >>>> > >>>> > } >>>> > >>>> > if (thread_in_trace) >>>> > { >>>> > >>>> > return cond_signal(condition); >>>> > >>>> > >>>> > } >>>> > >>>> > >>>> > >>>> > thread_in_trace = >>>> > 1; >>>> > >>>> > tracepoint(lttng_ust_pthread, pthread_cond_signal_begin, >>>> > condition, >>>> > LTTNG_UST_CALLER_IP()); >>>> > >>>> > retval = cond_signal(condition); >>>> > >>>> > tracepoint(lttng_ust_pthread, pthread_cond_signal_end, >>>> > condition, >>>> > LTTNG_UST_CALLER_IP()); >>>> > >>>> > thread_in_trace = >>>> > 0; >>>> > >>>> > return retval; >>>> > >>>> > } >>>> > >>>> > >>>> > >>>> > int pthread_cond_broadcast(pthread_cond_t >>>> > *condition) >>>> > >>>> > { >>>> > >>>> > static int (*cond_broadcast)(pthread_cond_t >>>> > *); >>>> > int retval; >>>> > >>>> > >>>> > >>>> > if (!cond_broadcast) >>>> > { >>>> > >>>> > cond_broadcast = dlsym(RTLD_NEXT, "pthread_cond_broadcast"); >>>> > >>>> > if (!cond_broadcast) >>>> > { >>>> > >>>> > if (thread_in_trace) >>>> > { >>>> > >>>> > >>>> > abort(); >>>> > >>>> > >>>> > } >>>> > >>>> > fprintf(stderr, "unable to initialize pthread wrapper >>>> > library.\n"); >>>> > return EINVAL; >>>> > >>>> > >>>> > } >>>> > >>>> > >>>> > } >>>> > >>>> > if (thread_in_trace) >>>> > { >>>> > >>>> > return cond_broadcast(condition); >>>> > >>>> > >>>> > } >>>> > >>>> > >>>> > >>>> > thread_in_trace = >>>> > 1; >>>> > >>>> > tracepoint(lttng_ust_pthread, pthread_cond_broadcast_begin, >>>> > condition, >>>> > LTTNG_UST_CALLER_IP()); >>>> > >>>> > retval = cond_broadcast(condition); >>>> > >>>> > tracepoint(lttng_ust_pthread, pthread_cond_broadcast_end, >>>> > condition, >>>> > LTTNG_UST_CALLER_IP()); >>>> > >>>> > thread_in_trace = >>>> > 0; >>>> > >>>> > return retval; >>>> > >>>> > } >>>> > >>>> > Things I have tried: >>>> > 1- Comment out pthread_cond_wait function --> segmentation fault >>>> > 2- Comment out pthread_cond_signal and _broadcast --> stuck (I can >>>> see the >>>> > program is still running but nor forward progress is being made) >>>> compared >>>> > to the case when the wrappers are not preloaded. >>>> > 3- Comment out all tracepoint related code (basically the wrapper >>>> just call >>>> > the corresponding pthreads function) --> same results as points 1 and >>>> 2. >>>> > >>>> > Any idea what might be causing this or how I could debug this problem? >>>> > >>>> > Thank you very much in advance. >>>> > >>>> > Best Regards, >>>> > Shehab >>>> >>>> > int pthread_cond_wait(pthread_cond_t *condition, pthread_mutex_t >>>> *mutex) >>>> > { >>>> >>>> >>>> >>>> > static int (*cond_wait)(pthread_cond_t *, pthread_mutex_t *); >>>> >>>> > int retval; >>>> >>>> > >>>> >>>> > if (!cond_wait) { >>>> >>>> > cond_wait = dlsym(RTLD_NEXT, "pthread_cond_wait"); >>>> >>>> > if (!cond_wait) { >>>> >>>> > if (thread_in_trace) { >>>> >>>> > abort(); >>>> >>>> > } >>>> >>>> > fprintf(stderr, "unable to initialize pthread wrapper >>>> library.\n"); >>>> > return EINVAL; >>>> >>>> > } >>>> >>>> > } >>>> >>>> > if (thread_in_trace) { >>>> >>>> > return cond_wait(condition, mutex); >>>> >>>> > } >>>> >>>> > >>>> >>>> > thread_in_trace = 1; >>>> >>>> > tracepoint(lttng_ust_pthread, pthread_cond_wait_begin, condition, >>>> mutex, >>>> > LTTNG_UST_CALLER_IP()); >>>> >>>> > retval = cond_wait(condition, mutex); >>>> >>>> > tracepoint(lttng_ust_pthread, pthread_cond_wait_end, condition, >>>> mutex, >>>> > LTTNG_UST_CALLER_IP()); >>>> >>>> > thread_in_trace = 0; >>>> >>>> > return retval; >>>> >>>> > } >>>> >>>> > >>>> >>>> > int pthread_cond_signal(pthread_cond_t *condition) >>>> >>>> > { >>>> >>>> > static int (*cond_signal)(pthread_cond_t *); >>>> >>>> > int retval; >>>> >>>> > >>>> >>>> > if (!cond_signal) { >>>> >>>> > cond_signal = dlsym(RTLD_NEXT, "pthread_cond_signal"); >>>> >>>> > if (!cond_signal) { >>>> >>>> > if (thread_in_trace) { >>>> >>>> > abort(); >>>> >>>> > } >>>> >>>> > fprintf(stderr, "unable to initialize pthread wrapper >>>> library.\n"); >>>> > return EINVAL; >>>> >>>> > } >>>> >>>> > } >>>> >>>> > if (thread_in_trace) { >>>> >>>> > return cond_signal(condition); >>>> >>>> > } >>>> >>>> > >>>> >>>> > thread_in_trace = 1; >>>> >>>> > tracepoint(lttng_ust_pthread, pthread_cond_signal_begin, >>>> condition, >>>> > LTTNG_UST_CALLER_IP()); >>>> >>>> > retval = cond_signal(condition); >>>> >>>> > tracepoint(lttng_ust_pthread, pthread_cond_signal_end, condition, >>>> >>>> > LTTNG_UST_CALLER_IP()); >>>> >>>> > thread_in_trace = 0; >>>> >>>> > return retval; >>>> >>>> > } >>>> >>>> > >>>> >>>> > int pthread_cond_broadcast(pthread_cond_t *condition) >>>> >>>> > { >>>> >>>> > static int (*cond_broadcast)(pthread_cond_t *); >>>> >>>> > int retval; >>>> >>>> > >>>> >>>> > if (!cond_broadcast) { >>>> >>>> > cond_broadcast = dlsym(RTLD_NEXT, "pthread_cond_broadcast"); >>>> >>>> > if (!cond_broadcast) { >>>> >>>> > if (thread_in_trace) { >>>> >>>> > abort(); >>>> >>>> > } >>>> >>>> > fprintf(stderr, "unable to initialize pthread wrapper >>>> library.\n"); >>>> > return EINVAL; >>>> >>>> > } >>>> >>>> > } >>>> >>>> > if (thread_in_trace) { >>>> >>>> > return cond_broadcast(condition); >>>> >>>> > } >>>> >>>> > >>>> >>>> > thread_in_trace = 1; >>>> >>>> > tracepoint(lttng_ust_pthread, pthread_cond_broadcast_begin, >>>> condition, >>>> > LTTNG_UST_CALLER_IP()); >>>> >>>> > retval = cond_broadcast(condition); >>>> >>>> > tracepoint(lttng_ust_pthread, pthread_cond_broadcast_end, >>>> condition, >>>> > LTTNG_UST_CALLER_IP()); >>>> >>>> > thread_in_trace = 0; >>>> >>>> > return retval; >>>> >>>> > } >>>> >>>> > _______________________________________________ >>>> > lttng-dev mailing list >>>> > lttng-dev at lists.lttng.org >>>> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >>>> >>>> >>>> -- >>>> Jonathan Rajotte-Julien >>>> EfficiOS >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: