gdb.Value returning a string of length 1 (linux lx-version bug)

gdb.Value returning a string of length 1 (linux lx-version bug)

[Kieran Bingham]

Hi Jan,

Me and Pete have just been looking into the gdb.Value object bug where
the lx-version command  returns a string of length 1.

As I recall at FOSDEM, you were also hitting the bug.
Out of interest, what compiler version are you using?

We've made some head way, but we are suspicious that compiler versions
may affect the issue ! :(

The crux of the issue, is that the valpy_string() calls c_get_string()
which in turn calls get_discrete_bounds().

On my broken version, get_discrete_bounds, is setting both lowp, and
highp to 0x0; where as on Peters working version, his highp gets set
correctly. (there is a +1 added later which results in the string
producing a strlen of 1)

I'm running gcc 5.2.1-22ubuntu2, whereas Pete is running gcc 4.8.4.

Pete's version of his compiled binutils-gdb always seems to function
correctly, where as I hit the bug - *occasionally*

Of course, since I have compiled with -g3 -O0 -fsanitize=undefined I
can't reproduce the issue, and now even removing the optimise levels I
haven't been able to reproduce.

We have a Heisenbug :(

Has anyone else on the GDB Mailinglist experienced any intermittent
errors with gdb.Value strings from python?
--
Kieran

Re: gdb.Value returning a string of length 1 (linux lx-version bug)

[Jan Kiszka]

Hi Kieran,

On 2016-03-22 18:54, Kieran Bingham wrote:
> Hi Jan,
> 
> Me and Pete have just been looking into the gdb.Value object bug where
> the lx-version command  returns a string of length 1.
> 
> As I recall at FOSDEM, you were also hitting the bug.
> Out of interest, what compiler version are you using?

gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064]

> 
> We've made some head way, but we are suspicious that compiler versions
> may affect the issue ! :(
> 
> The crux of the issue, is that the valpy_string() calls c_get_string()
> which in turn calls get_discrete_bounds().
> 
> On my broken version, get_discrete_bounds, is setting both lowp, and
> highp to 0x0; where as on Peters working version, his highp gets set
> correctly. (there is a +1 added later which results in the string
> producing a strlen of 1)
> 
> I'm running gcc 5.2.1-22ubuntu2, whereas Pete is running gcc 4.8.4.
> 
> Pete's version of his compiled binutils-gdb always seems to function
> correctly, where as I hit the bug - *occasionally*
> 
> Of course, since I have compiled with -g3 -O0 -fsanitize=undefined I
> can't reproduce the issue, and now even removing the optimise levels I
> haven't been able to reproduce.
> 
> We have a Heisenbug :(

Maybe wait for someone seeing this more reproducibly? :-/

Jan

> 
> Has anyone else on the GDB Mailinglist experienced any intermittent
> errors with gdb.Value strings from python?
> --
> Kieran
> 

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: gdb.Value returning a string of length 1 (linux lx-version bug)

[Kieran Bingham]

On 22/03/16 17:56, Jan Kiszka wrote:
> Hi Kieran,
> 
> On 2016-03-22 18:54, Kieran Bingham wrote:
>> Hi Jan,
>>
>> Me and Pete have just been looking into the gdb.Value object bug where
>> the lx-version command  returns a string of length 1.
>>
>> As I recall at FOSDEM, you were also hitting the bug.
>> Out of interest, what compiler version are you using?
> 
> gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064]

Thanks, well that means it's unlikely to be any optimiser changes, as
your compiler is earlier than Petes.

>>
>> We've made some head way, but we are suspicious that compiler versions
>> may affect the issue ! :(
>>
>> The crux of the issue, is that the valpy_string() calls c_get_string()
>> which in turn calls get_discrete_bounds().
>>
>> On my broken version, get_discrete_bounds, is setting both lowp, and
>> highp to 0x0; where as on Peters working version, his highp gets set
>> correctly. (there is a +1 added later which results in the string
>> producing a strlen of 1)
>>
>> I'm running gcc 5.2.1-22ubuntu2, whereas Pete is running gcc 4.8.4.
>>
>> Pete's version of his compiled binutils-gdb always seems to function
>> correctly, where as I hit the bug - *occasionally*
>>
>> Of course, since I have compiled with -g3 -O0 -fsanitize=undefined I
>> can't reproduce the issue, and now even removing the optimise levels I
>> haven't been able to reproduce.
>>
>> We have a Heisenbug :(
> 
> Maybe wait for someone seeing this more reproducibly? :-/

I think I may have found a way to reproduce reliably!

By running some commands, and ensured the issue occurred, and then
reducing the command set down until only one remained, I've narrowed
down the cause a little.

I have discovered that the issue only appears if you have run a
'backtrace (bt)' *before* calling lx-version.

(gdb) bt
  <bt output omitted>
(gdb) lx-version
L(gdb) ## Note the single char output on the left


Now - having determined how to reliably reproduce, I have re-enabled
-fsanitize=undefined :

(gdb) bt
/home/linuxembedded/linaro/lkd/openst-lkd/sources/binutils-gdb/gdb/dwarf2-frame.c:254:3:
runtime error: null pointer passed as argument 2, which is declared to
never be null
#0  0xc0222c68 in cpu_v7_do_idle ()
    at
/home/linuxembedded/linaro/lkd/openst/sources/linux/arch/arm/mm/proc-v7.S:74
#1  0xc0210728 in arch_cpu_idle ()
    at
/home/linuxembedded/linaro/lkd/openst/sources/linux/arch/arm/kernel/process.c:72
#2  0xc027de78 in cpu_startup_entry ()
    at
/home/linuxembedded/linaro/lkd/openst/sources/linux/kernel/sched/idle.c:150
#3  0xc027de78 in cpu_startup_entry ()
    at
/home/linuxembedded/linaro/lkd/openst/sources/linux/kernel/sched/idle.c:246
#4  0xc027de78 in cpu_startup_entry (state=<optimized out>)
    at
/home/linuxembedded/linaro/lkd/openst/sources/linux/kernel/sched/idle.c:294
#5  0xc09e0160 in rest_init () at
/home/linuxembedded/linaro/lkd/openst/sources/linux/init/main.c:412
#6  0xc0da2ca8 in start_kernel ()
    at /home/linuxembedded/linaro/lkd/openst/sources/linux/init/main.c:683
#7  0x8020807c in  ()
(gdb) lx-version
L(gdb)


UBSAN to the rescue!

I'll see if I can determine the cause / effect of the Null pointer, and
that is likely worth submitting as a dedicated GDB bug.
--
Kieran

> Jan
> 
>>
>> Has anyone else on the GDB Mailinglist experienced any intermittent
>> errors with gdb.Value strings from python?
>> --
>> Kieran
>>
> 

Re: gdb.Value returning a string of length 1 (linux lx-version bug)

[Kieran Bingham]

On 23/03/16 08:50, Kieran Bingham wrote:
> On 22/03/16 17:56, Jan Kiszka wrote:
>> Hi Kieran,
>>
>> On 2016-03-22 18:54, Kieran Bingham wrote:
>>> Hi Jan,
>>>
>>> Me and Pete have just been looking into the gdb.Value object bug where
>>> the lx-version command  returns a string of length 1.
>>>
>>> As I recall at FOSDEM, you were also hitting the bug.
>>> Out of interest, what compiler version are you using?
>>
>> gcc (SUSE Linux) 4.8.3 20140627 [gcc-4_8-branch revision 212064]
> 
> Thanks, well that means it's unlikely to be any optimiser changes, as
> your compiler is earlier than Petes.
> 
>>>
>>> We've made some head way, but we are suspicious that compiler versions
>>> may affect the issue ! :(
>>>
>>> The crux of the issue, is that the valpy_string() calls c_get_string()
>>> which in turn calls get_discrete_bounds().
>>>
>>> On my broken version, get_discrete_bounds, is setting both lowp, and
>>> highp to 0x0; where as on Peters working version, his highp gets set
>>> correctly. (there is a +1 added later which results in the string
>>> producing a strlen of 1)
>>>
>>> I'm running gcc 5.2.1-22ubuntu2, whereas Pete is running gcc 4.8.4.
>>>
>>> Pete's version of his compiled binutils-gdb always seems to function
>>> correctly, where as I hit the bug - *occasionally*
>>>
>>> Of course, since I have compiled with -g3 -O0 -fsanitize=undefined I
>>> can't reproduce the issue, and now even removing the optimise levels I
>>> haven't been able to reproduce.
>>>
>>> We have a Heisenbug :(
>>
>> Maybe wait for someone seeing this more reproducibly? :-/
> 
> I think I may have found a way to reproduce reliably!
> 
> By running some commands, and ensured the issue occurred, and then
> reducing the command set down until only one remained, I've narrowed
> down the cause a little.
> 
> I have discovered that the issue only appears if you have run a
> 'backtrace (bt)' *before* calling lx-version.
> 
> (gdb) bt
>   <bt output omitted>
> (gdb) lx-version
> L(gdb) ## Note the single char output on the left
> 
> 
> Now - having determined how to reliably reproduce, I have re-enabled
> -fsanitize=undefined :
> 
> (gdb) bt
> /home/linuxembedded/linaro/lkd/openst-lkd/sources/binutils-gdb/gdb/dwarf2-frame.c:254:3:
> runtime error: null pointer passed as argument 2, which is declared to
> never be null

This warning comes from a memcpy on a register struct in
dwarf2_frame_state_copy_regs(), however, the code correctly identifies
that there are 0 registers, and so tries to copy zero bytes from a NULL
pointer.

So I don't think this particular warning is causing the next fault - but
there is still some strong correlation between the bt command and the
failing string object. Peter has also tested the issue and can reliably
reproduce by running 'bt' before 'lx-version'

--
Kieran

> #0  0xc0222c68 in cpu_v7_do_idle ()
>     at
> /home/linuxembedded/linaro/lkd/openst/sources/linux/arch/arm/mm/proc-v7.S:74
> #1  0xc0210728 in arch_cpu_idle ()
>     at
> /home/linuxembedded/linaro/lkd/openst/sources/linux/arch/arm/kernel/process.c:72
> #2  0xc027de78 in cpu_startup_entry ()
>     at
> /home/linuxembedded/linaro/lkd/openst/sources/linux/kernel/sched/idle.c:150
> #3  0xc027de78 in cpu_startup_entry ()
>     at
> /home/linuxembedded/linaro/lkd/openst/sources/linux/kernel/sched/idle.c:246
> #4  0xc027de78 in cpu_startup_entry (state=<optimized out>)
>     at
> /home/linuxembedded/linaro/lkd/openst/sources/linux/kernel/sched/idle.c:294
> #5  0xc09e0160 in rest_init () at
> /home/linuxembedded/linaro/lkd/openst/sources/linux/init/main.c:412
> #6  0xc0da2ca8 in start_kernel ()
>     at /home/linuxembedded/linaro/lkd/openst/sources/linux/init/main.c:683
> #7  0x8020807c in  ()
> (gdb) lx-version
> L(gdb)
> 
> 
> UBSAN to the rescue!
> 
> I'll see if I can determine the cause / effect of the Null pointer, and
> that is likely worth submitting as a dedicated GDB bug.
> --
> Kieran
> 
>> Jan
>>
>>>
>>> Has anyone else on the GDB Mailinglist experienced any intermittent
>>> errors with gdb.Value strings from python?
>>> --
>>> Kieran
>>>
>>