Random SIGILL / SIGSEGV on ARMv7 while debugging ?

Random SIGILL / SIGSEGV on ARMv7 while debugging ?

[Antoine Tremblay]

Hi,
   I've been running the testsuite on a Odroid XU4 and I get random 
SIGILL / SIGSEGV signals while running the testsuite.

  It happens around 30% of the time on the same command.

  I can get the following log for example :

(gdb) spawn ../gdbserver/gdbserver --once :2346 
/home/hexa/src/binutils-gdb/build-arm/gdb/testsuite/gdb.trace/change-loc
Process 
/home/hexa/src/binutils-gdb/build-arm/gdb/testsuite/gdb.trace/change-loc 
created; pid = 1326
Listening on port 2346
target remote localhost:2346
Remote debugging using localhost:2346
Reading /lib/ld-linux-armhf.so.3 from remote target...
warning: File transfers from remote targets can be slow. Use "set 
sysroot" to access files locally instead.
Reading /lib/ld-linux-armhf.so.3 from remote target...
Reading symbols from target:/lib/ld-linux-armhf.so.3...Reading 
/lib/ld-2.21.so from remote target...
Reading /lib/.debug/ld-2.21.so from remote target...
(no debugging symbols found)...done.
0xb6fd7a40 in ?? () from target:/lib/ld-linux-armhf.so.3
(gdb) continue
Continuing.
Reading 
/home/hexa/src/binutils-gdb/build-arm/gdb/testsuite/gdb.trace/change-loc-1.sl 
from remote target...
Reading /lib/arm-linux-gnueabihf/libdl.so.2 from remote target...
Reading /lib/arm-linux-gnueabihf/libm.so.6 from remote target...
Reading /lib/arm-linux-gnueabihf/libc.so.6 from remote target...
Reading /lib/arm-linux-gnueabihf/libdl-2.21.so from remote target...
Reading /lib/arm-linux-gnueabihf/.debug/libdl-2.21.so from remote target...
Reading /lib/arm-linux-gnueabihf/libm-2.21.so from remote target...
Reading /lib/arm-linux-gnueabihf/.debug/libm-2.21.so from remote target...
Reading /lib/arm-linux-gnueabihf/libc-2.21.so from remote target...
Reading /lib/arm-linux-gnueabihf/.debug/libc-2.21.so from remote target...

Program received signal SIGILL, Illegal instruction.
0xb6fd99e2 in ?? () from target:/lib/ld-linux-armhf.so.3
(gdb) FAIL: gdb.trace/change-loc.exp: Can't run to main to check for 
trace support

I'm able to reproduce this also with GDB itself using a small program 
like so :

int
func1 (int b)
{
   return b + 2;
}

int
main ()
{
   int c = 1;
   while (c)
     c = func1 (2);
   return c;
}

and running gdb with that script :

set pagination off
start
while 1
   step
   end


I've tested this on kernel 3.10 and 4.2, same results, 4.2 seems a bit 
worse...

I've also tested on a Raspberry Pi2 with kernel 3.18.0 and I can't 
reproduce it.

Has anybody else seen this ?


Regards.
Antoine Tremblay

Re: Random SIGILL / SIGSEGV on ARMv7 while debugging ?

[Yao Qi]

Antoine Tremblay <antoine.tremblay@ericsson.com> writes:

I didn't see any SIGILL/SIGSEGV in my testing.

> Program received signal SIGILL, Illegal instruction.
> 0xb6fd99e2 in ?? () from target:/lib/ld-linux-armhf.so.3
> (gdb) FAIL: gdb.trace/change-loc.exp: Can't run to main to check for
> trace support

Did you see SIGILL with vanilla GDB or GDB with your tracepoint patches?
You can turn on debugging output, like "set debug infrun 1" and "set
debug remote 1", and you may find some clues from them.

-- 
Yao (齐尧)

Re: Random SIGILL / SIGSEGV on ARMv7 while debugging ?

[Antoine Tremblay]

On 11/20/2015 04:11 AM, Yao Qi wrote:
> Antoine Tremblay <antoine.tremblay@ericsson.com> writes:
>
> I didn't see any SIGILL/SIGSEGV in my testing.
>
>> Program received signal SIGILL, Illegal instruction.
>> 0xb6fd99e2 in ?? () from target:/lib/ld-linux-armhf.so.3
>> (gdb) FAIL: gdb.trace/change-loc.exp: Can't run to main to check for
>> trace support
>
> Did you see SIGILL with vanilla GDB or GDB with your tracepoint patches?

I tested both and the problem was present with vanilla too :

gdb --version
GNU gdb (Ubuntu 7.9-1ubuntu1) 7.9


> You can turn on debugging output, like "set debug infrun 1" and "set
> debug remote 1", and you may find some clues from them.

Yes I did that with gdbserver --debug and compared the output from a 
successful run with a failed run and there was no change.

This leads me to suspect the problem is not gdb/gdbserver. Especially 
since the Pi2 with the same gdb/gdbserver does not have the problem.

I'm guessing from your answer you've never seen that in your tests...

I think I'll have to work around it by using an other board, I'm looking 
at the testing the firefly one now : 
http://en.t-firefly.com/en/firenow/firefly_rk3288/

If ever you have another suggestion of a powerful ARMv7 board with eMMC 
storage that you know works well it's quite welcome.

Thank you,
Antoine

Re: Random SIGILL / SIGSEGV on ARMv7 while debugging ?

[Antoine Tremblay]

On 11/20/2015 07:55 AM, Antoine Tremblay wrote:
> On 11/20/2015 04:11 AM, Yao Qi wrote:

>> You can turn on debugging output, like "set debug infrun 1" and "set
>> debug remote 1", and you may find some clues from them.

For reference here is a pastebin of a run with gdb debug on and the 
fault appearing :

http://pastebin.com/nB8gPY2D

Re: Random SIGILL / SIGSEGV on ARMv7 while debugging ?

[Antoine Tremblay]

Antoine Tremblay writes:

> On 11/20/2015 07:55 AM, Antoine Tremblay wrote:
>> On 11/20/2015 04:11 AM, Yao Qi wrote:
>
>>> You can turn on debugging output, like "set debug infrun 1" and "set
>>> debug remote 1", and you may find some clues from them.
>
> For reference here is a pastebin of a run with gdb debug on and the 
> fault appearing :
>
> http://pastebin.com/nB8gPY2D

After some more investigation, and great help from Will Deacon.

It turns out that theses SIGILL are due to a SoC Hardware bug as
described by Will as such :

So the problem is that A15 has 64-byte I-cache lines and A7 has 32-byte
I-cache lines. That means that if the kernel is doing cache maintenance
on the A15, it will issue an invalidation for each 64 bytes of the region
it wants to invalidate. The A7 will then receive these invalidation
messages, but only invalidate 32 bytes for each one, leaving a bunch of
32-byte holes that didn't get invalidated.

This is an SoC hardware bug. The two cores should report the same line
size (as I mentioned, there's a tie-off on the A15 to make it report
32-byte cachelines)."

The tie-off documentation can be found in the Technical Reference Manual
under "Configuration signals" as IMINLN.

I tested the patch below with an Odroid-UX4 and the problem was fixed.
I have not tested yet with the Firefly RK3288, I'm missing a cable to do
it at the moment...

I will send this to the Odroid team...

diff --git a/arch/arm/mm/proc-macros.S b/arch/arm/mm/proc-macros.S
index ee1d805..573db9b 100644
--- a/arch/arm/mm/proc-macros.S
+++ b/arch/arm/mm/proc-macros.S
@@ -82,10 +82,7 @@
  * on ARMv7.
  */
        .macro  icache_line_size, reg, tmp
-       mrc     p15, 0, \tmp, c0, c0, 1         @ read ctr
-       and     \tmp, \tmp, #0xf                @ cache line size encoding
-       mov     \reg, #4                        @ bytes per word
-       mov     \reg, \reg, lsl \tmp            @ actual cache line size
+       mov     \reg, #32                       @ hack
        .endm