Re: [BuildBot] Notifications disabled for Debian-s390x-* and Fedora-ppc64*-* builders

David Edelsohn
2017-12-15T18:55:46Z

On Fri, Dec 15, 2017 at 12:29 PM, David Edelsohn <dje.gcc@gmail.com> wrote:
> On Fri, Dec 15, 2017 at 11:20 AM, Pedro Alves <palves@redhat.com> wrote:
>> On 12/15/2017 03:53 PM, David Edelsohn wrote:
>>> On Fri, Dec 15, 2017 at 10:42 AM, Pedro Alves <palves@redhat.com> wrote:
>>>> On 12/15/2017 03:06 PM, David Edelsohn wrote:
>>>>
>>>>> Third, the testsuite summaries that no one from the GDB community
>>>>> monitored show that the testsuite runtime jumped from a relatively
>>>>> short amount of time to over 9 hours for each run, which points to a
>>>>> newly introduced problem in GDB or in the testsuite (timeouts?).
>>>>
>>>> That may well be. Can you point at some representative builds,
>>>> before/after the jump?
>>>
>>> The testsuite runs for 6 minutes on RHEL7 s390x buildslave and 9 hours
>>> on Debian Jessie s390x buildslave.
>>
>> Those are separate machines. I'd like to see the jump on the same
>> machine, so we can maybe pinpoint what caused it.
>>
>> I was really asking for URLs. Here looks like there's some:
>>
>> https://gdb-build.sergiodj.net/builders/Debian-s390x-native-gdbserver-m64
>>
>> Here, for example:
>>
>> https://gdb-build.sergiodj.net/builders/Debian-s390x-native-gdbserver-m64/builds/4351
>>
>> "test gdb tested GDB failed (9 hrs, 2 mins, 56 secs)"
>>
>> That's definitely too long.
>>
>> I downloaded the gdb.log file, and did:
>>
>> $ grep FAIL gdb.log | grep timeout | sed 's/.exp.*/.exp/g' | sort | uniq -c | sort -n
>> 1 FAIL: gdb.base/watch-cond.exp
>> 1 FAIL: gdb.multi/watchpoint-multi-exit.exp
>> 1 FAIL: gdb.threads/interrupted-hand-call.exp
>> 1 FAIL: gdb.threads/thread-unwindonsignal.exp
>> 2 FAIL: gdb.base/value-double-free.exp
>> 3 FAIL: gdb.mi/mi-async.exp
>> 3 FAIL: gdb.threads/process-dies-while-detaching.exp
>> 4 FAIL: gdb.base/pr11022.exp
>> 10 FAIL: gdb.base/watch-bitfields.exp
>> 15 FAIL: gdb.base/watchpoints.exp
>> 20 FAIL: gdb.threads/interrupt-while-step-over.exp
>> 32 FAIL: gdb.threads/watchpoint-fork.exp
>> 45 FAIL: gdb.threads/step-over-trips-on-watchpoint.exp
>> 46 FAIL: gdb.base/display.exp
>> 51 FAIL: gdb.base/watchpoint.exp
>>
>> Not _that_ many. Could they explain the long time? I suspect not.
>>
>> We see this:
>>
>> $ grep "Test run by" gdb.log | head -n 3
>> Test run by dje on Tue Nov 21 03:23:01 2017
>> Test run by dje on Tue Nov 21 03:23:01 2017
>> Test run by dje on Tue Nov 21 03:23:01 2017
>>
>> $ grep "Test run by" gdb.log | tail -n 3
>> Test run by dje on Tue Nov 21 03:29:54 2017
>> Test run by dje on Tue Nov 21 03:29:54 2017
>> Test run by dje on Tue Nov 21 03:29:54 2017
>>
>> So most of the testsuite actually ran for 7 minutes. And then
>> something hung for 9 hours? I have no idea how that
>> could happen from the existing logs. The tail end of the log has:
>>
>> ~~~
>> FAIL: gdb.base/watchpoint.exp: delete all breakpoints in delete_breakpoints (timeout)
>> ERROR: breakpoints not deleted
>> ERROR: breakpoints not deleted
>>
>> command timed out: 1200 seconds without output running ['make', '-k', 'check', 'RUNTESTFLAGS=--target_board native-gdbserver', '-j8', 'FORCE_PARALLEL=1'], attempting to kill
>> process killed by signal 9
>> program finished with exit code -1
>> elapsedTime=32576.210392
>> ~~~
>>
>> I don't understand how 7 minutes plus 1200 seconds (~20min)
>> resulted in "elapsedTime=32576.210392" (~9h). Maybe that number
>> isn't to be trusted.
>>
>> Anyway, I'm sorry, but I really don't have the time to be
>> looking at this. Someone with the motivation and access to
>> the machine could try running the testsuite manually,
>> for example, see how long that takes, and where the hang is.
>
> I will try reverting to an older version of DejaGNU framework.

Older DejaGNU does not seem to have an effect. All of the processes
are stuck in "gdb.threads/process-dies-while-handling-bp"

Thanks, David