bigcore.exp on 64-bit systems

bigcore.exp on 64-bit systems

[Eli Zaretskii]

I'm trying to run the testsuite on a 64-bit machine that identifies
itself thusly:

    Linux fencepost 2.6.16.29-xen #1 SMP Wed Dec 6 07:32:36 EST 2006 x86_64 GNU/Linux

On this machine, bigcore.exp is a pest: it runs for a very long time,
produces a humongous core file (so large that I expect the sysadmins
to send me angry emails), and eventually fails (or so it seems).
Here's what gdb.sum says about it:

    Running ../.././gdb/testsuite/gdb.base/bigcore.exp ...
    PASS: gdb.base/bigcore.exp: set print sevenbit-strings; bigcore
    PASS: gdb.base/bigcore.exp: set width 0; bigcore
    PASS: gdb.base/bigcore.exp: tbreak 264
    PASS: gdb.base/bigcore.exp: continue
    PASS: gdb.base/bigcore.exp: next
    PASS: gdb.base/bigcore.exp: extract next heap (stop at 50)
    PASS: gdb.base/bigcore.exp: extract prev heap (stop at 50)
    PASS: gdb.base/bigcore.exp: save heap size
    PASS: gdb.base/bigcore.exp: grab pid
    FAIL: gdb.base/bigcore.exp: signal SIGABRT (timeout)
    FAIL: gdb.base/bigcore.exp: check core size (timeout)
    UNTESTED: gdb.base/bigcore.exp: check core size (system does not support large corefiles)

Is this normal?  What is the meaning of SIGABRT (timeout), and what,
if anything, should I do about the last line?

TIA

Re: bigcore.exp on 64-bit systems

[Joel Brobecker]

> On this machine, bigcore.exp is a pest:

I thought we had decided not to run this testcase on any 64bit platform?

-- 
Joel

Re: bigcore.exp on 64-bit systems

[Mark Kettenis]

> Date: Sat, 13 Jan 2007 16:09:37 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
>     Running ../.././gdb/testsuite/gdb.base/bigcore.exp ...
>     PASS: gdb.base/bigcore.exp: set print sevenbit-strings; bigcore
>     PASS: gdb.base/bigcore.exp: set width 0; bigcore
>     PASS: gdb.base/bigcore.exp: tbreak 264
>     PASS: gdb.base/bigcore.exp: continue
>     PASS: gdb.base/bigcore.exp: next
>     PASS: gdb.base/bigcore.exp: extract next heap (stop at 50)
>     PASS: gdb.base/bigcore.exp: extract prev heap (stop at 50)
>     PASS: gdb.base/bigcore.exp: save heap size
>     PASS: gdb.base/bigcore.exp: grab pid
>     FAIL: gdb.base/bigcore.exp: signal SIGABRT (timeout)
>     FAIL: gdb.base/bigcore.exp: check core size (timeout)
>     UNTESTED: gdb.base/bigcore.exp: check core size (system does not support large corefiles)
> 
> Is this normal?  What is the meaning of SIGABRT (timeout), and what,
> if anything, should I do about the last line?

It's not normal.  On a modern Linux kernel this test should pass, and
2.6.16 certainly qualifies as modern.  The idea is that the test will
create a sparse core file, that will take up almost no disk space.
But the fact that it takes more than 600 seconds to dump the core file
(that's the cause of the SIGABRT timeout), suggests that it isn't
doing that.

Are you by any chance running this on an NFS filesystem?

Mark

Re: bigcore.exp on 64-bit systems

[Eli Zaretskii]

> Date: Sat, 13 Jan 2007 15:50:32 +0100 (CET)
> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> CC: gdb@sourceware.org
> 
> Are you by any chance running this on an NFS filesystem?

No:

    eliz@fencepost:~/gdb-6.6.50.20070113$ df -hT .
    Filesystem    Type    Size  Used Avail Use% Mounted on
    /dev/sdb1      xfs    1.2T   90G  1.1T   8% /srv/data

Re: bigcore.exp on 64-bit systems

[Ulrich Weigand]

Mark Kettenis wrote:

> 2.6.16 certainly qualifies as modern.  The idea is that the test will
> create a sparse core file, that will take up almost no disk space.
> But the fact that it takes more than 600 seconds to dump the core file
> (that's the cause of the SIGABRT timeout), suggests that it isn't
> doing that.

The problem I've seen on certain Linux kernel versions is that while
generating the core file, the kernel fully populates the page tables
spanning the (zero) heap allocated by the bigcore test.

This means that while it doesn't allocate memory (or disk space) for
the heap itself, it will allocate non-pageable memory for the page
tables, and if the heap gets to several TB, that means several GB
of page tables.

To circumvent this, I generally set a reasonable ulimit -v limit
before running the test suite on a 64-bit system.

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  GNU Toolchain for Linux on System z and Cell BE
  Ulrich.Weigand@de.ibm.com

Re: bigcore.exp on 64-bit systems

[Daniel Jacobowitz]

On Sat, Jan 13, 2007 at 07:31:35PM +0100, Ulrich Weigand wrote:
> Mark Kettenis wrote:
> 
> > 2.6.16 certainly qualifies as modern.  The idea is that the test will
> > create a sparse core file, that will take up almost no disk space.
> > But the fact that it takes more than 600 seconds to dump the core file
> > (that's the cause of the SIGABRT timeout), suggests that it isn't
> > doing that.
> 
> The problem I've seen on certain Linux kernel versions is that while
> generating the core file, the kernel fully populates the page tables
> spanning the (zero) heap allocated by the bigcore test.
> 
> This means that while it doesn't allocate memory (or disk space) for
> the heap itself, it will allocate non-pageable memory for the page
> tables, and if the heap gets to several TB, that means several GB
> of page tables.

Hmm... very interesting.  I see that when I run it, it works itself up
to almost 1GB resident memory usage.  If you don't happen to have that
much free memory available without swapping other stuff out, then I'd
believe that the test could take longer than the timeout.  Fencepost
has enough RAM, but only barely, and is running plenty of other things.

The test was originally written for >2GB corefiles on 32-bit systems,
specifically to test signed / unsigned file offset bugs.  Testing that
on 64-bit systems would need a > 8EB (exabyte) core file.  That just
Doesn't Happen on today's systems; my 64-bit Linux system manages
to allocate "only" 1TB of RAM.

Ergo I don't think what it's doing right now is useful.  Should we
limit its allocation to, perhaps, 4GB (2**32)?

For comparison purposes, an 8GB limit on my system completes the core
dump in 1.6 seconds.  No limit yields a 1TB dump, and takes 97 seconds. 
I'd be happy to have that time back :-)

-- 
Daniel Jacobowitz
CodeSourcery

Re: bigcore.exp on 64-bit systems

[Mark Kettenis]

> Date: Sat, 13 Jan 2007 14:22:52 -0500
> From: Daniel Jacobowitz <drow@false.org>
>
> Ergo I don't think what it's doing right now is useful.  Should we
> limit its allocation to, perhaps, 4GB (2**32)?

I'd have no problem with that.

Mark

Re: bigcore.exp on 64-bit systems

[Eli Zaretskii]

> Date: Sat, 13 Jan 2007 14:22:52 -0500
> From: Daniel Jacobowitz <drow@false.org>
> Cc: Mark Kettenis <mark.kettenis@xs4all.nl>, eliz@gnu.org,
> 	gdb@sourceware.org
> 
> For comparison purposes, an 8GB limit on my system completes the core
> dump in 1.6 seconds.

Yep, that works for me on fencepost as well.  Even 256GB works and
finishes in reasonable time.

Re: bigcore.exp on 64-bit systems

[Daniel Jacobowitz]

On Sat, Jan 13, 2007 at 04:09:37PM +0200, Eli Zaretskii wrote:
> I'm trying to run the testsuite on a 64-bit machine that identifies
> itself thusly:
> 
>     Linux fencepost 2.6.16.29-xen #1 SMP Wed Dec 6 07:32:36 EST 2006 x86_64 GNU/Linux

Hi fencepost! :-)

> On this machine, bigcore.exp is a pest: it runs for a very long time,
> produces a humongous core file (so large that I expect the sysadmins
> to send me angry emails), and eventually fails (or so it seems).
> Here's what gdb.sum says about it:

The core file is probably not that large.  Could you check with 'du'?
As Mark said, it's supposed to be a sparse file - in case you're not
familiar with them, that means that ls -l will show something huge
but du will show something quite small.  Most of the all-zero blocks
will not have any on-disk representation.

Of course, how efficient large sparse files are depends on the
filesystem.  I've never used xfs; it might be an xfs-specific
problem.  Also, I know that there was a problem in the 2.6.1x series
somewhere (bad choice of locking) that led to this test taking a
really long time - but I don't know what version was affected.

It's not necessarily a problem on a 64-bit system; I run it on
one normally.  It takes twenty or thirty seconds to dump core,
which is much longer than it ought to take, but much shorter
than it's taking for you.

I'm not sure what to suggest :-(  If it is a sufficient nuisance,
perhaps we should add a way for you to disable the test.

-- 
Daniel Jacobowitz
CodeSourcery

Re: bigcore.exp on 64-bit systems

[Eli Zaretskii]

> Date: Sat, 13 Jan 2007 10:20:59 -0500
> From: Daniel Jacobowitz <drow@false.org>
> Cc: gdb@sourceware.org
> 
> > On this machine, bigcore.exp is a pest: it runs for a very long time,
> > produces a humongous core file (so large that I expect the sysadmins
> > to send me angry emails), and eventually fails (or so it seems).
> > Here's what gdb.sum says about it:
> 
> The core file is probably not that large.  Could you check with 'du'?

You are right, it's only 3.9MB.  But the test still fails with
timeout.

> If it is a sufficient nuisance, perhaps we should add a way for you
> to disable the test.

I can always rename the .exp file, like Joel suggested, but it seems
unclean to remove test files to disable them.

Re: bigcore.exp on 64-bit systems

[Daniel Jacobowitz]

On Sat, Jan 13, 2007 at 07:57:31PM +0200, Eli Zaretskii wrote:
> > If it is a sufficient nuisance, perhaps we should add a way for you
> > to disable the test.
> 
> I can always rename the .exp file, like Joel suggested, but it seems
> unclean to remove test files to disable them.

If someone wants to work out which kernel version it is that's
responsible for the slowdown, we could add a version check - this
is a native only test, so we can do that.  At least for now it's
native only.

-- 
Daniel Jacobowitz
CodeSourcery