Lots of SIGFPE and SIGSEGV found while debugging Sun hotspot vm on AMD64 machine

Lots of SIGFPE and SIGSEGV found while debugging Sun hotspot vm on AMD64 machine

[Neo]

hi,

Everything is fine when I am using gdb to debug Sun hotspot VM on IA32 
machine. But something wired happens on AMD64 machine. Since the code 
executed on VM is dynamically generated I cannot get any information 
from the "bt" command. Is there any other approach to debug? Without 
attaching gdb, there is no error. I am using the latest version gdb from 
CVS.

 > java_g -Xloggc:gc.log -XX:+Verbose -classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
VM option '+Verbose'
[SafePoint Polling address: 0x0000002a96465000]

Checking JVM

1st bounds test:        OK
Super: Class Super, public=34, protected=33, private=32
Sub: Class Super, public=804, protected=803, private=802
Super: prot=33, priv=32
Sub:  prot=111, priv=105
Subclass test OK
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
1 1 2 3 5 8 666 666 13 13 21 21
2nd bounds test:        OK
testIf:     OK
testArray:  OK
testBitOps: OK
testFor:    OK
testDiv:    OK
testTableSwitch:  OK
testLookupSwitch: OK
testHiddenField:  OK
checkRemainders:  long double OK
checkMathFcts: OK
Primes less than 50: 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47
testExc1(simple throw/catch):  OK
testExc2(skip catch clauses):  OK
testExc3(catch in inner):      OK
testExc4(catch in outer):      OK
testExc5(rethrow):             OK
testExc6(throw accross call):  OK
testExc7(throw accr. 2 calls): OK
loopExitContinueInExceptionHandler: OK
testStringHash:  OK
testClone:       OK
testObjectArray: OK
checkInstanceOf: OK
checkInterfaceInstanceOf: OK
testWaitNull: OK
testVarAndMethodNameClash: OK
testMisk: OK
testGC: OK
PepTest: OK

JVM Check OK
Constructing the company now   Hang....on

Loading Warehouse 1...



Loading properties from SPECjbb.props
The results will be in: results/SPECjbb.038.raw
Opened results/SPECjbb.038.results
Opened results/SPECjbb.038.raw

Licensed Materials - Property of SPEC
SPECjbb2000
Copyright (c) 2000 Standard Performance Evaluation Corporation 
(SPEC),All rights reserved,(C) Copyright IBM Corp., 1996 - 2000All 
rights reserved,
This source code is provided as is, without any express or implied warranty.

TPC Fair Use policy:

SPECjbb2000 is not a TPC Benchmark. SPECjbb2000 results are not 
comparable with
any TPC Benchmark results. The workload used by SPECjbb2000 is inspired
by the TPC-C specification, TPC Benchmark C , Standard Specification,
Revision 3.2, August 27 1996. TPC Benchmark is a trademark of the 
Transaction
Processing Performance Council.

Benchmark SPECjbb2000 1.02 now Opening


Input Properties:
  ramp_up_seconds = 30
  measurement_seconds = 120
  forcegc = false
  starting_number_warehouses = 1
  increment_number_warehouses = 1
  ending_number_warehouses = 1
jar validity is true


Checking whether run will be valid

Run will be COMPLIANT

Constructing the company now   Hang....on


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Loading Warehouse 1...


Start Terminals
  started terminals for Warehouse 1
Terminal Rampup began Thu Mar 22 10:58:38 CST 2007 for 0.5 minutes

Benchmark SPECjbb2000 1.01: warehouse 1 terminal 1

[cjia@neoworld /home/cjia/research/benchmark/jbb2000]$
 > /opt/gdb-cvs/bin/gdb --args java_g -classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
GNU gdb 6.6.50.20070322-cvs
Copyright (C) 2007 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
(gdb) r
Starting program: 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g 
-classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
[Thread debugging using libthread_db enabled]
[New Thread 182894205312 (LWP 16629)]
[New process 16629]

Checking JVM

1st bounds test:        OK
Super: Class Super, public=34, protected=33, private=32
Sub: Class Super, public=804, protected=803, private=802
Super: prot=33, priv=32
Sub:  prot=111, priv=105
Subclass test OK
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
1 1 2 3 5 8 666 666 13 13 21 21
2nd bounds test:        OK
testIf:     OK
testArray:  OK
testBitOps: OK
testFor:    OK
testDiv:    [New LWP 16629]

Program received signal SIGFPE, Arithmetic exception.
0x0000002a96ec93e8 in ?? ()
(gdb) c
Continuing.
OK
testTableSwitch:  OK
testLookupSwitch: OK
testHiddenField:  OK
checkRemainders:  long double OK
checkMathFcts: OK
Primes less than 50: 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47
testExc1(simple throw/catch):  OK
testExc2(skip catch clauses):  OK
testExc3(catch in inner):      OK
testExc4(catch in outer):      OK
testExc5(rethrow):             OK
testExc6(throw accross call):  OK
testExc7(throw accr. 2 calls): OK
loopExitContinueInExceptionHandler: OK
testStringHash:  OK
testClone:      
Program received signal SIGSEGV, Segmentation fault.
0x0000002a96ed5bbb in ?? ()
(gdb) c
Continuing.
OK
testObjectArray: OK
checkInstanceOf: OK
checkInterfaceInstanceOf: OK
testWaitNull: OK
testVarAndMethodNameClash: OK
testMisk: OK
testGC: OK
PepTest: OK

JVM Check OK
cConstructing the company now   Hang....on

Loading Warehouse 1...

Program received signal SIGSEGV, Segmentation fault.
0x0000002a96ff7685 in ?? ()
(gdb) bt
#0  0x0000002a96ff7685 in ?? ()
#1  0x0000002af1f6ebc8 in ?? ()
#2  0x0000007fbfffc948 in ?? ()
#3  0x0000002ad836e461 in ?? ()
#4  0x0000007fbfffc960 in ?? ()
#5  0x0000007fbfffc880 in ?? ()
#6  0x0000002ad7ebb140 in ?? ()
#7  0x0000007fbfffc8e0 in ?? ()
#8  0x0000002ad7f50160 in ?? ()
#9  0x0000002ad8364db8 in ?? ()
#10 0x0000002ad7ebb160 in ?? ()
#11 0x0000007fbfffc8d8 in ?? ()
#12 0x0000002a96fe1402 in ?? ()
#13 0x0000007fbfffc948 in ?? ()
#14 0x0000002af1f6ec20 in ?? ()
#15 0x000000000000000c in ?? ()
#16 0x0000007fbfffc8e0 in ?? ()
#17 0x0000002a96eade53 in ?? ()
#18 0x0000002af1f6ebc8 in ?? ()
#19 0x0000002af1f6e7a8 in ?? ()
#20 0x0000002af1f6e7a8 in ?? ()
#21 0x0000000000000001 in ?? ()
#22 0x0000002af1f6e7a8 in ?? ()
#23 0x0000007fbfffc900 in ?? ()
#24 0x0000002ad836e461 in ?? ()
#25 0x0000007fbfffc960 in ?? ()
#26 0x0000002ad8375ee0 in ?? ()
#27 0x0000000000000000 in ?? ()
(gdb) c
Continuing.



Loading properties from SPECjbb.props
The results will be in: results/SPECjbb.039.raw
Opened results/SPECjbb.039.results
Opened results/SPECjbb.039.raw

Licensed Materials - Property of SPEC
SPECjbb2000
Copyright (c) 2000 Standard Performance Evaluation Corporation 
(SPEC),All rights reserved,(C) Copyright IBM Corp., 1996 - 2000All 
rights reserved,
This source code is provided as is, without any express or implied warranty.

TPC Fair Use policy:

SPECjbb2000 is not a TPC Benchmark. SPECjbb2000 results are not 
comparable with
any TPC Benchmark results. The workload used by SPECjbb2000 is inspired
by the TPC-C specification, TPC Benchmark C , Standard Specification,
Revision 3.2, August 27 1996. TPC Benchmark is a trademark of the 
Transaction
Processing Performance Council.

Benchmark SPECjbb2000 1.02 now Opening


Program received signal SIGSEGV, Segmentation fault.
0x0000002a96ed5b96 in ?? ()
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x0000002a96ed5b96 in ?? ()
(gdb) bt
#0  0x0000002a96ed5b96 in ?? ()
#1  0x0000002a96eade53 in ?? ()
#2  0x0000000000000000 in ?? ()

Thanks,
Neo

-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

Re: Lots of SIGFPE and SIGSEGV found while debugging Sun hotspot vm on AMD64 machine

[Daniel Jacobowitz]

On Thu, Mar 22, 2007 at 12:07:22PM -0600, Neo wrote:
> hi,

> Everything is fine when I am using gdb to debug Sun hotspot VM on IA32 machine. 
> But something wired happens on AMD64 machine. Since the code executed on VM is 
> dynamically generated I cannot get any information from the "bt" command. Is 
> there any other approach to debug? Without attaching gdb, there is no error. I 
> am using the latest version gdb from CVS.

It's probably generating those signals deliberately.  Try using
"handle" to pass and ignore them.  That's fairly common for JVMs.


-- 
Daniel Jacobowitz
CodeSourcery

Re: Lots of SIGFPE and SIGSEGV found while debugging Sun hotspotvm on AMD64 machine

[Neo]

Michael Snyder wrote:
>
> On Thu, 2007-03-22 at 13:46 -0400, Daniel Jacobowitz wrote:
> > On Thu, Mar 22, 2007 at 12:07:22PM -0600, Neo wrote:
> > > hi,
> >
> > > Everything is fine when I am using gdb to debug Sun hotspot VM on 
> IA32 machine.
> > > But something wired happens on AMD64 machine. Since the code 
> executed on VM is
> > > dynamically generated I cannot get any information from the "bt" 
> command. Is
> > > there any other approach to debug? Without attaching gdb, there is 
> no error. I
> > > am using the latest version gdb from CVS.
> >
> > It's probably generating those signals deliberately.  Try using
> > "handle" to pass and ignore them.  That's fairly common for JVMs.
>
> Yes, the fact that "continue" after a SIGSEGV does not cause the
> process to die indicates that there is a signal handler.
>
> Try "handle SIGSEGV nostop pass noprint"
>
Dnaiel & Michael,

Yes, those annoying messages are gone after applying "handle". But the 
latest gdb seems cannot break on the functions.

This is from gdb 6.5.50.20060621-cvs, which can resolve the line number 
and can break.

 > gdb --args java_g -XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
GNU gdb 6.5.50.20060621-cvs
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...Using host 
libthread_db library "/lib64/tls/libthread_db.so.1".

(gdb) r
Starting program: 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g 
-XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
[Thread debugging using libthread_db enabled]
[New Thread 182894205312 (LWP 17940)]
[New process 17940]
Executing new program: 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g
[Thread debugging using libthread_db enabled]
[New Thread 182894205312 (LWP 17940)]
VM option '+UseSerialGC'
[New Thread 1074788704 (LWP 17943)]
[New Thread 1075837280 (LWP 17944)]
[New Thread 1076885856 (LWP 17945)]
[New Thread 1077934432 (LWP 17946)]
[New Thread 1078983008 (LWP 17947)]
[New Thread 1080031584 (LWP 17948)]
[New Thread 1081080160 (LWP 17949)]
[New Thread 1082128736 (LWP 17950)]
[New Thread 1083177312 (LWP 17951)]

Checking JVM

1st bounds test:        OK
Super: Class Super, public=34, protected=33, private=32
Sub: Class Super, public=804, protected=803, private=802
Super: prot=33, priv=32
Sub:  prot=111, priv=105
Subclass test OK
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
1 1 2 3 5 8 666 666 13 13 21 21
2nd bounds test:        OK
testIf:     OK
testArray:  OK
testBitOps: OK
testFor:    OK
testDiv:   
Program received signal SIGFPE, Arithmetic exception.
[Switching to Thread 182894205312 (LWP 17940)]
0x0000002a96eca3e8 in ?? ()
(gdb) b tenuredGeneration.cpp:289
Breakpoint 1 at 0x2a95ee5f0a: file 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/hotspot/src/share/vm/memory/tenuredGeneration.cpp, 
line 289.
(gdb) c
Continuing.
OK
testTableSwitch:  OK
testLookupSwitch: OK
testHiddenField:  OK
checkRemainders:  long double OK
checkMathFcts: OK
Primes less than 50: 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47
testExc1(simple throw/catch):  OK
testExc2(skip catch clauses):  OK
testExc3(catch in inner):      OK
testExc4(catch in outer):      OK
testExc5(rethrow):             OK
testExc6(throw accross call):  OK
testExc7(throw accr. 2 calls): OK
loopExitContinueInExceptionHandler: OK
testStringHash:  OK
testClone:       OK
testObjectArray: OK
checkInstanceOf: OK
checkInterfaceInstanceOf: OK
testWaitNull: OK
testVarAndMethodNameClash: OK
testMisk: OK
testGC: [Switching to Thread 1074788704 (LWP 17943)]

Breakpoint 1, TenuredGeneration::compute_new_size (this=0x557498)
    at 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/hotspot/src/share/vm/memory/tenuredGeneration.cpp:289
289       assert(_shrink_factor <= 100, "invalid shrink factor");
Current language:  auto; currently c++

This is from gdb 6.6.50.20070322-cvs, which cannot resolve the symbol 
and cannot break.

 > /opt/gdb-cvs/bin/gdb --args java_g -XX:+UseSerialGC -Xms144m -Xmx654m 
-classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
GNU gdb 6.6.50.20070322-cvs
Copyright (C) 2007 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
(gdb) r
Starting program: 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g 
-XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
[Thread debugging using libthread_db enabled]
[New Thread 182894205312 (LWP 17984)]
[New process 17984]
VM option '+UseSerialGC'

Checking JVM

1st bounds test:        OK
Super: Class Super, public=34, protected=33, private=32
Sub: Class Super, public=804, protected=803, private=802
Super: prot=33, priv=32
Sub:  prot=111, priv=105
Subclass test OK
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
1 1 2 3 5 8 666 666 13 13 21 21
2nd bounds test:        OK
testIf:     OK
testArray:  OK
testBitOps: OK
testFor:    OK
testDiv:    [New LWP 17984]

Program received signal SIGFPE, Arithmetic exception.
0x0000002a96eca3e8 in ?? ()
(gdb) b tenuredGeneration.cpp:289
No source file named tenuredGeneration.cpp.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (tenuredGeneration.cpp:289) pending.
(gdb) c
Continuing.
OK
testTableSwitch:  OK
testLookupSwitch: OK
testHiddenField:  OK
checkRemainders:  long double OK
checkMathFcts: OK
Primes less than 50: 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47
testExc1(simple throw/catch):  OK
testExc2(skip catch clauses):  OK
testExc3(catch in inner):      OK
testExc4(catch in outer):      OK
testExc5(rethrow):             OK
testExc6(throw accross call):  OK
testExc7(throw accr. 2 calls): OK
loopExitContinueInExceptionHandler: OK
testStringHash:  OK
testClone:       OK
testObjectArray: OK
checkInstanceOf: OK
checkInterfaceInstanceOf: OK
testWaitNull: OK
testVarAndMethodNameClash: OK
testMisk: OK
testGC: OK
PepTest: OK

JVM Check OK
Constructing the company now   Hang....on

Loading Warehouse 1...



Loading properties from SPECjbb.props
The results will be in: results/SPECjbb.050.raw
Opened results/SPECjbb.050.results
Opened results/SPECjbb.050.raw

Licensed Materials - Property of SPEC
SPECjbb2000
Copyright (c) 2000 Standard Performance Evaluation Corporation 
(SPEC),All rights reserved,(C) Copyright IBM Corp., 1996 - 2000All 
rights reserved,
This source code is provided as is, without any express or implied warranty.

TPC Fair Use policy:

SPECjbb2000 is not a TPC Benchmark. SPECjbb2000 results are not 
comparable with
any TPC Benchmark results. The workload used by SPECjbb2000 is inspired
by the TPC-C specification, TPC Benchmark C , Standard Specification,
Revision 3.2, August 27 1996. TPC Benchmark is a trademark of the 
Transaction
Processing Performance Council.

Benchmark SPECjbb2000 1.02 now Opening


Input Properties:
  ramp_up_seconds = 30
  measurement_seconds = 120
  forcegc = false
  starting_number_warehouses = 1
  increment_number_warehouses = 1
  ending_number_warehouses = 1


Thanks,
Neo

-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!

Re: Lots of SIGFPE and SIGSEGV found while debugging Sun hotspotvm on AMD64 machine

[Michael Snyder]

On Thu, 2007-03-22 at 15:33 -0600, Neo wrote:

> >
> Dnaiel & Michael,
> 
> Yes, those annoying messages are gone after applying "handle". But the 
> latest gdb seems cannot break on the functions.

I don't know -- but I notice considerable difference in the
thread reporting between your two sessions below.

Are you sure the only difference between the two sessions
is the gdb version?

'nother question -- is tenuredGeneration.cpp in a shared library?
Is gdb finding the shared libraries?  You can get more info about
shared library symbol loading by saying "set verbose" before "run".


> 
> This is from gdb 6.5.50.20060621-cvs, which can resolve the line number 
> and can break.
> 
>  > gdb --args java_g -XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
> ./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
> spec.jbb.JBBmain -propfile SPECjbb.props
> GNU gdb 6.5.50.20060621-cvs
> Copyright (C) 2006 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain 
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...Using host 
> libthread_db library "/lib64/tls/libthread_db.so.1".
> 
> (gdb) r
> Starting program: 
> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g 
> -XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
> ./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
> spec.jbb.JBBmain -propfile SPECjbb.props
> [Thread debugging using libthread_db enabled]
> [New Thread 182894205312 (LWP 17940)]
> [New process 17940]
> Executing new program: 
> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g
> [Thread debugging using libthread_db enabled]
> [New Thread 182894205312 (LWP 17940)]
> VM option '+UseSerialGC'
> [New Thread 1074788704 (LWP 17943)]
> [New Thread 1075837280 (LWP 17944)]
> [New Thread 1076885856 (LWP 17945)]
> [New Thread 1077934432 (LWP 17946)]
> [New Thread 1078983008 (LWP 17947)]
> [New Thread 1080031584 (LWP 17948)]
> [New Thread 1081080160 (LWP 17949)]
> [New Thread 1082128736 (LWP 17950)]
> [New Thread 1083177312 (LWP 17951)]
> 
> Checking JVM
> 
> 1st bounds test:        OK
> Super: Class Super, public=34, protected=33, private=32
> Sub: Class Super, public=804, protected=803, private=802
> Super: prot=33, priv=32
> Sub:  prot=111, priv=105
> Subclass test OK
> 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
> 1 1 2 3 5 8 666 666 13 13 21 21
> 2nd bounds test:        OK
> testIf:     OK
> testArray:  OK
> testBitOps: OK
> testFor:    OK
> testDiv:   
> Program received signal SIGFPE, Arithmetic exception.
> [Switching to Thread 182894205312 (LWP 17940)]
> 0x0000002a96eca3e8 in ?? ()
> (gdb) b tenuredGeneration.cpp:289
> Breakpoint 1 at 0x2a95ee5f0a: file 
> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/hotspot/src/share/vm/memory/tenuredGeneration.cpp, 
> line 289.
> (gdb) c
> Continuing.
> OK
> testTableSwitch:  OK
> testLookupSwitch: OK
> testHiddenField:  OK
> checkRemainders:  long double OK
> checkMathFcts: OK
> Primes less than 50: 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47
> testExc1(simple throw/catch):  OK
> testExc2(skip catch clauses):  OK
> testExc3(catch in inner):      OK
> testExc4(catch in outer):      OK
> testExc5(rethrow):             OK
> testExc6(throw accross call):  OK
> testExc7(throw accr. 2 calls): OK
> loopExitContinueInExceptionHandler: OK
> testStringHash:  OK
> testClone:       OK
> testObjectArray: OK
> checkInstanceOf: OK
> checkInterfaceInstanceOf: OK
> testWaitNull: OK
> testVarAndMethodNameClash: OK
> testMisk: OK
> testGC: [Switching to Thread 1074788704 (LWP 17943)]
> 
> Breakpoint 1, TenuredGeneration::compute_new_size (this=0x557498)
>     at 
> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/hotspot/src/share/vm/memory/tenuredGeneration.cpp:289
> 289       assert(_shrink_factor <= 100, "invalid shrink factor");
> Current language:  auto; currently c++
> 
> This is from gdb 6.6.50.20070322-cvs, which cannot resolve the symbol 
> and cannot break.
> 
>  > /opt/gdb-cvs/bin/gdb --args java_g -XX:+UseSerialGC -Xms144m -Xmx654m 
> -classpath 
> ./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
> spec.jbb.JBBmain -propfile SPECjbb.props
> GNU gdb 6.6.50.20070322-cvs
> Copyright (C) 2007 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain 
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> Using host libthread_db library "/lib64/tls/libthread_db.so.1".
> (gdb) r
> Starting program: 
> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g 
> -XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
> ./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
> spec.jbb.JBBmain -propfile SPECjbb.props
> [Thread debugging using libthread_db enabled]
> [New Thread 182894205312 (LWP 17984)]
> [New process 17984]
> VM option '+UseSerialGC'
> 
> Checking JVM
> 
> 1st bounds test:        OK
> Super: Class Super, public=34, protected=33, private=32
> Sub: Class Super, public=804, protected=803, private=802
> Super: prot=33, priv=32
> Sub:  prot=111, priv=105
> Subclass test OK
> 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
> 1 1 2 3 5 8 666 666 13 13 21 21
> 2nd bounds test:        OK
> testIf:     OK
> testArray:  OK
> testBitOps: OK
> testFor:    OK
> testDiv:    [New LWP 17984]
> 
> Program received signal SIGFPE, Arithmetic exception.
> 0x0000002a96eca3e8 in ?? ()
> (gdb) b tenuredGeneration.cpp:289
> No source file named tenuredGeneration.cpp.
> Make breakpoint pending on future shared library load? (y or [n]) y
> Breakpoint 1 (tenuredGeneration.cpp:289) pending.
> (gdb) c
> Continuing.
> OK
> testTableSwitch:  OK
> testLookupSwitch: OK
> testHiddenField:  OK
> checkRemainders:  long double OK
> checkMathFcts: OK
> Primes less than 50: 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47
> testExc1(simple throw/catch):  OK
> testExc2(skip catch clauses):  OK
> testExc3(catch in inner):      OK
> testExc4(catch in outer):      OK
> testExc5(rethrow):             OK
> testExc6(throw accross call):  OK
> testExc7(throw accr. 2 calls): OK
> loopExitContinueInExceptionHandler: OK
> testStringHash:  OK
> testClone:       OK
> testObjectArray: OK
> checkInstanceOf: OK
> checkInterfaceInstanceOf: OK
> testWaitNull: OK
> testVarAndMethodNameClash: OK
> testMisk: OK
> testGC: OK
> PepTest: OK
> 
> JVM Check OK
> Constructing the company now   Hang....on
> 
> Loading Warehouse 1...
> 
> 
> 
> Loading properties from SPECjbb.props
> The results will be in: results/SPECjbb.050.raw
> Opened results/SPECjbb.050.results
> Opened results/SPECjbb.050.raw
> 
> Licensed Materials - Property of SPEC
> SPECjbb2000
> Copyright (c) 2000 Standard Performance Evaluation Corporation 
> (SPEC),All rights reserved,(C) Copyright IBM Corp., 1996 - 2000All 
> rights reserved,
> This source code is provided as is, without any express or implied warranty.
> 
> TPC Fair Use policy:
> 
> SPECjbb2000 is not a TPC Benchmark. SPECjbb2000 results are not 
> comparable with
> any TPC Benchmark results. The workload used by SPECjbb2000 is inspired
> by the TPC-C specification, TPC Benchmark C , Standard Specification,
> Revision 3.2, August 27 1996. TPC Benchmark is a trademark of the 
> Transaction
> Processing Performance Council.
> 
> Benchmark SPECjbb2000 1.02 now Opening
> 
> 
> Input Properties:
>   ramp_up_seconds = 30
>   measurement_seconds = 120
>   forcegc = false
>   starting_number_warehouses = 1
>   increment_number_warehouses = 1
>   ending_number_warehouses = 1
> 
> 
> Thanks,
> Neo
> 

Re: Lots of SIGFPE and SIGSEGV found while debugging Sun hotspotvm on AMD64 machine

[Neo]

Michael Snyder wrote:
> On Thu, 2007-03-22 at 15:33 -0600, Neo wrote:
>
>   
>> Dnaiel & Michael,
>>
>> Yes, those annoying messages are gone after applying "handle". But the 
>> latest gdb seems cannot break on the functions.
>>     
>
> I don't know -- but I notice considerable difference in the
> thread reporting between your two sessions below.
>
> Are you sure the only difference between the two sessions
> is the gdb version?
>
> 'nother question -- is tenuredGeneration.cpp in a shared library?
> Is gdb finding the shared libraries?  You can get more info about
> shared library symbol loading by saying "set verbose" before "run".
>
>
>   
The following is what I got after "set verbose". It seems that the 
latest gdb does not find the proper .so file. BTW, that .cpp file is in 
the .so. file.

GDB (6.5)

 > gdb --args java_g -XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
GNU gdb 6.5.50.20060621-cvs
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...Using host 
libthread_db library "/lib64/tls/libthread_db.so.1".

(gdb) set verbose
(gdb) r
Starting program: 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g 
-XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
Reading symbols from /lib64/tls/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 182894205312 (LWP 20219)]
Reading symbols from /lib64/libdl.so.2...done.
Reading symbols from /lib64/tls/libc.so.6...done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
[New process 20219]
Executing new program: 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g
Reading symbols from 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g...done.
Reading symbols from /lib64/tls/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 182894205312 (LWP 20219)]
Reading symbols from /lib64/libdl.so.2...done.
Reading symbols from /lib64/tls/libc.so.6...done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Reading symbols from 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/lib/amd64/server/libjvm_g.so...done.
Reading symbols from /lib64/tls/libm.so.6...done.
VM option '+UseSerialGC'
Reading symbols from 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/lib/amd64/native_threads/libhpi_g.so...done.
Reading symbols from /lib64/libnsl.so.1...done.
Reading symbols from /lib64/libnss_files.so.2...done.
Reading symbols from 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/lib/amd64/libverify_g.so...done.
Reading symbols from 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/lib/amd64/libjava_g.so...done.
Reading symbols from 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/lib/amd64/libzip_g.so...done.
[New Thread 1074788704 (LWP 20222)]
[New Thread 1075837280 (LWP 20223)]
[New Thread 1076885856 (LWP 20224)]
[New Thread 1077934432 (LWP 20225)]
[New Thread 1078983008 (LWP 20226)]
[New Thread 1080031584 (LWP 20227)]
[New Thread 1081080160 (LWP 20228)]
[New Thread 1082128736 (LWP 20229)]
[New Thread 1083177312 (LWP 20230)]

Checking JVM

1st bounds test:        OK
Super: Class Super, public=34, protected=33, private=32
Sub: Class Super, public=804, protected=803, private=802
Super: prot=33, priv=32
Sub:  prot=111, priv=105
Subclass test OK
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
1 1 2 3 5 8 666 666 13 13 21 21
2nd bounds test:        OK
testIf:     OK
testArray:  OK
testBitOps: OK
testFor:    OK
testDiv:   
Program received signal SIGFPE, Arithmetic exception.
[Switching to Thread 182894205312 (LWP 20219)]
0x0000002a96eca3e8 in ?? ()

GDB (6.6)

 > /opt/gdb-cvs/bin/gdb --args java_g -XX:+UseSerialGC -Xms144m -Xmx654m 
-classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
GNU gdb 6.6.50.20070322-cvs
Copyright (C) 2007 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
(gdb) set verbose
(gdb) r
Starting program: 
/home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g 
-XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
spec.jbb.JBBmain -propfile SPECjbb.props
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Reading symbols from /lib64/tls/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 182894205312 (LWP 20204)]
Reading symbols from /lib64/libdl.so.2...done.
Reading symbols from /lib64/tls/libc.so.6...done.
[New process 20204]
VM option '+UseSerialGC'

Checking JVM

1st bounds test:        OK
Super: Class Super, public=34, protected=33, private=32
Sub: Class Super, public=804, protected=803, private=802
Super: prot=33, priv=32
Sub:  prot=111, priv=105
Subclass test OK
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
1 1 2 3 5 8 666 666 13 13 21 21
2nd bounds test:        OK
testIf:     OK
testArray:  OK
testBitOps: OK
testFor:    OK
testDiv:    [New LWP 20204]

Program received signal SIGFPE, Arithmetic exception.
0x0000002a96eca3e8 in ?? ()


Thanks,
Neo

>> This is from gdb 6.5.50.20060621-cvs, which can resolve the line number 
>> and can break.
>>
>>  > gdb --args java_g -XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
>> ./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
>> spec.jbb.JBBmain -propfile SPECjbb.props
>> GNU gdb 6.5.50.20060621-cvs
>> Copyright (C) 2006 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you are
>> welcome to change it and/or distribute copies of it under certain 
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for details.
>> This GDB was configured as "x86_64-unknown-linux-gnu"...Using host 
>> libthread_db library "/lib64/tls/libthread_db.so.1".
>>
>> (gdb) r
>> Starting program: 
>> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g 
>> -XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
>> ./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
>> spec.jbb.JBBmain -propfile SPECjbb.props
>> [Thread debugging using libthread_db enabled]
>> [New Thread 182894205312 (LWP 17940)]
>> [New process 17940]
>> Executing new program: 
>> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g
>> [Thread debugging using libthread_db enabled]
>> [New Thread 182894205312 (LWP 17940)]
>> VM option '+UseSerialGC'
>> [New Thread 1074788704 (LWP 17943)]
>> [New Thread 1075837280 (LWP 17944)]
>> [New Thread 1076885856 (LWP 17945)]
>> [New Thread 1077934432 (LWP 17946)]
>> [New Thread 1078983008 (LWP 17947)]
>> [New Thread 1080031584 (LWP 17948)]
>> [New Thread 1081080160 (LWP 17949)]
>> [New Thread 1082128736 (LWP 17950)]
>> [New Thread 1083177312 (LWP 17951)]
>>
>> Checking JVM
>>
>> 1st bounds test:        OK
>> Super: Class Super, public=34, protected=33, private=32
>> Sub: Class Super, public=804, protected=803, private=802
>> Super: prot=33, priv=32
>> Sub:  prot=111, priv=105
>> Subclass test OK
>> 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
>> 1 1 2 3 5 8 666 666 13 13 21 21
>> 2nd bounds test:        OK
>> testIf:     OK
>> testArray:  OK
>> testBitOps: OK
>> testFor:    OK
>> testDiv:   
>> Program received signal SIGFPE, Arithmetic exception.
>> [Switching to Thread 182894205312 (LWP 17940)]
>> 0x0000002a96eca3e8 in ?? ()
>> (gdb) b tenuredGeneration.cpp:289
>> Breakpoint 1 at 0x2a95ee5f0a: file 
>> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/hotspot/src/share/vm/memory/tenuredGeneration.cpp, 
>> line 289.
>> (gdb) c
>> Continuing.
>> OK
>> testTableSwitch:  OK
>> testLookupSwitch: OK
>> testHiddenField:  OK
>> checkRemainders:  long double OK
>> checkMathFcts: OK
>> Primes less than 50: 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47
>> testExc1(simple throw/catch):  OK
>> testExc2(skip catch clauses):  OK
>> testExc3(catch in inner):      OK
>> testExc4(catch in outer):      OK
>> testExc5(rethrow):             OK
>> testExc6(throw accross call):  OK
>> testExc7(throw accr. 2 calls): OK
>> loopExitContinueInExceptionHandler: OK
>> testStringHash:  OK
>> testClone:       OK
>> testObjectArray: OK
>> checkInstanceOf: OK
>> checkInterfaceInstanceOf: OK
>> testWaitNull: OK
>> testVarAndMethodNameClash: OK
>> testMisk: OK
>> testGC: [Switching to Thread 1074788704 (LWP 17943)]
>>
>> Breakpoint 1, TenuredGeneration::compute_new_size (this=0x557498)
>>     at 
>> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/hotspot/src/share/vm/memory/tenuredGeneration.cpp:289
>> 289       assert(_shrink_factor <= 100, "invalid shrink factor");
>> Current language:  auto; currently c++
>>
>> This is from gdb 6.6.50.20070322-cvs, which cannot resolve the symbol 
>> and cannot break.
>>
>>  > /opt/gdb-cvs/bin/gdb --args java_g -XX:+UseSerialGC -Xms144m -Xmx654m 
>> -classpath 
>> ./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
>> spec.jbb.JBBmain -propfile SPECjbb.props
>> GNU gdb 6.6.50.20070322-cvs
>> Copyright (C) 2007 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you are
>> welcome to change it and/or distribute copies of it under certain 
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for details.
>> This GDB was configured as "x86_64-unknown-linux-gnu"...
>> Using host libthread_db library "/lib64/tls/libthread_db.so.1".
>> (gdb) r
>> Starting program: 
>> /home/cjia/research/jdk/jdk5-dynamic_gc.svn/control/build/linux-amd64/bin/java_g 
>> -XX:+UseSerialGC -Xms144m -Xmx654m -classpath 
>> ./jbb.jar:./jbb_no_precompile.jar:./check.jar:./reporter.jar:/home/cjia/jdk1.5.0_05/lib/dt.jar:/home/cjia/jdk1.5.0_05/lib/tools.jar 
>> spec.jbb.JBBmain -propfile SPECjbb.props
>> [Thread debugging using libthread_db enabled]
>> [New Thread 182894205312 (LWP 17984)]
>> [New process 17984]
>> VM option '+UseSerialGC'
>>
>> Checking JVM
>>
>> 1st bounds test:        OK
>> Super: Class Super, public=34, protected=33, private=32
>> Sub: Class Super, public=804, protected=803, private=802
>> Super: prot=33, priv=32
>> Sub:  prot=111, priv=105
>> Subclass test OK
>> 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
>> 1 1 2 3 5 8 666 666 13 13 21 21
>> 2nd bounds test:        OK
>> testIf:     OK
>> testArray:  OK
>> testBitOps: OK
>> testFor:    OK
>> testDiv:    [New LWP 17984]
>>
>> Program received signal SIGFPE, Arithmetic exception.
>> 0x0000002a96eca3e8 in ?? ()
>> (gdb) b tenuredGeneration.cpp:289
>> No source file named tenuredGeneration.cpp.
>> Make breakpoint pending on future shared library load? (y or [n]) y
>> Breakpoint 1 (tenuredGeneration.cpp:289) pending.
>> (gdb) c
>> Continuing.
>> OK
>> testTableSwitch:  OK
>> testLookupSwitch: OK
>> testHiddenField:  OK
>> checkRemainders:  long double OK
>> checkMathFcts: OK
>> Primes less than 50: 2 3 5 7 11 13 17 19 23 29 31 37 41 43 47
>> testExc1(simple throw/catch):  OK
>> testExc2(skip catch clauses):  OK
>> testExc3(catch in inner):      OK
>> testExc4(catch in outer):      OK
>> testExc5(rethrow):             OK
>> testExc6(throw accross call):  OK
>> testExc7(throw accr. 2 calls): OK
>> loopExitContinueInExceptionHandler: OK
>> testStringHash:  OK
>> testClone:       OK
>> testObjectArray: OK
>> checkInstanceOf: OK
>> checkInterfaceInstanceOf: OK
>> testWaitNull: OK
>> testVarAndMethodNameClash: OK
>> testMisk: OK
>> testGC: OK
>> PepTest: OK
>>
>> JVM Check OK
>> Constructing the company now   Hang....on
>>
>> Loading Warehouse 1...
>>
>>
>>
>> Loading properties from SPECjbb.props
>> The results will be in: results/SPECjbb.050.raw
>> Opened results/SPECjbb.050.results
>> Opened results/SPECjbb.050.raw
>>
>> Licensed Materials - Property of SPEC
>> SPECjbb2000
>> Copyright (c) 2000 Standard Performance Evaluation Corporation 
>> (SPEC),All rights reserved,(C) Copyright IBM Corp., 1996 - 2000All 
>> rights reserved,
>> This source code is provided as is, without any express or implied warranty.
>>
>> TPC Fair Use policy:
>>
>> SPECjbb2000 is not a TPC Benchmark. SPECjbb2000 results are not 
>> comparable with
>> any TPC Benchmark results. The workload used by SPECjbb2000 is inspired
>> by the TPC-C specification, TPC Benchmark C , Standard Specification,
>> Revision 3.2, August 27 1996. TPC Benchmark is a trademark of the 
>> Transaction
>> Processing Performance Council.
>>
>> Benchmark SPECjbb2000 1.02 now Opening
>>
>>
>> Input Properties:
>>   ramp_up_seconds = 30
>>   measurement_seconds = 120
>>   forcegc = false
>>   starting_number_warehouses = 1
>>   increment_number_warehouses = 1
>>   ending_number_warehouses = 1
>>
>>
>> Thanks,
>> Neo
>>
>>     
>
>   


-- 
I would remember that if researchers were not ambitious
probably today we haven't the technology we are using!