Test suite docs

Test suite docs

[Eli Zaretskii]

This is my first experience running the test suite, and it is quite
frustrating.  All I wanted was to run the tests before and after a
change I'm about to suggest on gdb-patches.  Unfortunately, I ended up
wasting my scarce free time on figuring out several gory details.

While I'm no newcomer to Free Software, and I expect to spend some
time figuring out things on my own when it comes to using a new piece
of software, the test suite makes it exceptionally hard, IMHO.  Some
of the reasons are out of our control: the tests use several software
packages (Dejagnu which uses Expect which uses TCL), so answers are
potentially scattered across several unrelated packages, and the fact
that none of them has GNU standard Info manuals (or at least I
couldn't find them on fencepost.gnu.org) doesn't help.  But that's
just one more reason to have a good user-level documentation in GDB to
help overcome these difficulties.

I try to summarize the problems I needed to solve below.  Could
someone experienced in running the test suite please consider writing
up some user-friendly instructions for first-time users of the test
suite?  I volunteer to do whatever it takes to add them to the
appropriate parts of the GDB documentation, but I need someone ``in
the know'' of the test suite internals to provide the content.

I did find the few words that gdb/README says about running the tests,
and the brief section in gdbint.texinfo; however, they still fall
short of what I needed to know, see below.

TIA

Here are the questions I couldn't find answers to:

  . Where do I find the canonical results for my platform?

    People talk about XFAILs and ``unexpected failures'', but there
    seems to be no place to consult the expected results for all the
    tests and see if what you get is okay or not.  The test suite
    prints a summary of the tests, but how do I find out what are
    those ``unexpected successes'' and ``expected failures''?  What
    are those XPASS, XFAIL, UNTESTED, and other indications displayed
    while the suite runs?

  . How do I compare two runs?  If diff'ing testsuite/gdb.sum is the
    right way, it seems to not be documented anywhere, and gdb.sum
    doesn't seem to be preserved across runs, so one must manually
    copy it to avoid overwriting it.  Am I missing something?

  . How does one disable a specific test?  Suppose some test takes an
    exceptionally long time -- how do I run the suite without it?
    
    gdbint.texinfo tells how to _run_ a specific test or a short list
    of test, but that method is not practical for _disabling_ a small
    number of tests and running all the rest.  gdbint.texinfo also
    says something about not ``adding expected failures lightly'', but
    keeps silent about how does one make a test an expected failure.
    In general, the language in that section of gdbint assumes the
    reader is already an experienced writer of Dejagnu tests, which is
    not a good assumption for a manual.

  . Where do I look for definitions and docs of specific subroutines
    that the *.exp files use?

    Suppose I've found out about the set_xfail subroutine, and want to
    look into it and see whether it can be used to disable a test:
    where do I look for its definition or its documentation?  There's
    the test's *.exp file, there's the testsuite/lib/ subdirectory
    (which, btw, is only mentioned in passing in gdbint.texinfo), and
    then there are Dejagnu and Expect and TCL.  Could we please have a
    list of files or directories to look in, and a list of
    documentation files to browse?

Re: Test suite docs

[Joel Brobecker]

Hello Eli,

I am really sorry to learn that running the testsuite turned out
to be so trying. Don't hesitate to ask us for tips and tricks
before wasting more of your time.

I cannot say that I am a dejagnu expert. In fact, I cannot
say that I like this technology and the less time I spend fiddling
with it, the better I feel.

So, for lack of knowledge,  I will not volunteer to write the doco up.
However, here some of the things I do. Bear in mind that they might
not be the most efficient way of doing things, but they should
help you in the meantime.

> While I'm no newcomer to Free Software, and I expect to spend some
> time figuring out things on my own when it comes to using a new piece
> of software, the test suite makes it exceptionally hard, IMHO.  Some
> of the reasons are out of our control: the tests use several software
> packages (Dejagnu which uses Expect which uses TCL), so answers are
> potentially scattered across several unrelated packages, and the fact
> that none of them has GNU standard Info manuals (or at least I
> couldn't find them on fencepost.gnu.org) doesn't help.  But that's
> just one more reason to have a good user-level documentation in GDB to
> help overcome these difficulties.

I use the latest official dejagnu release, along with one of the latest
TCL and expect releases. They were the latest at the time when I did
the install, which was a while ago (perhaps a couple of years).

It used to be that we were supposed to use the dejagnu that was
in the src tree, but I don't think this is the case anymore. I think
this part of src has been deprecated, or perhaps removed.

>   . Where do I find the canonical results for my platform?
> 
>     People talk about XFAILs and ``unexpected failures'', but there
>     seems to be no place to consult the expected results for all the
>     tests and see if what you get is okay or not.  The test suite
>     prints a summary of the tests, but how do I find out what are
>     those ``unexpected successes'' and ``expected failures''?  What
>     are those XPASS, XFAIL, UNTESTED, and other indications displayed
>     while the suite runs?

This is actually a very tricky question, IMO. The results of the
testsuite are so dependent on the quality of the compiler that
I've given up on marking tests are xfails or kfails, etc.

What I have been doing ever since I started working on GDB, is
that I run the testsuite before I make the change, and then after
(as you suggest below). If I don't see any regressions, then I consider
that the change passes the testsuite (that's why I always mention
"no regression" when I check a change against the testsuite).

>   . How do I compare two runs?  If diff'ing testsuite/gdb.sum is the
>     right way, it seems to not be documented anywhere, and gdb.sum
>     doesn't seem to be preserved across runs, so one must manually
>     copy it to avoid overwriting it.  Am I missing something?

I personnally use a home-made program that is essentially a smart
diff tool for .sum files. It presents the information in 3 columns,
showing only the differences. Column one, the result in the reference
test; column 2, the result in the second .sum file; and column 3
the label of the test.

Here is an example of the output:
% sumtool -c gdb.sum.ref gdb.sum

* gdb.ada:
+------------+------------+----------------------------------------------------+
|       FAIL | PASS       | null_record.exp: ptype on null record              |
+------------+------------+----------------------------------------------------+

* gdb.threads:
+------------+------------+----------------------------------------------------+
|       PASS | FAIL       | schedlock.exp: thread 0 ran                        |
|       PASS | FAIL       | schedlock.exp: thread 1 ran                        |
|       PASS | FAIL       | schedlock.exp: thread 2 ran                        |
|       PASS | FAIL       | schedlock.exp: thread 3 ran                        |
|       PASS |            | schedlock.exp: other thread 3 didn't run           |
|            | PASS       | schedlock.exp: other thread 4 didn't run           |
|       PASS |            | schedlock.exp: other thread 3 didn't run (step ... |
|            |            | ... ping)                                          |
|            | PASS       | schedlock.exp: other thread 4 didn't run (step ... |
|            |            | ... ping)                                          |
+------------+------------+----------------------------------------------------+

Text doesn't allow me to show everything to you, but the "-c" option
mean "color", so PASS are printed in green, while "FAIL" are printed
in red. It's very easy to scan the information and pickup what's wrong.

If you like, I can send you the source. It's an Ada program, however,
so you'll need an Ada compiler.

>   . How does one disable a specific test?  Suppose some test takes an
>     exceptionally long time -- how do I run the suite without it?

The way I do it: either rename the .exp file, or even delete it.

>   . Where do I look for definitions and docs of specific subroutines
>     that the *.exp files use?
> 
>     Suppose I've found out about the set_xfail subroutine, and want to
>     look into it and see whether it can be used to disable a test:
>     where do I look for its definition or its documentation?  There's
>     the test's *.exp file, there's the testsuite/lib/ subdirectory
>     (which, btw, is only mentioned in passing in gdbint.texinfo), and
>     then there are Dejagnu and Expect and TCL.  Could we please have a
>     list of files or directories to look in, and a list of
>     documentation files to browse?

For this item, I don't know how to help you. You know about the testsuite/lib
dir, which is as much as I know. I often have a look at the TCL reference
manuals when I need something. But I've managed to get by only by pretty
much copy/paste'ing previous code.

-- 
Joel

Re: Test suite docs

[Eli Zaretskii]

> Date: Sat, 13 Jan 2007 17:28:46 +0400
> From: Joel Brobecker <brobecker@adacore.com>
> Cc: gdb@sourceware.org
> 
> So, for lack of knowledge,  I will not volunteer to write the doco up.
> However, here some of the things I do. Bear in mind that they might
> not be the most efficient way of doing things, but they should
> help you in the meantime.

Thanks.

> I use the latest official dejagnu release, along with one of the latest
> TCL and expect releases. They were the latest at the time when I did
> the install, which was a while ago (perhaps a couple of years).

I asked gnu.org sysadmins to install dejagnu, and they installed
version 1.4.4, which I think is the latest official release.  So this
is okay in my case.

> >   . How do I compare two runs?  If diff'ing testsuite/gdb.sum is the
> >     right way, it seems to not be documented anywhere, and gdb.sum
> >     doesn't seem to be preserved across runs, so one must manually
> >     copy it to avoid overwriting it.  Am I missing something?
> 
> I personnally use a home-made program that is essentially a smart
> diff tool for .sum files. It presents the information in 3 columns,
> showing only the differences. Column one, the result in the reference
> test; column 2, the result in the second .sum file; and column 3
> the label of the test.
> 
> Here is an example of the output:
> % sumtool -c gdb.sum.ref gdb.sum

How do you produce gdb.sum.ref? is it by manually copying some
previous gdb.sum, or is there a better way?

Re: Test suite docs

[Joel Brobecker]

> > Here is an example of the output:
> > % sumtool -c gdb.sum.ref gdb.sum
> 
> How do you produce gdb.sum.ref? is it by manually copying some
> previous gdb.sum, or is there a better way?

It's a copy of a previous gdb.sum (the one I got from running
the testsuite before making my change). I am realizing that my
situation might be different from yours, since in my case, the
compiler can change daily. I also work on dozens of different
machines, many different OSes, arches, etc. What I mean that by
is that everything changes all the time. But perhaps in your
case, keeping a reference gdb.sum around might be a good option
for you and save you the trouble of having to generate it every
time. There is still the fact that the testsuite also evolves
quite a bit, so even in your case this might not be a practical
suggestion.

-- 
Joel

Re: Test suite docs

[Mark Kettenis]

> Date: Sat, 13 Jan 2007 12:26:11 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> 
> This is my first experience running the test suite, and it is quite
> frustrating.  All I wanted was to run the tests before and after a
> change I'm about to suggest on gdb-patches.  Unfortunately, I ended up
> wasting my scarce free time on figuring out several gory details.
> 
> While I'm no newcomer to Free Software, and I expect to spend some
> time figuring out things on my own when it comes to using a new piece
> of software, the test suite makes it exceptionally hard, IMHO.  Some
> of the reasons are out of our control: the tests use several software
> packages (Dejagnu which uses Expect which uses TCL), so answers are
> potentially scattered across several unrelated packages, and the fact
> that none of them has GNU standard Info manuals (or at least I
> couldn't find them on fencepost.gnu.org) doesn't help.

DejaGnu has an info manual, although it isn't too helpful.  But really
for a normal Unix-like systems, once you've installed DejaGnu and its
dependencies, running the testsuite is as easy as typing "make
check-gdb" in the toplevel directory.  Up to now, that has always
worked for me.

> But that's just one more reason to have a good user-level
> documentation in GDB to help overcome these difficulties.

I wonder if that effort isn't better spent on improving the DejaGnu
manual.

> Here are the questions I couldn't find answers to:
> 
>   . Where do I find the canonical results for my platform?

In theory one should not see any FAILS, and one should work on
eliminating any KFAILS.

> 
>     People talk about XFAILs and ``unexpected failures'', but there
>     seems to be no place to consult the expected results for all the
>     tests and see if what you get is okay or not.  The test suite
>     prints a summary of the tests, but how do I find out what are
>     those ``unexpected successes'' and ``expected failures''?  What
>     are those XPASS, XFAIL, UNTESTED, and other indications displayed
>     while the suite runs?

Apart from the obvious PASS and FAIL, we have:

XFAIL

  The test failed but this was expected because of problems out of our
  control, for example OS or compiler bugs that cannot be easiliy
  worked around.

XPASS

  The test was expected to fail, but passed.  Shouldn't happen, but
  sometimes we accidentally fix bugs.  It could also be that an OS or
  compiler bug got fixed, and the testsuite needs to be adjusted to
  recognize that.

KFAIL

  The test was known to fail.  This is not a new bug but a known bug
  in gdb.

UNTESTED

  The system lacks functionality to run the test, for example because
  of a missing compiler, or an unimplemented feature in the OS or the
  particular GDB config under test.

>   . How do I compare two runs?  If diff'ing testsuite/gdb.sum is the
>     right way, it seems to not be documented anywhere, and gdb.sum
>     doesn't seem to be preserved across runs, so one must manually
>     copy it to avoid overwriting it.  Am I missing something?

This is what I always do.

>   . How does one disable a specific test?  Suppose some test takes an
>     exceptionally long time -- how do I run the suite without it?

All tests should complete within a reasonable amount of time.  If you
see any FAILS because of timeouts, there's a reasonable chance it's
actually the test itself that is broken.

If running a test on a particular platform really is a bad idea, you
can add some code to make it bail out.  Many tests in
testsuite/gdb.arch do this.

>     gdbint.texinfo tells how to _run_ a specific test or a short list
>     of test, but that method is not practical for _disabling_ a small
>     number of tests and running all the rest.  gdbint.texinfo also
>     says something about not ``adding expected failures lightly'', but
>     keeps silent about how does one make a test an expected failure.
>     In general, the language in that section of gdbint assumes the
>     reader is already an experienced writer of Dejagnu tests, which is
>     not a good assumption for a manual.

Well, that information should be found in the DejaGnu manual isn't it?
In fact that manual documents the setup_xfail command.

Re: Test suite docs

[Eli Zaretskii]

> Date: Sat, 13 Jan 2007 15:26:07 +0100 (CET)
> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> CC: gdb@sourceware.org
> 
> DejaGnu has an info manual, although it isn't too helpful.

Am I supposed to get it by simply typing "info dejagnu"?  If so,
perhaps the sysadmins didn't install some package, and I should ask
them to.

> But really
> for a normal Unix-like systems, once you've installed DejaGnu and its
> dependencies, running the testsuite is as easy as typing "make
> check-gdb" in the toplevel directory.

I needed the docs to look for answers for the questions I posted, not
for running the suite (which indeed ``just works'' if one types "make
check").  I'm not used to ask questions without first trying to find
the answers in the available docs.

> > But that's just one more reason to have a good user-level
> > documentation in GDB to help overcome these difficulties.
> 
> I wonder if that effort isn't better spent on improving the DejaGnu
> manual.

I will be able to comment on that once I see that manual ;-)

> >   . Where do I find the canonical results for my platform?
> 
> In theory one should not see any FAILS, and one should work on
> eliminating any KFAILS.

Granted, the theory is understood: this is, after all, a test suite,
so all tests should pass.  I was asking about the practice, and
specifically about the current state of affairs.  It'd be great if the
results on each platform where we test regularly would be available
with every snapshot tarball.  Can we perhaps set up the snapshot
script to produce that?

> Apart from the obvious PASS and FAIL, we have:
> 
> XFAIL
> 
>   The test failed but this was expected because of problems out of our
>   control, for example OS or compiler bugs that cannot be easiliy
>   worked around.
> 
> XPASS
> 
>   The test was expected to fail, but passed.  Shouldn't happen, but
>   sometimes we accidentally fix bugs.  It could also be that an OS or
>   compiler bug got fixed, and the testsuite needs to be adjusted to
>   recognize that.
> 
> KFAIL
> 
>   The test was known to fail.  This is not a new bug but a known bug
>   in gdb.
> 
> UNTESTED
> 
>   The system lacks functionality to run the test, for example because
>   of a missing compiler, or an unimplemented feature in the OS or the
>   particular GDB config under test.

Thanks!

> All tests should complete within a reasonable amount of time.  If you
> see any FAILS because of timeouts, there's a reasonable chance it's
> actually the test itself that is broken.

Well, in my case it was the bigcore.exp test, see my other message.

> If running a test on a particular platform really is a bad idea, you
> can add some code to make it bail out.  Many tests in
> testsuite/gdb.arch do this.

Thanks, I will take a look.

Re: Test suite docs

[Mark Kettenis]

> Date: Sat, 13 Jan 2007 16:48:39 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> CC: gdb@sourceware.org
> Reply-to: Eli Zaretskii <eliz@gnu.org>
> X-XS4ALL-DNSBL-Checked: mxdrop12.xs4all.nl checked 192.114.186.73 against DNS blacklists
> X-Virus-Scanned: by XS4ALL Virus Scanner
> X-XS4ALL-Spam-Score: 0.0 () DK_POLICY_SIGNSOME,SPF_SOFTFAIL
> X-XS4ALL-Spam: NO
> Envelope-To: mark.kettenis@xs4all.nl
> X-UIDL: 1168699735._smtp.mxdrop12.27183,S=3959
> 
> > Date: Sat, 13 Jan 2007 15:26:07 +0100 (CET)
> > From: Mark Kettenis <mark.kettenis@xs4all.nl>
> > CC: gdb@sourceware.org
> > 
> > DejaGnu has an info manual, although it isn't too helpful.
> 
> Am I supposed to get it by simply typing "info dejagnu"?  If so,
> perhaps the sysadmins didn't install some package, and I should ask
> them to.

Well, I do "C-h i g (dejagnu)", but "info dejagnu" works for me too ;-).

> > But really
> > for a normal Unix-like systems, once you've installed DejaGnu and its
> > dependencies, running the testsuite is as easy as typing "make
> > check-gdb" in the toplevel directory.
> 
> I needed the docs to look for answers for the questions I posted, not
> for running the suite (which indeed ``just works'' if one types "make
> check").  I'm not used to ask questions without first trying to find
> the answers in the available docs.
> 
> > > But that's just one more reason to have a good user-level
> > > documentation in GDB to help overcome these difficulties.
> > 
> > I wonder if that effort isn't better spent on improving the DejaGnu
> > manual.
> 
> I will be able to comment on that once I see that manual ;-)
> 
> > >   . Where do I find the canonical results for my platform?
> > 
> > In theory one should not see any FAILS, and one should work on
> > eliminating any KFAILS.
> 
> Granted, the theory is understood: this is, after all, a test suite,
> so all tests should pass.  I was asking about the practice, and
> specifically about the current state of affairs.  It'd be great if the
> results on each platform where we test regularly would be available
> with every snapshot tarball.  Can we perhaps set up the snapshot
> script to produce that?

Well, there is the gdb-testers mailing list that some people send
their testresults to.

> > All tests should complete within a reasonable amount of time.  If you
> > see any FAILS because of timeouts, there's a reasonable chance it's
> > actually the test itself that is broken.
> 
> Well, in my case it was the bigcore.exp test, see my other message.

If you ask me, that testcase is dangerously broken, but then it is
rather hard to write a robust testcase for the functionality being
tested.

Re: Test suite docs

[Eli Zaretskii]

> Date: Sat, 13 Jan 2007 15:26:07 +0100 (CET)
> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> CC: gdb@sourceware.org
> > 
> >   . Where do I find the canonical results for my platform?
> 
> In theory one should not see any FAILS, and one should work on
> eliminating any KFAILS.

FWIW, I present below the failures and unexpected successes on the
system where I ran the test suite (an x86_64 Ubuntu box).  With the
exception of Ada/gnatmake related failures, they all seem to be real
problems.  In particular, maint.exp, corefile.exp, sigaltstack.exp,
sigbpt.exp, sigstep.exp, and staticthreads/watchthreads look
disturbing.  Does anyone else get them on a GNU/Linux system?

    UNSUPPORTED: gdb.ada/array_return.exp: Ada compilation failed: default_target_compile: Can't find gnatmake.
    UNSUPPORTED: gdb.ada/arrayidx.exp: Ada compilation failed: default_target_compile: Can't find gnatmake.
    UNSUPPORTED: gdb.ada/catch_ex.exp: Ada compilation failed: default_target_compile: Can't find gnatmake.
    UNSUPPORTED: gdb.ada/exec_changed.exp: Ada compilation failed: default_target_compile: Can't find gnatmake.
    UNSUPPORTED: gdb.ada/fixed_points.exp: Ada compilation failed: default_target_compile: Can't find gnatmake.
    UNSUPPORTED: gdb.ada/null_record.exp: Ada compilation failed: default_target_compile: Can't find gnatmake.
    UNSUPPORTED: gdb.ada/packed_array.exp: Ada compilation failed: default_target_compile: Can't find gnatmake.
    UNSUPPORTED: gdb.ada/start.exp: Ada compilation failed: default_target_compile: Can't find gnatmake.
    UNSUPPORTED: gdb.ada/watch_arg.exp: Ada compilation failed: default_target_compile: Can't find gnatmake.
    XFAIL: gdb.base/completion.exp: complete 'p "break1.'
    FAIL: gdb.base/corefile.exp: accessing mmapped data in core file (mapping address not found in core file)
    XFAIL: gdb.base/dbx.exp: func sum
    XFAIL: gdb.base/dbx.exp: func print_average
    KFAIL: gdb.base/ena-dis-br.exp: continue with ignore count, not stopped at bpt (PRMS: gdb/1689)
    UNSUPPORTED: gdb.base/ending-run.exp: continue after exit
    XFAIL: gdb.base/exprs.exp: truncate (void*) 0x00000000ffffffff + 1
    XFAIL: gdb.base/exprs.exp: truncate (void*) 0xffffffff00000000 - 1
    XFAIL: gdb.base/list.exp: list line 1 with unlimited listsize
    XFAIL: gdb.base/list.exp: list filename:function; wrong filename rejected
    XFAIL: gdb.base/macscp.exp: executable includes no macro debugging information
    FAIL: gdb.base/maint.exp: maint info sections .text
    KFAIL: gdb.base/printcmds.exp: ptype &*"foo" (PRMS: gdb/538)
    KFAIL: gdb.base/radix.exp: print 20.; expect 24; output radix 8 (PRMS: gdb/1715)
    KFAIL: gdb.base/radix.exp: print 20.; expect 14; output radix 16 (PRMS: gdb/1715)
    FAIL: gdb.base/sigaltstack.exp: finish from catch LEAF (the program exited)
    FAIL: gdb.base/sigaltstack.exp: finish to throw INNER
    FAIL: gdb.base/sigaltstack.exp: finish to catch INNER
    FAIL: gdb.base/sigaltstack.exp: finish from catch INNER
    FAIL: gdb.base/sigaltstack.exp: finish to OUTER
    FAIL: gdb.base/sigaltstack.exp: finish to catch MAIN
    FAIL: gdb.base/sigaltstack.exp: finish to MAIN
    FAIL: gdb.base/sigbpt.exp: stepi; stepi out of handler
    FAIL: gdb.base/sigbpt.exp: stepi bp before segv; stepi out of handler
    FAIL: gdb.base/sigbpt.exp: stepi bp at segv; stepi out of handler
    FAIL: gdb.base/sigbpt.exp: stepi bp before and at segv; stepi out of handler
    FAIL: gdb.base/siginfo.exp: step out of handler
    KFAIL: gdb.base/signals.exp: override SIGINT (PRMS: gdb/1707)
    FAIL: gdb.base/sigstep.exp: step from handler; leave handler
    FAIL: gdb.base/sigstep.exp: stepi from handleri; leave handler
    FAIL: gdb.base/sigstep.exp: stepi from handleri; leave signal trampoline
    FAIL: gdb.base/sigstep.exp: next from handler; leave handler
    FAIL: gdb.base/sigstep.exp: nexti from handleri; leave handler
    FAIL: gdb.base/sigstep.exp: nexti from handleri; leave signal trampoline
    FAIL: gdb.base/sigstep.exp: finish from handleri; leave handler (hit breakpoint again)
    FAIL: gdb.base/sigstep.exp: finish from handleri; leave signal trampoline
    FAIL: gdb.base/sigstep.exp: return from handleri; leave handler
    FAIL: gdb.base/sigstep.exp: return from handleri; leave signal trampoline
    FAIL: gdb.base/sigstep.exp: step to handler; resync
    FAIL: gdb.base/sigstep.exp: step to handler; performing step (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next to handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next to handler; performing next (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue to handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue to handler; performing continue (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step to handler entry; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step to handler entry; performing step (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next to handler entry; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next to handler entry; performing next (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue to handler entry; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue to handler entry; performing continue (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step over handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step over handler; performing step (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next over handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next over handler; performing next (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue over handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue over handler; performing continue (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step on breakpoint, to handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step on breakpoint, to handler; continue to infinite loop (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step on breakpoint, to handler; performing step (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next on breakpoint, to handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next on breakpoint, to handler; continue to infinite loop (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next on breakpoint, to handler; performing next (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler; continue to infinite loop (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler; performing continue (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step on breakpoint, to handler entry; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step on breakpoint, to handler entry; continue to infinite loop (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step on breakpoint, to handler entry; performing step (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next on breakpoint, to handler entry; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next on breakpoint, to handler entry; continue to infinite loop (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next on breakpoint, to handler entry; performing next (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler entry; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler entry; continue to infinite loop (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue on breakpoint, to handler entry; performing continue (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step on breakpoint, skip handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step on breakpoint, skip handler; continue to infinite loop (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: step on breakpoint, skip handler; performing step (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next on breakpoint, skip handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next on breakpoint, skip handler; continue to infinite loop (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: next on breakpoint, skip handler; performing next (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue on breakpoint, skip handler; resync (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue on breakpoint, skip handler; continue to infinite loop (the program is no longer running)
    FAIL: gdb.base/sigstep.exp: continue on breakpoint, skip handler; performing continue (the program is no longer running)
    KFAIL: gdb.base/solib-weak.exp: run to breakpoint - lib1 nodebug, lib2 debug, lib1 first (PRMS: gdb/1824)
    KFAIL: gdb.base/solib-weak.exp: run to breakpoint - lib1 debug, lib2 nodebug, lib2 first (PRMS: gdb/1824)
    KFAIL: gdb.base/watchpoint.exp: next after watch x (PRMS: gdb/38)
    KFAIL: gdb.cp/annota2.exp: watch triggered on a.x (PRMS: gdb/38)
    KFAIL: gdb.cp/annota3.exp: watch triggered on a.x (PRMS: gdb/38)
    KFAIL: gdb.cp/classes.exp: ptype obj_with_enum // unrecognized line type 1: PrivEnum priv_enum; (PRMS: gdb/57)
    KFAIL: gdb.cp/classes.exp: print (ClassWithEnum::PrivEnum) 42 (PRMS: gdb/1588)
    KFAIL: gdb.cp/classes.exp: print ('ClassWithEnum::PrivEnum') 42 (PRMS: gdb/57)
    KFAIL: gdb.cp/demangle.exp: lucid: __vtbl__3foo__vt_cc_main_ (PRMS: gdb/945)
    KFAIL: gdb.cp/inherit.exp: print g_D.A::a (PRMS: gdb/68)
    KFAIL: gdb.cp/inherit.exp: print g_D.A::x (PRMS: gdb/68)
    KFAIL: gdb.cp/inherit.exp: print g_E.A::a (PRMS: gdb/68)
    KFAIL: gdb.cp/inherit.exp: print g_E.A::x (PRMS: gdb/68)
    KFAIL: gdb.cp/inherit.exp: ptype g_D.A::a (PRMS: gdb/68)
    KFAIL: gdb.cp/inherit.exp: ptype g_D.A::x (PRMS: gdb/68)
    KFAIL: gdb.cp/inherit.exp: ptype g_E.A::a (PRMS: gdb/68)
    KFAIL: gdb.cp/inherit.exp: ptype g_E.A::x (PRMS: gdb/68)
    KFAIL: gdb.cp/local.exp: ptype InnerLocal::NestedInnerLocal (PRMS: gdb/482)
    KFAIL: gdb.cp/namespace.exp: ptype ::C::CClass (PRMS: gdb/1448)
    KFAIL: gdb.cp/namespace.exp: ptype ::C::CClass::NestedClass (PRMS: gdb/1448)
    KFAIL: gdb.cp/namespace.exp: ptype ::C::NestedClass (PRMS: gdb/1448)
    KFAIL: gdb.cp/namespace.exp: ptype ::C::OtherFileClass (PRMS: gdb/1448)
    KFAIL: gdb.cp/templates.exp: ptype T5<int> (PRMS: gdb/1111)
    KFAIL: gdb.cp/templates.exp: ptype T5<int> (PRMS: gdb/1111)
    KFAIL: gdb.cp/templates.exp: constructor breakpoint (PRMS: gdb/1062)
    KFAIL: gdb.cp/templates.exp: destructor breakpoint (PRMS: gdb/1112)
    KFAIL: gdb.cp/templates.exp: ptype fvpchar (PRMS: gdb/1512)
    KFAIL: gdb.cp/templates.exp: print Foo<volatile char *>::foo (PRMS: gdb/931)
    KFAIL: gdb.cp/templates.exp: print Foo<volatile char*>::foo (PRMS: gdb/931)
    KFAIL: gdb.cp/templates.exp: ptype quxint (PRMS: gdb/1512)
    KFAIL: gdb.cp/virtfunc.exp: print pEe->D::vg() (PRMS: gdb/1064)
    XFAIL: gdb.mi/mi-break.exp: break-insert -r operation
    XFAIL: gdb.mi/mi-break.exp: insert breakpoint with regexp callee2
    XFAIL: gdb.mi/mi-break.exp: insert breakpoint with regexp callee
    XFAIL: gdb.mi/mi-break.exp: insert breakpoint with regexp .*llee
    XFAIL: gdb.mi/mi-break.exp: list of breakpoints
    KFAIL: gdb.mi/mi-until.exp: until after while loop (went backwards) (PRMS: gdb/2104)
    XFAIL: gdb.mi/mi-var-block.exp: update inner_foo: should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi-var-block.exp: evaluate inner_foo: should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi-var-block.exp: update foo2: should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi-var-block.exp: update foo2 should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi-var-block.exp: update foo should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi-var-child.exp: update all vars psnp->next->long_ptr (and 1.long_ptr) changed
    XFAIL: gdb.mi/mi-var-cmd.exp: update all vars: changed FIXME
    XFAIL: gdb.mi/mi-var-display.exp: eval variable e
    XPASS: gdb.mi/mi-var-display.exp: eval variable anone
    XFAIL: gdb.mi/mi2-break.exp: break-insert -r operation
    XFAIL: gdb.mi/mi2-break.exp: insert breakpoint with regexp callee2
    XFAIL: gdb.mi/mi2-break.exp: insert breakpoint with regexp callee
    XFAIL: gdb.mi/mi2-break.exp: insert breakpoint with regexp .*llee
    XFAIL: gdb.mi/mi2-break.exp: list of breakpoints
    KFAIL: gdb.mi/mi2-console.exp: Hello message (PRMS: gdb/623)
    KFAIL: gdb.mi/mi2-until.exp: until after while loop (went backwards) (PRMS: gdb/2104)
    XFAIL: gdb.mi/mi2-var-block.exp: update inner_foo: should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi2-var-block.exp: evaluate inner_foo: should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi2-var-block.exp: update foo2: should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi2-var-block.exp: update foo2 should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi2-var-block.exp: update foo should be out of scope: KNOWN PROBLEM
    XFAIL: gdb.mi/mi2-var-child.exp: update all vars psnp->next->long_ptr (and 1.long_ptr) changed
    XFAIL: gdb.mi/mi2-var-cmd.exp: update all vars: changed FIXME
    XFAIL: gdb.mi/mi2-var-display.exp: eval variable e
    XPASS: gdb.mi/mi2-var-display.exp: eval variable anone
    UNSUPPORTED: gdb.objc/basicclass.exp: Couldn't compile ../.././gdb/testsuite/gdb.objc/basicclass.m: unrecognized error
    UNSUPPORTED: gdb.objc/nondebug.exp: Couldn't compile ../.././gdb/testsuite/gdb.objc/nondebug.m: unrecognized error
    UNSUPPORTED: gdb.objc/objcdecode.exp: Couldn't compile ../.././gdb/testsuite/gdb.objc/objcdecode.m: unrecognized error
    XFAIL: gdb.stabs/weird.exp: print struct constant
    XFAIL: gdb.stabs/weird.exp: array0 with strange index
    XFAIL: gdb.stabs/weird.exp: array1 with strange index
    XFAIL: gdb.stabs/weird.exp: whatis pointer_to_int_var
    XFAIL: gdb.stabs/weird.exp: whatis intp_var
    XFAIL: gdb.stabs/weird.exp: p common0var1
    XFAIL: gdb.stabs/weird.exp: p common0var2
    FAIL: gdb.threads/staticthreads.exp: running to main in runto
    FAIL: gdb.threads/staticthreads.exp: Continue to main's call of sem_post
    FAIL: gdb.threads/staticthreads.exp: handle SIG32 helps
    KFAIL: gdb.threads/tls.exp: info address me (PRMS: gdb/1294)
    FAIL: gdb.threads/watchthreads.exp: threaded watch loop
    FAIL: gdb.threads/watchthreads.exp: first watchpoint on args[1] hit
    FAIL: gdb.threads/watchthreads.exp: watchpoint on args[0] hit in thread
    FAIL: gdb.threads/watchthreads.exp: watchpoint on args[1] hit in thread
    FAIL: gdb.threads/watchthreads.exp: combination of threaded watchpoints = 30

Re: Test suite docs

[Eli Zaretskii]

> Date: Sat, 13 Jan 2007 16:05:02 +0100 (CET)
> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> CC: mark.kettenis@xs4all.nl, gdb@sourceware.org
> 
> Well, there is the gdb-testers mailing list that some people send
> their testresults to.

Thanks for the pointer.

Re: Test suite docs

[Daniel Jacobowitz]

On Sat, Jan 13, 2007 at 05:07:42PM +0200, Eli Zaretskii wrote:
> > >   . Where do I find the canonical results for my platform?
> > 
> > In theory one should not see any FAILS, and one should work on
> > eliminating any KFAILS.

I agree with Mark and I have been working extensively over the last
month just to eliminate FAILs on one single platform - we have not
cared for the testsuite well, and it's a fantastically hard thing
to do because of all the variables.

In practice, I always do exactly what Joel described.  I keep the
last gdb.sum around and diff them.  I just use diff.  At work we
have some rather more sophisticated scripts to compare .sum files
but I'm not familiar with them.

> FWIW, I present below the failures and unexpected successes on the
> system where I ran the test suite (an x86_64 Ubuntu box).  With the
> exception of Ada/gnatmake related failures, they all seem to be real
> problems.  In particular, maint.exp, corefile.exp, sigaltstack.exp,
> sigbpt.exp, sigstep.exp, and staticthreads/watchthreads look
> disturbing.  Does anyone else get them on a GNU/Linux system?

I would not be disturbed by any of them.

>     FAIL: gdb.base/corefile.exp: accessing mmapped data in core file (mapping address not found in core file)

A change in behavior of the Linux kernel.  David Miller and I talked
about it again a week or two ago.  I know how to make the test pass
again but I'm waiting to see what we do to the kernel behavior first.

>     FAIL: gdb.base/maint.exp: maint info sections .text

No idea.

>     FAIL: gdb.base/sigaltstack.exp: finish from catch LEAF (the program exited)

The frame_unwind_address_in_block work I was discussing with Mark is
for this test and the related signal tests...

>     FAIL: gdb.base/sigbpt.exp: stepi; stepi out of handler

...except that some of them are caused by a bug in glibc, which was
just recently fixed; your system's probably has the bug.  It's not
a serious bug in practice, for anything except the gdb testsuite.

>     FAIL: gdb.threads/staticthreads.exp: running to main in runto
>     FAIL: gdb.threads/staticthreads.exp: Continue to main's call of sem_post
>     FAIL: gdb.threads/staticthreads.exp: handle SIG32 helps

I have submitted potential patches for this bug to the glibc
maintainers periodically for over a year.  I've about given up.

>     FAIL: gdb.threads/watchthreads.exp: threaded watch loop
>     FAIL: gdb.threads/watchthreads.exp: first watchpoint on args[1] hit
>     FAIL: gdb.threads/watchthreads.exp: watchpoint on args[0] hit in thread
>     FAIL: gdb.threads/watchthreads.exp: watchpoint on args[1] hit in thread
>     FAIL: gdb.threads/watchthreads.exp: combination of threaded watchpoints = 30

GDB doesn't support this feature.  We added the testcase but not the
rest of the patch.  As I reduce the list of failures on my first target
platform to almost none this is coming back to the top of my list.
I asked someone from Red Hat to submit an updated version but
got no response; I'll poke them again later.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Test suite docs

[Mark Kettenis]

> Date: Sat, 13 Jan 2007 17:07:42 +0200
> From: Eli Zaretskii <eliz@gnu.org>

> FWIW, I present below the failures and unexpected successes on the
> system where I ran the test suite (an x86_64 Ubuntu box).  With the
> exception of Ada/gnatmake related failures, they all seem to be real
> problems.  In particular, maint.exp, corefile.exp, sigaltstack.exp,
> sigbpt.exp, sigstep.exp, and staticthreads/watchthreads look
> disturbing.  Does anyone else get them on a GNU/Linux system?
> 
>     FAIL: gdb.base/corefile.exp: accessing mmapped data in core file (mapping address not found in core file)

I don't think I've ever seen this one myself.  The problem could be
that your ulimit is set such that your system creates truncated core
files.

>     FAIL: gdb.base/maint.exp: maint info sections .text

Never seen this one.

>     FAIL: gdb.base/sigaltstack.exp: finish from catch LEAF (the program exited)
>     FAIL: gdb.base/sigaltstack.exp: finish to throw INNER
>     FAIL: gdb.base/sigaltstack.exp: finish to catch INNER
>     FAIL: gdb.base/sigaltstack.exp: finish from catch INNER
>     FAIL: gdb.base/sigaltstack.exp: finish to OUTER
>     FAIL: gdb.base/sigaltstack.exp: finish to catch MAIN
>     FAIL: gdb.base/sigaltstack.exp: finish to MAIN
>     FAIL: gdb.base/sigbpt.exp: stepi; stepi out of handler
>     FAIL: gdb.base/sigbpt.exp: stepi bp before segv; stepi out of handler
>     FAIL: gdb.base/sigbpt.exp: stepi bp at segv; stepi out of handler
>     FAIL: gdb.base/sigbpt.exp: stepi bp before and at segv; stepi out of handler
>     FAIL: gdb.base/siginfo.exp: step out of handler
>     KFAIL: gdb.base/signals.exp: override SIGINT (PRMS: gdb/1707)
>     FAIL: gdb.base/sigstep.exp: step from handler; leave handler
>     FAIL: gdb.base/sigstep.exp: stepi from handleri; leave handler
>     FAIL: gdb.base/sigstep.exp: stepi from handleri; leave signal trampoline
>     FAIL: gdb.base/sigstep.exp: next from handler; leave handler
>     ...

These are fairly common.  I think we know how to fix them properly,
although this might only work for some Linux systems.

>     UNSUPPORTED: gdb.objc/basicclass.exp: Couldn't compile ../.././gdb/testsuite/gdb.objc/basicclass.m: unrecognized error
>     UNSUPPORTED: gdb.objc/nondebug.exp: Couldn't compile ../.././gdb/testsuite/gdb.objc/nondebug.m: unrecognized error
>     UNSUPPORTED: gdb.objc/objcdecode.exp: Couldn't compile ../.././gdb/testsuite/gdb.objc/objcdecode.m: unrecognized error

Could be that the Objective C compiler isn't installed on your system.
If that's the cause of the problem, we should improve the tests such
that they print something more obvious.

>     FAIL: gdb.threads/staticthreads.exp: running to main in runto
>     FAIL: gdb.threads/staticthreads.exp: Continue to main's call of sem_post
>     FAIL: gdb.threads/staticthreads.exp: handle SIG32 helps
>     KFAIL: gdb.threads/tls.exp: info address me (PRMS: gdb/1294)
>     FAIL: gdb.threads/watchthreads.exp: threaded watch loop
>     FAIL: gdb.threads/watchthreads.exp: first watchpoint on args[1] hit
>     FAIL: gdb.threads/watchthreads.exp: watchpoint on args[0] hit in thread
>     FAIL: gdb.threads/watchthreads.exp: watchpoint on args[1] hit in thread
>     FAIL: gdb.threads/watchthreads.exp: combination of threaded watchpoints = 30

The threads stuff only works if you have the right combination of
kernel and glibc.  I'm not aware of any "official" Linux distribution
that gets all the bits right.

This is what makes it so difficult to get the testsuite FAIL-free.
The sheer number of Linux distributions out there that all appear as
i386-unknown-linux-gnu to us, makes it very hard to selectively XFAIL
tests.  On systems with a more formal release engineering procedure,
this would be much easier since for example m88k-unknown-openbsd3.9
refers to a single official release, and if I know somethings is
broken there but fixed in m88k-unknown-openbsd4.0 I can put the proper
setup_xfail in the test.

That said, even on OpenBSD I have been pretty lazy with putting in the
proper setup_xfails.  Not many people besides me actually build the
FSF release of gdb on OpenBSD, since I make sure the gdb shipped with
OpenBSD releases is fairly up to date and works well.

Mark

Re: Test suite docs

[Daniel Jacobowitz]

On Sat, Jan 13, 2007 at 04:33:24PM +0100, Mark Kettenis wrote:
> >     UNSUPPORTED: gdb.objc/basicclass.exp: Couldn't compile ../.././gdb/testsuite/gdb.objc/basicclass.m: unrecognized error
> >     UNSUPPORTED: gdb.objc/nondebug.exp: Couldn't compile ../.././gdb/testsuite/gdb.objc/nondebug.m: unrecognized error
> >     UNSUPPORTED: gdb.objc/objcdecode.exp: Couldn't compile ../.././gdb/testsuite/gdb.objc/objcdecode.m: unrecognized error
> 
> Could be that the Objective C compiler isn't installed on your system.
> If that's the cause of the problem, we should improve the tests such
> that they print something more obvious.

The patch Nathan had to revert should improve that - when we get back
in a week I'm sure he'll fix it up so he can reapply it.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Test suite docs

[Eli Zaretskii]

> Date: Sat, 13 Jan 2007 16:33:24 +0100 (CET)
> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> CC: gdb@sourceware.org
> > 
> >     FAIL: gdb.base/corefile.exp: accessing mmapped data in core file (mapping address not found in core file)
> 
> I don't think I've ever seen this one myself.  The problem could be
> that your ulimit is set such that your system creates truncated core
> files.

"ulimit -c" is set to 200MB.

> This is what makes it so difficult to get the testsuite FAIL-free.
> The sheer number of Linux distributions out there that all appear as
> i386-unknown-linux-gnu to us, makes it very hard to selectively XFAIL
> tests.

Maybe we shouldn't test so many kernel-related issues.

> That said, even on OpenBSD I have been pretty lazy with putting in the
> proper setup_xfails.

One more reason to have the canonical results recorded somewhere.

Re: Test suite docs

[Michael Snyder]

On Sat, 2007-01-13 at 17:28 +0400, Joel Brobecker wrote:
> 
> I cannot say that I am a dejagnu expert. In fact, I cannot
> say that I like this technology and the less time I spend fiddling
> with it, the better I feel.

Sorry to come to this thread so late, but I'd just like to say
that dejagnu is like democracy -- it's the worst system imaginable, 
except for all the others.

I say so as someone who used to have to maintain a test suite
for a debugger without benefit of expect or dejagnu.  You have
no idea what a world of difference it makes, just to be able to
test with regular expressions!

I don't mean to invalidate any of the complaints about either
the testsuite or its documentation.  Especially the documentation.