[RFC] Framework for selectively omitting individual testsuite tests or directories for local use only

[RFC] Framework for selectively omitting individual testsuite tests or directories for local use only

[Fred Fish]

Very often when I'm working with a gdb that has very poor testsuite
results, I want to set things up so that I can selectively omit
individual troublesome tests or even entire directories of tests.

That way the I can get a completely clean gdb testsuite run, and then
start tackling the troublesome tests one a time, without having to
filter out the hundreds of other failures I haven't yet gotten around
to fixing.  Also, a clean testsuite runs *MUCH* faster than one that
has hundreds of timeouts or other such issues.  For example, the
current sh-elf gdb seems to hang forever if you don't avoid running
the gdbtk tests.

Dejagnu supports "ignoretests" and "ignoredirs" lists of specific
individual tests or directories to ignore.  Below is a patch that I
frequently apply locally to any gdb tree I'm working on.

I'd like to propose that we add it, or something like it, to the
testsuite strictly for use as a debugging aid.  It's not intended to
become a permanent way for various configurations to hide their more
problematic test cases, just a framework for developers to use
locally.

Comments?

-Fred


Index: Makefile.in
===================================================================
RCS file: /cvs/cvsfiles/devo/gdb/testsuite/Makefile.in,v
retrieving revision 1.66
diff -c -p -r1.66 Makefile.in
*** Makefile.in	9 Sep 2003 21:03:53 -0000	1.66
--- Makefile.in	2 Feb 2004 23:57:50 -0000
*************** host_canonical = @host@
*** 31,36 ****
--- 31,39 ----
  target_canonical = @target@
  target_cpu = @gdb_target_cpu@
  
+ ignoredirs = @ignoredirs@
+ ignoretests = @ignoretests@
+ 
  SHELL = @SHELL@
  EXEEXT = @EXEEXT@
  SUBDIRS = @subdirs@
*************** site.exp: ./config.status Makefile
*** 111,116 ****
--- 114,121 ----
  	@echo "set build_triplet ${build_canonical}" >> ./tmp0
  	@echo "set srcdir ${srcdir}" >> ./tmp0
  	@echo "set tool gdb" >> ./tmp0
+ 	@echo "lappend ignoredirs ${ignoredirs}" >>./tmp0
+ 	@echo "lappend ignoretests ${ignoretests}" >>./tmp0
  	@echo "## All variables above are generated by configure. Do Not Edit ##" >> ./tmp0
  		@cat ./tmp0 > site.exp
  	@cat site.bak | sed \
Index: configure.in
===================================================================
RCS file: /cvs/cvsfiles/devo/gdb/testsuite/configure.in,v
retrieving revision 1.59.6.1
diff -c -p -r1.59.6.1 configure.in
*** configure.in	22 Oct 2003 00:34:16 -0000	1.59.6.1
--- configure.in	2 Feb 2004 23:57:50 -0000
*************** esac
*** 77,84 ****
--- 77,102 ----
  # Add gdbtk tests when appropriate.
  if test $enable_gdbtk = yes; then
    configdirs="$configdirs gdb.gdbtk"
+ else
+   ignoredirs="${ignoredirs} gdb.gdbtk"
  fi
  
+ # Suppress some individual tests or groups of tests for specific targets.
+ 
+ ignoredirs=""
+ case "${target}" in
+  xxx-yyy-zzz)
+ 	ignoredirs="${ignoredirs} gdb.cp"
+ 	;;
+ esac
+ 
+ ignoretests=""
+ case "${target}" in
+  xxx-yyy-zzz)
+ 	ignoretests="${ignoretests} fileio.exp"
+ 	;;
+ esac
+ 
  # Enable shared libraries.
  AC_ARG_ENABLE(shared,
  [  --enable-shared         build shared libraries [deault=yes]],,
*************** AC_CHECK_HEADERS(pthread.h)
*** 112,117 ****
--- 130,137 ----
  AC_EXEEXT
  
  AC_CONFIG_SUBDIRS($configdirs)
+ AC_SUBST(ignoredirs)
+ AC_SUBST(ignoretests)
  AC_OUTPUT([Makefile gdb.arch/Makefile gdb.asm/Makefile gdb.base/Makefile \
    gdb.cp/Makefile gdb.disasm/Makefile gdb.java/Makefile gdb.mi/Makefile \
    gdb.objc/Makefile gdb.threads/Makefile gdb.trace/Makefile])

Re: [RFC] Framework for selectively omitting individual testsuite tests or directories for local use only

[Daniel Jacobowitz]

On Mon, Feb 02, 2004 at 05:13:43PM -0700, Fred Fish wrote:
> Very often when I'm working with a gdb that has very poor testsuite
> results, I want to set things up so that I can selectively omit
> individual troublesome tests or even entire directories of tests.
> 
> That way the I can get a completely clean gdb testsuite run, and then
> start tackling the troublesome tests one a time, without having to
> filter out the hundreds of other failures I haven't yet gotten around
> to fixing.  Also, a clean testsuite runs *MUCH* faster than one that
> has hundreds of timeouts or other such issues.  For example, the
> current sh-elf gdb seems to hang forever if you don't avoid running
> the gdbtk tests.
> 
> Dejagnu supports "ignoretests" and "ignoredirs" lists of specific
> individual tests or directories to ignore.  Below is a patch that I
> frequently apply locally to any gdb tree I'm working on.
> 
> I'd like to propose that we add it, or something like it, to the
> testsuite strictly for use as a debugging aid.  It's not intended to
> become a permanent way for various configurations to hide their more
> problematic test cases, just a framework for developers to use
> locally.
> 
> Comments?

To be honest, I don't see the point of adding this.  Take a look at
what your change does:

> *************** site.exp: ./config.status Makefile
> *** 111,116 ****
> --- 114,121 ----
>   	@echo "set build_triplet ${build_canonical}" >> ./tmp0
>   	@echo "set srcdir ${srcdir}" >> ./tmp0
>   	@echo "set tool gdb" >> ./tmp0
> + 	@echo "lappend ignoredirs ${ignoredirs}" >>./tmp0
> + 	@echo "lappend ignoretests ${ignoretests}" >>./tmp0
>   	@echo "## All variables above are generated by configure. Do Not Edit ##" >> ./tmp0
>   		@cat ./tmp0 > site.exp
>   	@cat site.bak | sed \

OK... there is a much easier way.

drow@nevyn:~% echo $DEJAGNU
/home/drow/.dejagnu/site.exp
drow@nevyn:~% cat >> $DEJAGNU <<EOF
if { "$tool" == "gdb" } {
  lappend ignoredirs gdb.cp
}
EOF

$target_alias is there too.  Even $srcdir should be available by this
point, giving you great freedom.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [RFC] Framework for selectively omitting individual testsuite tests or directories for local use only

[Daniel Jacobowitz]

On Mon, Feb 02, 2004 at 05:13:43PM -0700, Fred Fish wrote:
> Dejagnu supports "ignoretests" and "ignoredirs" lists of specific
> individual tests or directories to ignore.  Below is a patch that I
> frequently apply locally to any gdb tree I'm working on.

Oh, and a note: thanks for mentioning this.  Learn a cryptic feature of
dejagnu every month, that's my motto.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer