From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 2301 invoked by alias); 13 Jan 2007 10:26:21 -0000 Received: (qmail 2293 invoked by uid 22791); 13 Jan 2007 10:26:20 -0000 X-Spam-Check-By: sourceware.org Received: from romy.inter.net.il (HELO romy.inter.net.il) (192.114.186.66) by sourceware.org (qpsmtpd/0.31) with ESMTP; Sat, 13 Jan 2007 10:26:15 +0000 Received: from HOME-C4E4A596F7 (IGLD-80-230-11-156.inter.net.il [80.230.11.156]) by romy.inter.net.il (MOS 3.7.3-GA) with ESMTP id GUR64741 (AUTH halo1); Sat, 13 Jan 2007 12:26:11 +0200 (IST) Date: Sat, 13 Jan 2007 10:26:00 -0000 Message-Id: From: Eli Zaretskii To: gdb@sourceware.org Subject: Test suite docs Reply-to: Eli Zaretskii X-IsSubscribed: yes Mailing-List: contact gdb-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-owner@sourceware.org X-SW-Source: 2007-01/txt/msg00221.txt.bz2 This is my first experience running the test suite, and it is quite frustrating. All I wanted was to run the tests before and after a change I'm about to suggest on gdb-patches. Unfortunately, I ended up wasting my scarce free time on figuring out several gory details. While I'm no newcomer to Free Software, and I expect to spend some time figuring out things on my own when it comes to using a new piece of software, the test suite makes it exceptionally hard, IMHO. Some of the reasons are out of our control: the tests use several software packages (Dejagnu which uses Expect which uses TCL), so answers are potentially scattered across several unrelated packages, and the fact that none of them has GNU standard Info manuals (or at least I couldn't find them on fencepost.gnu.org) doesn't help. But that's just one more reason to have a good user-level documentation in GDB to help overcome these difficulties. I try to summarize the problems I needed to solve below. Could someone experienced in running the test suite please consider writing up some user-friendly instructions for first-time users of the test suite? I volunteer to do whatever it takes to add them to the appropriate parts of the GDB documentation, but I need someone ``in the know'' of the test suite internals to provide the content. I did find the few words that gdb/README says about running the tests, and the brief section in gdbint.texinfo; however, they still fall short of what I needed to know, see below. TIA Here are the questions I couldn't find answers to: . Where do I find the canonical results for my platform? People talk about XFAILs and ``unexpected failures'', but there seems to be no place to consult the expected results for all the tests and see if what you get is okay or not. The test suite prints a summary of the tests, but how do I find out what are those ``unexpected successes'' and ``expected failures''? What are those XPASS, XFAIL, UNTESTED, and other indications displayed while the suite runs? . How do I compare two runs? If diff'ing testsuite/gdb.sum is the right way, it seems to not be documented anywhere, and gdb.sum doesn't seem to be preserved across runs, so one must manually copy it to avoid overwriting it. Am I missing something? . How does one disable a specific test? Suppose some test takes an exceptionally long time -- how do I run the suite without it? gdbint.texinfo tells how to _run_ a specific test or a short list of test, but that method is not practical for _disabling_ a small number of tests and running all the rest. gdbint.texinfo also says something about not ``adding expected failures lightly'', but keeps silent about how does one make a test an expected failure. In general, the language in that section of gdbint assumes the reader is already an experienced writer of Dejagnu tests, which is not a good assumption for a manual. . Where do I look for definitions and docs of specific subroutines that the *.exp files use? Suppose I've found out about the set_xfail subroutine, and want to look into it and see whether it can be used to disable a test: where do I look for its definition or its documentation? There's the test's *.exp file, there's the testsuite/lib/ subdirectory (which, btw, is only mentioned in passing in gdbint.texinfo), and then there are Dejagnu and Expect and TCL. Could we please have a list of files or directories to look in, and a list of documentation files to browse?