RFC: Additional testsuite alternative

RFC: Additional testsuite alternative

[Daniel Jacobowitz]

OK, the files were on a different machine, so this wasn't a couple minutes
after I promised it this morning.  Can't win 'em all.

I very much admire the way GCC's testsuite works.  You don't have to write
code for new tests; you can just drop them in.  Binutils has a two-file
version (GCC's is all one-file, using the DejaGNU "dg" harness) that's
comparable.

GDB testing is more complicated, but I think that for a significant number
of tests we can get the same result.  Some more complex tests will still
want to be their own .exp files, of course.  Here's how what I've
implemented so far looks.

Source file two.cc:
===
struct OneStruct {
  int simple;
};
struct OneStruct StrOne;
const struct OneStruct *ConstStrOnePtr;

int FunctionWithPtrs (const struct OneStruct *one, const int *two)
{
  return 0;
}

int
main ()
{
  return 0;
}
===

Source file two.x:
===
#compile two.cc two.exe executable debug
#runto main
#test "ptype StrOne"
type = class OneStruct {
  public:
    int simple;
[synthetic OneStruct]}
#test "ptype ConstStrOnePtr"
type = const class OneStruct {
  public:
    int simple;
[synthetic OneStruct]} \*
===

Lines starting with "#[a-z]" are commands.  The ones we have so far (since
they were all I needed for the test I was writing at the time :) are:

#compile <source> <binfile> <type> <options>

Works just like a call to gdb_compile, but the source is relative to the
location of the .x file.

#runto <function>

Calls either runto or runto_main depending on the argument.

#test [-const] "command"

Sends "command" to GDB and watches for the response, which is a series of
lines not starting with #.  If -const is specified then consts (volatiles,
etc.) will be left alone; otherwise they are made optional iff the debug
format is stabs.  Later I'll refine it to "iff the debug format is stabs and
the compiler does not produce const type qualifiers in its stabs".

The string [synthetic ClassName] is special and expands to a regex (iff
stabs) that matches the synthesized constructors and assignment operator
that GCC emits when using stabs (simplisticly; it's not meant to be perfect,
just to reduce clutter in testing simple structures, and I haven't thought
of a way to properly prevent the synthesized methods from showing up.  I
think I just did, though, and if it works this construct will die.)


Obviously the syntax isn't complete.  It doesn't support comments yet but
that's easy.  It's not set in stone; I'd kind of like to use something other
than '#' so that I can use '#' for comments.  Maybe '%'?


The general intention is that this makes it easier to write tests, and
drastically easier to read them and figure out what the expected output is.

Thoughts?  Is this interesting to anyone else?

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: Additional testsuite alternative

[Daniel Jacobowitz]

Does anyone have any reaction to this?  Fernando, how would you feel
about adding the harness for this to the testsuite?

The background is that I'm probably going to change the behaviour (and
definitely going to increase the visibility) of c_print_type, so I want
to have some unit tests written for it first.

On Mon, Sep 16, 2002 at 03:25:46PM -0400, Daniel Jacobowitz wrote:
> OK, the files were on a different machine, so this wasn't a couple minutes
> after I promised it this morning.  Can't win 'em all.
> 
> I very much admire the way GCC's testsuite works.  You don't have to write
> code for new tests; you can just drop them in.  Binutils has a two-file
> version (GCC's is all one-file, using the DejaGNU "dg" harness) that's
> comparable.
> 
> GDB testing is more complicated, but I think that for a significant number
> of tests we can get the same result.  Some more complex tests will still
> want to be their own .exp files, of course.  Here's how what I've
> implemented so far looks.
> 
> Source file two.cc:
> ===
> struct OneStruct {
>   int simple;
> };
> struct OneStruct StrOne;
> const struct OneStruct *ConstStrOnePtr;
> 
> int FunctionWithPtrs (const struct OneStruct *one, const int *two)
> {
>   return 0;
> }
> 
> int
> main ()
> {
>   return 0;
> }
> ===
> 
> Source file two.x:
> ===
> #compile two.cc two.exe executable debug
> #runto main
> #test "ptype StrOne"
> type = class OneStruct {
>   public:
>     int simple;
> [synthetic OneStruct]}
> #test "ptype ConstStrOnePtr"
> type = const class OneStruct {
>   public:
>     int simple;
> [synthetic OneStruct]} \*
> ===
> 
> Lines starting with "#[a-z]" are commands.  The ones we have so far (since
> they were all I needed for the test I was writing at the time :) are:
> 
> #compile <source> <binfile> <type> <options>
> 
> Works just like a call to gdb_compile, but the source is relative to the
> location of the .x file.
> 
> #runto <function>
> 
> Calls either runto or runto_main depending on the argument.
> 
> #test [-const] "command"
> 
> Sends "command" to GDB and watches for the response, which is a series of
> lines not starting with #.  If -const is specified then consts (volatiles,
> etc.) will be left alone; otherwise they are made optional iff the debug
> format is stabs.  Later I'll refine it to "iff the debug format is stabs and
> the compiler does not produce const type qualifiers in its stabs".
> 
> The string [synthetic ClassName] is special and expands to a regex (iff
> stabs) that matches the synthesized constructors and assignment operator
> that GCC emits when using stabs (simplisticly; it's not meant to be perfect,
> just to reduce clutter in testing simple structures, and I haven't thought
> of a way to properly prevent the synthesized methods from showing up.  I
> think I just did, though, and if it works this construct will die.)
> 
> 
> Obviously the syntax isn't complete.  It doesn't support comments yet but
> that's easy.  It's not set in stone; I'd kind of like to use something other
> than '#' so that I can use '#' for comments.  Maybe '%'?
> 
> 
> The general intention is that this makes it easier to write tests, and
> drastically easier to read them and figure out what the expected output is.
> 
> Thoughts?  Is this interesting to anyone else?
> 
> -- 
> Daniel Jacobowitz
> MontaVista Software                         Debian GNU/Linux Developer
> 

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: Additional testsuite alternative

[David Carlton]

On Thu, 26 Sep 2002 14:21:32 -0400, Daniel Jacobowitz <drow@mvista.com> said:

> Does anyone have any reaction to this?

I like the basic idea; anything to make it easier to add tests is a
boon.  (Especially now that I've started following the refactoring
people's suggestion of making small, semantics-preserving changes to
code and testing frequently.)

David Carlton
carlton@math.stanford.edu

Re: RFC: Additional testsuite alternative

[Fernando Nasser]

Daniel,

I don't think something like this would be of general use.  The .exp 
files have the full power of a script language and nothing can beat 
that.  Compiler tests are pretty much different from debugger tests, 
because debuggers are interactive beasts.

But as a special harness to drive C++ tests I think it is a good idea. 
The majority of tests deal with checking for some formatted output of a 
C++ construct and maybe the full power of scripting is not needed. 
Maybe it can even be adapted to other languages where what is being 
tested is of similar nature. There is a precedent already in that the 
gdbtk tests use their own spec files (.test).

Anyway, I suggest that you do not try and make it too general, but just 
something that is capable of simplifying these types of C++ tests.  Use 
.exp for the non-trivial tests.

One more question:  You still need a minimum .exp file, I believe, which 
is what runtest will find and try to run.  It is also part of what 
identify tests in the results and so one.

Regards,
Fernando


Daniel Jacobowitz wrote:
> Does anyone have any reaction to this?  Fernando, how would you feel
> about adding the harness for this to the testsuite?
> 
> The background is that I'm probably going to change the behaviour (and
> definitely going to increase the visibility) of c_print_type, so I want
> to have some unit tests written for it first.
> 
> On Mon, Sep 16, 2002 at 03:25:46PM -0400, Daniel Jacobowitz wrote:
> 
>>OK, the files were on a different machine, so this wasn't a couple minutes
>>after I promised it this morning.  Can't win 'em all.
>>
>>I very much admire the way GCC's testsuite works.  You don't have to write
>>code for new tests; you can just drop them in.  Binutils has a two-file
>>version (GCC's is all one-file, using the DejaGNU "dg" harness) that's
>>comparable.
>>
>>GDB testing is more complicated, but I think that for a significant number
>>of tests we can get the same result.  Some more complex tests will still
>>want to be their own .exp files, of course.  Here's how what I've
>>implemented so far looks.
>>
>>Source file two.cc:
>>===
>>struct OneStruct {
>>  int simple;
>>};
>>struct OneStruct StrOne;
>>const struct OneStruct *ConstStrOnePtr;
>>
>>int FunctionWithPtrs (const struct OneStruct *one, const int *two)
>>{
>>  return 0;
>>}
>>
>>int
>>main ()
>>{
>>  return 0;
>>}
>>===
>>
>>Source file two.x:
>>===
>>#compile two.cc two.exe executable debug
>>#runto main
>>#test "ptype StrOne"
>>type = class OneStruct {
>>  public:
>>    int simple;
>>[synthetic OneStruct]}
>>#test "ptype ConstStrOnePtr"
>>type = const class OneStruct {
>>  public:
>>    int simple;
>>[synthetic OneStruct]} \*
>>===
>>
>>Lines starting with "#[a-z]" are commands.  The ones we have so far (since
>>they were all I needed for the test I was writing at the time :) are:
>>
>>#compile <source> <binfile> <type> <options>
>>
>>Works just like a call to gdb_compile, but the source is relative to the
>>location of the .x file.
>>
>>#runto <function>
>>
>>Calls either runto or runto_main depending on the argument.
>>
>>#test [-const] "command"
>>
>>Sends "command" to GDB and watches for the response, which is a series of
>>lines not starting with #.  If -const is specified then consts (volatiles,
>>etc.) will be left alone; otherwise they are made optional iff the debug
>>format is stabs.  Later I'll refine it to "iff the debug format is stabs and
>>the compiler does not produce const type qualifiers in its stabs".
>>
>>The string [synthetic ClassName] is special and expands to a regex (iff
>>stabs) that matches the synthesized constructors and assignment operator
>>that GCC emits when using stabs (simplisticly; it's not meant to be perfect,
>>just to reduce clutter in testing simple structures, and I haven't thought
>>of a way to properly prevent the synthesized methods from showing up.  I
>>think I just did, though, and if it works this construct will die.)
>>
>>
>>Obviously the syntax isn't complete.  It doesn't support comments yet but
>>that's easy.  It's not set in stone; I'd kind of like to use something other
>>than '#' so that I can use '#' for comments.  Maybe '%'?
>>
>>
>>The general intention is that this makes it easier to write tests, and
>>drastically easier to read them and figure out what the expected output is.
>>
>>Thoughts?  Is this interesting to anyone else?
>>
>>-- 
>>Daniel Jacobowitz
>>MontaVista Software                         Debian GNU/Linux Developer
>>
> 
> 


-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: RFC: Additional testsuite alternative

[Daniel Jacobowitz]

On Thu, Sep 26, 2002 at 04:20:52PM -0400, Fernando Nasser wrote:
> Daniel,
> 
> I don't think something like this would be of general use.  The .exp 
> files have the full power of a script language and nothing can beat 
> that.  Compiler tests are pretty much different from debugger tests, 
> because debuggers are interactive beasts.
> 
> But as a special harness to drive C++ tests I think it is a good idea. 
> The majority of tests deal with checking for some formatted output of a 
> C++ construct and maybe the full power of scripting is not needed. 
> Maybe it can even be adapted to other languages where what is being 
> tested is of similar nature. There is a precedent already in that the 
> gdbtk tests use their own spec files (.test).
> 
> Anyway, I suggest that you do not try and make it too general, but just 
> something that is capable of simplifying these types of C++ tests.  Use 
> .exp for the non-trivial tests.

OK... I may just use it for type formatting for now, then.  That's and
simple expression printing is what it is best suited for.

> One more question:  You still need a minimum .exp file, I believe, which 
> is what runtest will find and try to run.  It is also part of what 
> identify tests in the results and so one.

Yeah.  I've been just using the harness as the .exp file; I'll keep it
that way for now unless this is needed somewhere else.

I'll submit the harness in a couple of days then.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: Additional testsuite alternative

[Andrew Cagney]

> Source file two.cc:
> ===
> struct OneStruct {
>   int simple;
> };
> struct OneStruct StrOne;
> const struct OneStruct *ConstStrOnePtr;
> 
> int FunctionWithPtrs (const struct OneStruct *one, const int *two)
> {
>   return 0;
> }
> 
> int
> main ()
> {
>   return 0;
> }

The nice thing about GCC's framework is that it is a single C file.  Is 
the same possible here?  Comments would indicate where to set 
breakpoints and what values to print.

The test then involves running the program, and for each breakpoint, 
printing and checking the output.

(How would this .x file handle regular expressions?  That is the one 
thing I never figured out with the GCC framework.)

enjoy,
Andrew

Re: RFC: Additional testsuite alternative

[Daniel Jacobowitz]

On Thu, Sep 26, 2002 at 04:47:53PM -0400, Andrew Cagney wrote:
> 
> >Source file two.cc:
> >===
> >struct OneStruct {
> >  int simple;
> >};
> >struct OneStruct StrOne;
> >const struct OneStruct *ConstStrOnePtr;
> >
> >int FunctionWithPtrs (const struct OneStruct *one, const int *two)
> >{
> >  return 0;
> >}
> >
> >int
> >main ()
> >{
> >  return 0;
> >}
> 
> The nice thing about GCC's framework is that it is a single C file.  Is 
> the same possible here?  Comments would indicate where to set 
> breakpoints and what values to print.
> 
> The test then involves running the program, and for each breakpoint, 
> printing and checking the output.

I thought about it and decided it wasn't worthwhile.  That works on
GCC: You run GCC, you get output based on particular lines.  GDB has a
source and then a session.  So I have two files; one is source, the
other is the session.

> (How would this .x file handle regular expressions?  That is the one 
> thing I never figured out with the GCC framework.)

It just works :)  It's a little awkward for complex expressions but
works just fine for the simple cases I needed it for.  The response to
a #test directive is a regex.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: Additional testsuite alternative

[Jim Blandy]

It's a pity to implement a little language in Tcl.  People always
introduce little languages, saying, "I just need a very simple
language for this one situation; I don't need anything complex."
And then, minor feature by minor feature, it grows, until you've got
the Bourne shell.

Doesn't the Tcl we're using nowadays have namespaces?  Can't we make
.x files just Tcl scripts that run in a namespace that has all the
right commands in it?  Then Daniel's example:

    #compile two.cc two.exe executable debug
    #runto main
    #test "ptype StrOne"
    type = class OneStruct {
      public:
        int simple;
    [synthetic OneStruct]}
    #test "ptype ConstStrOnePtr"
    type = const class OneStruct {
      public:
        int simple;
    [synthetic OneStruct]} \*

could just be:

    compile two.cc two.exe executable debug
    runto main
    test "ptype StrOne" {
    type = class OneStruct {
      public:
        int simple;
    [synthetic OneStruct]}
    }
    test "ptype ConstStrOnePtr" {
    type = const class OneStruct {
      public:
        int simple;
    [synthetic OneStruct]} \*
    }

And you've got all your quoting stuff, comments, substitution, etc for
free.  Designed carefully, not grown incrementally.

Re: RFC: Additional testsuite alternative

[Daniel Jacobowitz]

On Thu, Sep 26, 2002 at 03:50:07PM -0500, Jim Blandy wrote:
> 
> It's a pity to implement a little language in Tcl.  People always
> introduce little languages, saying, "I just need a very simple
> language for this one situation; I don't need anything complex."
> And then, minor feature by minor feature, it grows, until you've got
> the Bourne shell.
> 
> Doesn't the Tcl we're using nowadays have namespaces?  Can't we make
> .x files just Tcl scripts that run in a namespace that has all the
> right commands in it?  Then Daniel's example:
> 
>     #compile two.cc two.exe executable debug
>     #runto main
>     #test "ptype StrOne"
>     type = class OneStruct {
>       public:
>         int simple;
>     [synthetic OneStruct]}
>     #test "ptype ConstStrOnePtr"
>     type = const class OneStruct {
>       public:
>         int simple;
>     [synthetic OneStruct]} \*
> 
> could just be:
> 
>     compile two.cc two.exe executable debug
>     runto main
>     test "ptype StrOne" {
>     type = class OneStruct {
>       public:
>         int simple;
>     [synthetic OneStruct]}
>     }
>     test "ptype ConstStrOnePtr" {
>     type = const class OneStruct {
>       public:
>         int simple;
>     [synthetic OneStruct]} \*
>     }
> 
> And you've got all your quoting stuff, comments, substitution, etc for
> free.  Designed carefully, not grown incrementally.

Now that's the kind of suggestion I was really fishing for.  I'll
investigate further - I like it.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer