Need suggestions on how to test observer.c

Need suggestions on how to test observer.c

[Joel Brobecker]

Hello,

I would like your advice re unit-testing the code in observer.c.
Here is the exported interface (from observer.h):

        typedef void (observer_normal_stop_ftype) (void);
        
        extern struct observer *
          observer_attach_normal_stop (observer_normal_stop_ftype *f);
        extern void observer_detach_normal_stop (struct observer *observer);
        extern void observer_notify_normal_stop (void);

Before committing my code, I tested it with a little program that
simply set up some observers (up to 3 at the same time), triggered
some notifications, and verified that the callbacks were properly
invoked.

The ideal solution would be to use this little program as the unit-test
driver, and have a dejagnu testcase check its output. Unfortunately,
this program would of course depend on observer.o, which depends on
utils.o which depends on [... etc]. Not easy to put in place. One
possible approach would be to link the test program against libgdb.a
and dependents. In fine, we would more or less link the test program
with the same command as for gdb itself. This is a rather brutal approach,
but this should be easy to use to generate other unit-test drivers.
This would require a large change in the testsuite Makefiles, and
I haven't looked at how this could be done.

A more adhoc approach would be to artificially resolve the few
dependencies from observer.c: So far, observer.c only needs a
few functions: xmalloc, xfree and warning. So I defined trivial
ones in my little test driver. Building it then became just a
matter of adding observer.o to the link command.

The last time I needed to do some unit-testing, it was for the xfullpath
command, and it was pretty easy to test it inside gdb.gdb by debugging
gdb itself, and use inferior function calls with appropriate parameters.
It would be good to use the same approach one more time.  Unfortunately,
it is not so easy this time. We could use the same trick, except that
one of the parameters is a function pointer. That means we would need to
find some functions in GDB that could be used as callbacks:
  - First case: we use some functions already defined in GDB that would
    be benign enough that we could use them as callbacks for the
    notifications. To switch my current unit-test to such a scheme, I
    would need to find 3 such functions. I am not sure that finding 3
    benign functions with the appropriate profile would always be
    possible. Also, how would be make sure that these functions were
    called the appropriate number of time when sending the
    notifications? One possible generic approach is then to place a
    breakpoint on these functions, but then we would have to write the
    test in such a way that it is insensitive to the callback order,
    making it difficult to count how many times each callback is
    invoked...
  - Second case: we add 3 dummy functions to the GDB code that would
    do exactly what that functions in my testdrive do: increment an
    integer each time they are called. We would then use them as
    callbacks, and use the integers to check that the notifiation
    worked properly.

So, to summarize, we have several options (in order of personnal
preference):
  1. Compile and run my little test_observer.c program outside of GDB
     and check the output. To compile it, use the trick above to avoid
     the dependency on the entire GDB code.
  2. Update the makefile to give runtest the information necessary in
     order to be able to find libgdb.a and all dependents. Then setup
     the test to compile test_observer with that information. Run
     this program outside of GDB, and check the output.
  3. Add new benign functions inside observer.c to allow the testing
     to be done via inferior function calls.
  4. Find and use already defined functions in GDB, test the observer
     via inferior function calls.
  5. Do not add an observer-specific test. When a bug shows up due to
     an observer.c bug, we'll then add a testcase as usual. The testing
     coverage for observer.c will slowly increase.

Let me know what you prefer, and I'll go from there.

Thanks,
-- 
Joel

Re: Need suggestions on how to test observer.c

[Daniel Jacobowitz]

On Mon, Mar 10, 2003 at 02:43:56PM -0800, Joel Brobecker wrote:
> The last time I needed to do some unit-testing, it was for the xfullpath
> command, and it was pretty easy to test it inside gdb.gdb by debugging
> gdb itself, and use inferior function calls with appropriate parameters.
> It would be good to use the same approach one more time.  Unfortunately,
> it is not so easy this time. We could use the same trick, except that
> one of the parameters is a function pointer. That means we would need to
> find some functions in GDB that could be used as callbacks:

>   - Second case: we add 3 dummy functions to the GDB code that would
>     do exactly what that functions in my testdrive do: increment an
>     integer each time they are called. We would then use them as
>     callbacks, and use the integers to check that the notifiation
>     worked properly.

I have no objection to adding these dummies to observer.c; I think I
prefer that, because it tests observer code in the closer to the same
way it will be used.

> 
> So, to summarize, we have several options (in order of personnal
> preference):
>   1. Compile and run my little test_observer.c program outside of GDB
>      and check the output. To compile it, use the trick above to avoid
>      the dependency on the entire GDB code.
>   2. Update the makefile to give runtest the information necessary in
>      order to be able to find libgdb.a and all dependents. Then setup
>      the test to compile test_observer with that information. Run
>      this program outside of GDB, and check the output.
>   3. Add new benign functions inside observer.c to allow the testing
>      to be done via inferior function calls.
>   4. Find and use already defined functions in GDB, test the observer
>      via inferior function calls.
>   5. Do not add an observer-specific test. When a bug shows up due to
>      an observer.c bug, we'll then add a testcase as usual. The testing
>      coverage for observer.c will slowly increase.
> 
> Let me know what you prefer, and I'll go from there.
> 
> Thanks,
> -- 
> Joel
> 

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Need suggestions on how to test observer.c

[Joel Brobecker]

> I have no objection to adding these dummies to observer.c; I think I
> prefer that, because it tests observer code in the closer to the same
> way it will be used.

That makes good sense. I'll way for a few more day (say until the end of
this week), and then take this approach if nobody objects.

Thanks Daniel!
-- 
Joel

Re: Need suggestions on how to test observer.c

[David Carlton]

On Mon, 10 Mar 2003 14:43:56 -0800, Joel Brobecker <brobecker@gnat.com> said:

> The ideal solution would be to use this little program as the unit-test
> driver, and have a dejagnu testcase check its output. Unfortunately,
> this program would of course depend on observer.o, which depends on
> utils.o which depends on [... etc]. Not easy to put in place. One
> possible approach would be to link the test program against libgdb.a
> and dependents. In fine, we would more or less link the test program
> with the same command as for gdb itself. This is a rather brutal approach,
> but this should be easy to use to generate other unit-test drivers.
> This would require a large change in the testsuite Makefiles, and
> I haven't looked at how this could be done.

I agree that GDB should have more unit tests, and the idea of linking
libgdb.a was the best idea I could come up with.  Some difficulties
that might arise:

* Getting the Makefiles to work.  Probably not too hard, but who
  knows.

* Making sure that we can generate output that dejagnu likes.  dejagnu
  actually comes with a header file dejagnu.h that could be helpful
  here (though that might be a C++-only thing; I'm not sure).

* Would linking against libgdb.a take a long time?  If so, that would
  pose problems.  On the other hand, if the alternative is to run GDB
  on itself, then that's not the speediest thing to do either.

* And, of course, GDB isn't always modular enough to make writing unit
  tests easy, but there's not much we can do about that.

David Carlton
carlton@math.stanford.edu