[proposal/testsuite] require build == host

[proposal/testsuite] require build == host

[Michael Chastain]

I'm working on a test script for the 'edit' command.  In order to test
the 'edit' command, the test script has to manage files on the host
machine (where gdb is running), not the build machine (where dejagnu is
running).

A recapitulation:

  build    runs dejagnu and the *.exp scripts
  host     runs gcc, as, ld, and gdb
  target   runs the test program such as "callfuncs" or "structs2"

I started looking at how other gdb test scripts work in a configuration
where build and host are different machines, and, well, they don't.
There are about a dozen scripts that compile test files with header
files, and except for list.exp, none of them download their *.h files to
the host machine.  gcore.exp manipulates files that gdb creates on the
host machine.  It needs to "remote upload" the files that gdb creates
back to the build machine, but it doesn't.  And so on.

Of course the test suite supports a target machine which is separate
from the build/host machine.  That is not at issue.

I don't think anyone actually uses a configuration with separate build
and host.  So I propose this patch to document that the gdb test suite
requires that build and host be the same machine.

I'll give this proposal a week for discussion.  I'm hoping that the
discussion goes "you're right, build-cross-host is silly and
unnecessary, just drop it already", but someone may have a good reason
for wanting it.

Testing: I built gdbint.texinfo with makeinfo 4.7 to check the syntax.
But that's not really important yet.  This is about the proposal,
not about the specific patch.

Comments?

Michael C

2004-07-24  Michael Chastain  <mec.gnu@mindspring.com>

	* gdbint.texinfo (Testsuite): Document that build and host
	must be the same machine.

Index: gdbint.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdbint.texinfo,v
retrieving revision 1.214
diff -c -3 -p -r1.214 gdbint.texinfo
*** gdbint.texinfo	1 Jul 2004 20:25:54 -0000	1.214
--- gdbint.texinfo	25 Jul 2004 00:01:08 -0000
*************** difficult to test, such as code that han
*** 6315,6320 ****
--- 6315,6325 ----
  in particular versions of compilers, and it's OK not to try to write
  tests for all of those.
  
+ DejaGNU supports separate build, host, and target machines.  However,
+ the @value{GDBN} testsuite requires that the host machine be the
+ same as the build machine.  This requirement simplifies the writing
+ of test files.
+ 
  @section Testsuite Organization
  
  @cindex test suite organization

Re: [proposal/testsuite] require build == host

[Felix Lee]

mec.gnu@mindspring.com (Michael Chastain):
> I don't think anyone actually uses a configuration with separate build
> and host.  So I propose this patch to document that the gdb test suite
> requires that build and host be the same machine.

hm.  gdb runs on more systems than expect does, and my feeling at
the moment is that it's sometimes easier to do build!=host
testing than it is to get expect running, like on weird embedded
linux devices.

not that I particularly like the testsuite complications added by
build!=host, but I don't know a better alternative yet.
--

Re: [proposal/testsuite] require build == host

[Daniel Jacobowitz]

On Sat, Jul 24, 2004 at 06:18:16PM -0700, Felix Lee wrote:
> mec.gnu@mindspring.com (Michael Chastain):
> > I don't think anyone actually uses a configuration with separate build
> > and host.  So I propose this patch to document that the gdb test suite
> > requires that build and host be the same machine.
> 
> hm.  gdb runs on more systems than expect does, and my feeling at
> the moment is that it's sometimes easier to do build!=host
> testing than it is to get expect running, like on weird embedded
> linux devices.
> 
> not that I particularly like the testsuite complications added by
> build!=host, but I don't know a better alternative yet.

Have you ever actually needed to do this?

When I tested GDB on embedded Linux devices, expect was not the issue. 
Expect is easy if you have enough RAM footprint to fit a GDB binary!

Also, what is a host on which GDB works but expect does not?

-- 
Daniel Jacobowitz

Re: [proposal/testsuite] require build == host

[Eli Zaretskii]

> Date: Sat, 24 Jul 2004 20:03:44 -0400 (EDT)
> From: mec.gnu@mindspring.com (Michael Chastain)
> 
> 2004-07-24  Michael Chastain  <mec.gnu@mindspring.com>
> 
> 	* gdbint.texinfo (Testsuite): Document that build and host
> 	must be the same machine.

I don't have an opinion about the issue, but the patch is okay with
me, provided that we agree to the requirement itself.

Perhaps this paragraph should be indexed, though: it sounds important
enough to warrant that.

Re: [proposal/testsuite] require build == host

[Felix Lee]

Daniel Jacobowitz <drow@false.org>:
> Also, what is a host on which GDB works but expect does not?

I remember running build!=host tests in some situations, but I
don't remember enough detail to demonstrate that it was really
necessary.  at the moment I'm mostly using retrograde reasoning:
the build!=host machinery is nontrivial, and it seems unlikely
that people would go to all that trouble if it were easy to just
get build==host working.

the issue isn't memory footprint, it's stuff like OS environment.
one thing that comes to mind: expect won't work if you don't have
ptys, but gdb works fine without ptys.  also, BeOS apparently has
a weird select() call that's hard to work with.

I think most of the build!=host machinery was invented in the
early days of cygwin testing.  it isn't necessary for cygwin now,
but it might be needed elsewhere.
--

Re: [proposal/testsuite] require build == host

[Michael Chastain]

Felix Lee <felix.1@canids.net> wrote:
lee> I remember running build!=host tests in some situations, but I
lee> don't remember enough detail to demonstrate that it was really
lee> necessary.

I won't say "show that there's no other way to get the same effect".
Because when one is trying to put a testbed together from several
components, it's much easier when more of the components are more
flexible.

If you've used build != host in the past, that's valid.

My question is: how far in the past?  Because it seems like
everything these days is gdb remote protocol.

lee> the issue isn't memory footprint, it's stuff like OS environment.
lee> one thing that comes to mind: expect won't work if you don't have
lee> ptys, but gdb works fine without ptys.  also, BeOS apparently has
lee> a weird select() call that's hard to work with.

If build != host, then the host does not have to run expect.
But the host does have to run some kind of network server
like telnet/ftp or rlogin/rcp (or kermit or tip or ...)

It's a tradeoff.  The situation right now is that there are 1-2 dozen
scripts which do not work in a build != host environment, and they've
been that way for several years.  I can spend time fixing these and
actually running some build != host test runs.  Or we can change the
policy so that build != host is not supported.

Hmmm, I just checked Dan Kegel's crosstool FAQ, and he wrote a
section on build != host testing for gcc, last revised
2004-01-04.  This argues against removing build != host from gdb.

Michael C

Re: [proposal/testsuite] require build == host

[Andrew Cagney]

What do the dejagnu maintainers have to say about the feature?  I can on 
the one hand see it being abandoned while on the other hand being used 
for something like parallel testing?

As for a system that would/should use this - djgpp.

Andrew

Re: [proposal/testsuite] require build == host

[Michael Chastain]

ac> What do the dejagnu maintainers have to say about the feature?

I haven't asked, but it's a supported feature of dejagnu and I'm
willing to accept that they will keep on supporting it.

Dan Kegel is at least one user of dejagnu who uses build != host
with gcc.  He wrote a whole section in his cross-gcc faq about it.
People are testing gcc with dejagnu using build != host,
so it would be good if gdb testing worked in the same environment.

ac> As for a system that would/should use this - djgpp.

You got me there.  It's been a while since I used windows,
but I suspect that windows has telnet/ftp servers.

Eli, is it possible to ftp into a windows box and upload and
download files?  More important, is it possible to telnet into
a windows box and run a command-line interpreter?

It looks like I should go with Plan B: file PR's against the
test scripts that don't work on build != host, and eventually
fix them.

Michael C

Re: [proposal/testsuite] require build == host

[Eli Zaretskii]

> Date: Mon, 26 Jul 2004 20:52:26 -0400
> From: Michael Chastain <mec.gnu@mindspring.com>
> 
> ac> As for a system that would/should use this - djgpp.
> 
> You got me there.  It's been a while since I used windows,
> but I suspect that windows has telnet/ftp servers.

(Btw, DJGPP produces DOS programs, not Windows programs.  These
programs do run on Windows, but Windows treats them as DOS
executables.)

> Eli, is it possible to ftp into a windows box and upload and
> download files?  More important, is it possible to telnet into
> a windows box and run a command-line interpreter?

It is possible to do that on Windows XP (perhaps after enabling some
service that is off by default, I don't remember).

Re: [proposal/testsuite] require build == host

[Felix Lee]

Michael Chastain <mec.gnu@mindspring.com>:
> If you've used build != host in the past, that's valid.
> My question is: how far in the past?  Because it seems like
> everything these days is gdb remote protocol.

can't remember last time I did it, this stuff never stays in my
mind long.  but friend says they still do build!=host testing at
TiVo.

> If build != host, then the host does not have to run expect.
> But the host does have to run some kind of network server
> like telnet/ftp or rlogin/rcp (or kermit or tip or ...)

yeah, I suspect whatever problems expect has could be fixed, any
type of network loopback could be used instead of the pty code.
it seems to be a nontrivial amount of work though.

> It's a tradeoff.  The situation right now is that there are 1-2 dozen
> scripts which do not work in a build != host environment, and they've
> been that way for several years.  I can spend time fixing these and
> actually running some build != host test runs.  Or we can change the
> policy so that build != host is not supported.

or those specific scripts could just yield unsupported when
build!=host.
--

Re: [proposal/testsuite] require build == host

[Andrew Cagney]

> ac> What do the dejagnu maintainers have to say about the feature?
> 
> I haven't asked, but it's a supported feature of dejagnu and I'm
> willing to accept that they will keep on supporting it.

It would be useful, just so that we know where they are going.

> Dan Kegel is at least one user of dejagnu who uses build != host
> with gcc.  He wrote a whole section in his cross-gcc faq about it.
> People are testing gcc with dejagnu using build != host,
> so it would be good if gdb testing worked in the same environment.
> 
> ac> As for a system that would/should use this - djgpp.
> 
> You got me there.  It's been a while since I used windows,
> but I suspect that windows has telnet/ftp servers.
> 
> Eli, is it possible to ftp into a windows box and upload and
> download files?  More important, is it possible to telnet into
> a windows box and run a command-line interpreter?

This is how the original cygwin testing was done - at a dos prompt.

> It looks like I should go with Plan B: file PR's against the
> test scripts that don't work on build != host, and eventually
> fix them.

For the moment, lets stick with the status quo.  Tests are required to 
work host==build.  Having a test work host!=build is an added bonus, but 
not a requrement.  We can hardly enforce host!=build when there's no way 
to test it and the return on investment is marginal.

Andrew

Re: [proposal/testsuite] require build == host

[Christopher Faylor]

On Tue, Jul 27, 2004 at 11:14:20AM -0400, Andrew Cagney wrote:
>>Eli, is it possible to ftp into a windows box and upload and download
>>files?  More important, is it possible to telnet into a windows box and
>>run a command-line interpreter?
>
>This is how the original cygwin testing was done - at a dos prompt.

Right.  And the test suite shennanigans necessary to support this were
truly ugly and, for some tests, only worked randomly.

>>It looks like I should go with Plan B: file PR's against the test
>>scripts that don't work on build != host, and eventually fix them.
>
>For the moment, lets stick with the status quo.  Tests are required to
>work host==build.  Having a test work host!=build is an added bonus,
>but not a requrement.  We can hardly enforce host!=build when there's
>no way to test it and the return on investment is marginal.

FWIW, I agree.

cgf

Re: [proposal/testsuite] require build == host

[Michael Chastain]

ac> It would be useful, just so that we know where they are going.

I talked to Rob Savoye a month or two ago, and basically
Dejagnu is mature at this point.  I don't think it's going to
add or drop major features.

ac> For the moment, lets stick with the status quo.  Tests are required to 
ac> work host==build.  Having a test work host!=build is an added bonus, but 
ac> not a requrement.  We can hardly enforce host!=build when there's no way 
ac> to test it and the return on investment is marginal.

The machine resources are easy to get.  build == my laptop,
host == hp test drive machine (they support inbound telnet and ftp).
All I have to do is write a board file!

I'm going to move off the status quo in one direction or the other.
Either build != host is supported, or it's not.  If it is supported,
then I have to test it (sigh), and at least characterize how many tests
are broken.  If it's not supported, then I want the doco to say so.

Michael C