[RFA/commit] configure: search waddstr only when building with TUI

[Joel Brobecker <brobecker@adacore.com>]

[-- Attachment #1: Type: text/plain, Size: 2267 bytes --]

Hello,

This one has been annoying us for a long time before we finally
found the source of the problem.  It happens only on Tru64, and
the symptoms are a screwed terminal after GDB has started (missing
carriage returns).  The source of the problem is the following:

  * readline's configure determines that libtermcap is sufficient
    for its purposes, and thus includes termcap headers, etc.

  * GDB's configure has the following search:
     # For the TUI, we need enhanced curses functionality.
     [...]
     AC_SEARCH_LIBS(waddstr, [ncurses cursesX curses])
    This search is done even when TUI is not requested, and so
    causes GDB to be compiled with libcurses.

On Tru64, unfortunately, the two libraries are not compatible.
I provide a little more details in the patch itself.

Ideally, we would want to adapt the build such that readline and
GDB end up using the same library.  But I haven't seen any other
system but Tru64 where this is a problem, and fixing this seems
a little tricky to coordinate.

I side-stepped the issue and declared gdb with TUI broken on Tru64,
and adjusted the configure to search for waddstr only when TUI was
requested (enable_tui either "yes" or "auto").  On Tru64, we error
out if TUI is requested, or disable it if "auto" was selected. The
nice consequence of this change is that this should get rid of
the dependency on libcurses on the platforms where its inclusion
was useless...

2008-10-02  Joel Brobecker  <brobecker@adacore.com>

        * configure.ac: On alpha-osf, error out if enable_tui is set to
        "yes", and set enable_tui to "no" if previously set to "auto".
        Check for waddstr only if TUI support was requested. Move the
        part of the configure script that updates various Makefile
        variables up, together with the check for waddstr.
        * configure: Regenerate.

The patch looks big, but I'm just moving things around inside an if
(except for the part were I error-out if TUI is requested on Tru64).
Tested on x86-linux, no regression. Tested on tru64 as well, but I
don't remember whether I ran the testsuite or not. I did verify
that GDB is now linked with libtermcap rater than libcurses.

Any objection? I'd like to commit this in about a week.

Thank you,
-- 
Joel

[-- Attachment #2: tui-configure.diff --]
[-- Type: text/plain, Size: 5108 bytes --]

diff -r 45c21f56c590 -r d8f2631b4e8c configure.ac
--- a/configure.ac	Thu Oct 02 16:18:27 2008 -0700
+++ b/configure.ac	Thu Oct 02 16:19:56 2008 -0700
@@ -429,19 +429,77 @@ AC_SEARCH_LIBS(socketpair, socket)
 # Link in zlib if we can.  This allows us to read compressed debug sections.
 AC_SEARCH_LIBS(zlibVersion, z, [AC_CHECK_HEADERS(zlib.h)])
 
-# For the TUI, we need enhanced curses functionality.
-#
-# FIXME: kettenis/20040905: We prefer ncurses over the vendor-supplied
-# curses library because the latter might not provide all the
-# functionality we need.  However, this leads to problems on systems
-# where the linker searches /usr/local/lib, but the compiler doesn't
-# search /usr/local/include, if ncurses is installed in /usr/local.  A
-# default installation of ncurses on alpha*-dec-osf* will lead to such
-# a situation.
-AC_SEARCH_LIBS(waddstr, [ncurses cursesX curses])
-
 # On HP/UX we may need libxpdl for dlgetmodinfo (used by solib-pa64.c).
 AC_SEARCH_LIBS(dlgetmodinfo, [dl xpdl])
+
+# brobecker/2008-09-18: On alpha-osf, it appears that libtermcap
+# and libcurses are not compatible.  See for instance the following
+# comment in /usr/include/curses.h:
+# 
+# /*
+#  * We must not let BSD programs use the termlib/termcap routines built
+#  * in to libcurses, so we pull the same trick of redefining all the
+#  * function calls.  These new routines are incorporated into libcurses,
+#  * where usualy they have remained seprate in libtermcap.
+#  * Note: These defines are duplicated in the curses library C modules.
+#  */
+#
+# The symptoms we observed so far is GDB unexpectedly changing
+# the terminal settings when tgetent is called - this is particularly
+# visible as the output is missing carriage returns, and so rapidly
+# becomes very hard to read.
+#
+# The readline configure script has already decided that libtermcap
+# was enough for its purposes, and so decided to build readline using
+# libtermcap.  Since the TUI mode requires curses, building GDB with
+# TUI enabled results in both libraries to be used at the same time,
+# which is not allowed.  This basically means that GDB with TUI is
+# broken on alpha-osf.
+
+case $host_os in
+  alpha*-*-osf* )
+    if "$enable_tui" = "yes"; then
+      AC_MSG_ERROR([Building GDB with TUI mode is not supported on this host])
+    fi
+    if "$enable_tui" = "auto"; then
+      enable_tui=no
+    fi
+    ;;
+esac
+
+# Check whether we should enable the TUI, but only do so if we really
+# can.
+if test x"$enable_tui" != xno; then
+  if test -d $srcdir/tui; then
+    # For the TUI, we need enhanced curses functionality.
+    #
+    # FIXME: kettenis/20040905: We prefer ncurses over the vendor-supplied
+    # curses library because the latter might not provide all the
+    # functionality we need.  However, this leads to problems on systems
+    # where the linker searches /usr/local/lib, but the compiler doesn't
+    # search /usr/local/include, if ncurses is installed in /usr/local.  A
+    # default installation of ncurses on alpha*-dec-osf* will lead to such
+    # a situation.
+    AC_SEARCH_LIBS(waddstr, [ncurses cursesX curses])
+
+    if test "$ac_cv_search_waddstr" != no; then
+      CONFIG_OBS="$CONFIG_OBS \$(SUBDIR_TUI_OBS)"
+      CONFIG_DEPS="$CONFIG_DEPS \$(SUBDIR_TUI_DEPS)"
+      CONFIG_SRCS="$CONFIG_SRCS \$(SUBDIR_TUI_SRCS)"
+      ENABLE_CFLAGS="$ENABLE_CFLAGS \$(SUBDIR_TUI_CFLAGS)"
+      CONFIG_ALL="${CONFIG_ALL} all-tui"
+      CONFIG_CLEAN="${CONFIG_CLEAN} clean-tui"
+      CONFIG_INSTALL="${CONFIG_INSTALL} install-tui"
+      CONFIG_UNINSTALL="${CONFIG_UNINSTALL} uninstall-tui"
+    else
+      if test x"$enable_tui" = xyes; then
+	AC_MSG_ERROR([no enhanced curses library found; disable TUI])
+      else
+	AC_MSG_WARN([no enhanced curses library found; disabling TUI])
+      fi
+    fi
+  fi
+fi
 
 # Since GDB uses Readline, we need termcap functionality.  In many
 # cases this will be provided by the curses library, but some systems
@@ -1748,29 +1806,6 @@ AC_SUBST(GDBTK_SRC_DIR)
 
 AC_PATH_X
 
-# Check whether we should enable the TUI, but only do so if we really
-# can.
-if test x"$enable_tui" != xno; then
-  if test -d $srcdir/tui; then
-    if test "$ac_cv_search_waddstr" != no; then
-      CONFIG_OBS="$CONFIG_OBS \$(SUBDIR_TUI_OBS)"
-      CONFIG_DEPS="$CONFIG_DEPS \$(SUBDIR_TUI_DEPS)"
-      CONFIG_SRCS="$CONFIG_SRCS \$(SUBDIR_TUI_SRCS)"
-      ENABLE_CFLAGS="$ENABLE_CFLAGS \$(SUBDIR_TUI_CFLAGS)"
-      CONFIG_ALL="${CONFIG_ALL} all-tui"
-      CONFIG_CLEAN="${CONFIG_CLEAN} clean-tui"
-      CONFIG_INSTALL="${CONFIG_INSTALL} install-tui"
-      CONFIG_UNINSTALL="${CONFIG_UNINSTALL} uninstall-tui"
-    else
-      if test x"$enable_tui" = xyes; then
-	AC_MSG_ERROR([no enhanced curses library found; disable TUI])
-      else
-	AC_MSG_WARN([no enhanced curses library found; disabling TUI])
-      fi
-    fi
-  fi
-fi
-
 # Unlike the sim directory, whether a simulator is linked is controlled by 
 # presence of a gdb_sim definition in the target configure.tgt entry.  
 # This code just checks for a few cases where we'd like to ignore those