Re: Spurious testsuite failures due to multibyte locale

Re: Spurious testsuite failures due to multibyte locale

[Michael Elizabeth Chastain]

Andreas Schwab writes:
> I'm getting numerous spurious testsuite failures when running in a
> multibyte locale like utf8 because the version of readline included in
> gdb forces the command line to be always redrawn completely when
> running in a multibyte locale.  The output of the prompt will look
> like "\r\n\r\r(gdb) \r(gdb) " which won't be matched by $gdb_prompt.

Me too:

  http://sources.redhat.com/ml/gdb-patches/2002-12/msg00294.html

At first I thought it was an artifact of my test bed configuration, but
I've straightened out my test bed and it still happens.  This is with
host=i686-pc-linux-gnu, osversion=red-hat-8.0, with $LANG=en_us.UTF-8.

I work around it now by setting $LANG=en_us when I run the test suite.

> -# Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000
> +# Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2003

As long as you are touching this line, can you add 2001 and 2002
to the copyright years?  I checked "cvs log gdb.exp" and there
were plenty of changes during those two years.

> +# Make sure we are using the C locale.
> +set env(LC_ALL) "C"
> +

Seems okay to me but I don't have the resources to test it right now.

I've appended my brain dump just in case anyone is curious.
It's on my TODO list to write a small "readline" test program and
file a bug report with the readline maintainers.

Michael C

===

# 2002-12-13: readline 4.3 was imported into gdb HEAD last week.
# It causes perverse screen output: instead of "(gdb) print x",
# I see "(gdb) ^M(gdb) p^M(gdb) pr^M(gdb) pri^m..." with a
# carriage return for each character.
#
# This output style comes as a surprise to all the dejagnu
# patterns.  The symptoms are massive timeouts starting in
# gdb.base/annota1.exp and continuing throughout the test suite.
#
# It looks like readline/display.c:_rl_move_cursor_relative
# has a multibyte bug (MB_CUR_MAX != 1 &&d rl_byte_oriented == 0).
# The first optimization "_rl_last_c_pos == new" is disabled
# because "_rl_last_c_pos" is a display position but "new"
# is an index in a a multibyte string, so they are not comparable.
# That is okay.  But later on, this block appears:
#
#  #if defined (HANDLE_MULTIBYTE)
#  /* NEW points to the buffer point, but _rl_last_c_pos is the display point.
#       The byte length of the string is probably bigger than the column width
#       of the string, which means that if NEW == _rl_last_c_pos, then NEW's
#       display point is less than _rl_last_c_pos. */
#    else if (_rl_last_c_pos >= new)
#  #else
#    else if (_rl_last_c_pos > new)
#  #endif
#
# Not so fast, cowboy!  _rl_last_c_pos == new here, but all the
# characters are plain, so they might match just fine.  But readline
# pessimizes out and issues a CR and reprints the line every time.
# There should be logic here to compare the strings rather than just
# assume that a left-move is needed.
#
# To work around this, I just change $LANG.  The default $LANG
# on my installation of red hat linux 8 is "en_US.UTF-8".
# Just plain "en_US" appears to work.

Re: Spurious testsuite failures due to multibyte locale

[Daniel Jacobowitz]

On Sun, Jan 05, 2003 at 11:28:51AM -0600, Michael Elizabeth Chastain wrote:
> Andreas Schwab writes:
> > I'm getting numerous spurious testsuite failures when running in a
> > multibyte locale like utf8 because the version of readline included in
> > gdb forces the command line to be always redrawn completely when
> > running in a multibyte locale.  The output of the prompt will look
> > like "\r\n\r\r(gdb) \r(gdb) " which won't be matched by $gdb_prompt.
> 
> Me too:
> 
>   http://sources.redhat.com/ml/gdb-patches/2002-12/msg00294.html
> 
> At first I thought it was an artifact of my test bed configuration, but
> I've straightened out my test bed and it still happens.  This is with
> host=i686-pc-linux-gnu, osversion=red-hat-8.0, with $LANG=en_us.UTF-8.
> 
> I work around it now by setting $LANG=en_us when I run the test suite.
> 
> > -# Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000
> > +# Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2003
> 
> As long as you are touching this line, can you add 2001 and 2002
> to the copyright years?  I checked "cvs log gdb.exp" and there
> were plenty of changes during those two years.

Did this yesterday :)

> > +# Make sure we are using the C locale.
> > +set env(LC_ALL) "C"
> > +
> 
> Seems okay to me but I don't have the resources to test it right now.
> 
> I've appended my brain dump just in case anyone is curious.
> It's on my TODO list to write a small "readline" test program and
> file a bug report with the readline maintainers.

I'm inclined to agree.  Tests which want to test something locale
related can do so explicitly.  Andreas, if no one objects in the next
day or so, please check this in.


-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Spurious testsuite failures due to multibyte locale

[Elena Zannoni]

Daniel Jacobowitz writes:
 > On Sun, Jan 05, 2003 at 11:28:51AM -0600, Michael Elizabeth Chastain wrote:
 > > Andreas Schwab writes:
 > > > I'm getting numerous spurious testsuite failures when running in a
 > > > multibyte locale like utf8 because the version of readline included in
 > > > gdb forces the command line to be always redrawn completely when
 > > > running in a multibyte locale.  The output of the prompt will look
 > > > like "\r\n\r\r(gdb) \r(gdb) " which won't be matched by $gdb_prompt.
 > > 
 > > Me too:
 > > 
 > >   http://sources.redhat.com/ml/gdb-patches/2002-12/msg00294.html
 > > 
 > > At first I thought it was an artifact of my test bed configuration, but
 > > I've straightened out my test bed and it still happens.  This is with
 > > host=i686-pc-linux-gnu, osversion=red-hat-8.0, with $LANG=en_us.UTF-8.
 > > 
 > > I work around it now by setting $LANG=en_us when I run the test suite.
 > > 
 > > > -# Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000
 > > > +# Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2003
 > > 
 > > As long as you are touching this line, can you add 2001 and 2002
 > > to the copyright years?  I checked "cvs log gdb.exp" and there
 > > were plenty of changes during those two years.
 > 
 > Did this yesterday :)
 > 
 > > > +# Make sure we are using the C locale.
 > > > +set env(LC_ALL) "C"
 > > > +
 > > 
 > > Seems okay to me but I don't have the resources to test it right now.
 > > 
 > > I've appended my brain dump just in case anyone is curious.
 > > It's on my TODO list to write a small "readline" test program and
 > > file a bug report with the readline maintainers.
 > 
 > I'm inclined to agree.  Tests which want to test something locale
 > related can do so explicitly.  Andreas, if no one objects in the next
 > day or so, please check this in.
 > 
 > 

I am not objecting at all, but I think Fernando should have the last
word. Not that I think he will object either...

Elena

 > -- 
 > Daniel Jacobowitz
 > MontaVista Software                         Debian GNU/Linux Developer

Re: Spurious testsuite failures due to multibyte locale

[Michael Elizabeth Chastain]

Okay, I wrote a standalone test program, it really wasn't painful.  :)

I sent a bug report to bug-readline@gnu.org.  I can't find a Web link
to it so I am including a copy here for gdb history.

Now the readline maintainers can fix the bug, spin a release, and
Elena Z can pick it up.  Meanwhile, we can use the workaround from
Andreas S.

Michael C

===

Synopsis

  Readline 4.3 does perverse refresh in UTF-8 mode.
  This breaks the gdb automated test suite.

Environment

  target      = i686-pc-linux-gnu
  host        = i686-pc-linux-gnu
  osversion   = red-hat-8.0
  gcc         = 3.2.1
  binutils    = 2.13.2.1
  libreadline = 4.3
  libtermcap  = 2.0.8-31-rh
  libc        = 2.2.93-5-rh

Description

  I see perverse screen output.  Instead of "(gdb) foo bar", I see
  "(gdb) ^M(gdb) f^M(gdb) fo^M(Gdb) foo^M ..." with a carriage return
  for each character.

  This output style comes as a surprise to all the dejagnu patterns
  in the gdb test suite, breaking the test suite.  The symptoms are
  massive timeouts starting in gdb.base/annota1.exp and continuing
  throughout the test suite.

How to Reproduce

  Script started on Sun Jan  5 19:56:44 2003

  bash-2.05b$ cat z1.c
  #include <stdio.h>
  #include <readline/readline.h>
  
  int main ()
  {
    char * line;
  
    for (;;)
    {
      line = readline ("(prompt) ");
      if (line == NULL)
	break;
  
      printf ("%s\n", line);
    }
  
    printf ("\n");
    return 0;
  }

  bash-2.05b$ gcc -v -g -c z1.c
  Reading specs from /berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/lib/gcc-lib/i686-pc-linux-gnu/3.2.1/specs
  Configured with: /berman/migchain/source/gcc-3.2.1/configure --prefix=/berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1 --disable-shared --with-gnu-as --with-as=/berman/migchain/install/target/native/binutils-2.13.2.1/bin/as --with-gnu-ld --with-ld=/berman/migchain/install/target/native/binutils-2.13.2.1/bin/ld
  Thread model: posix
  gcc version 3.2.1
   /berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/lib/gcc-lib/i686-pc-linux-gnu/3.2.1/cc1 -lang-c -v -D__GNUC__=3 -D__GNUC_MINOR__=2 -D__GNUC_PATCHLEVEL__=1 -D__GXX_ABI_VERSION=102 -D__ELF__ -Dunix -D__gnu_linux__ -Dlinux -D__ELF__ -D__unix__ -D__gnu_linux__ -D__linux__ -D__unix -D__linux -Asystem=posix -D__NO_INLINE__ -D__STDC_HOSTED__=1 -Acpu=i386 -Amachine=i386 -Di386 -D__i386 -D__i386__ -D__tune_i686__ -D__tune_pentiumpro__ z1.c -quiet -dumpbase z1.c -g -version -o /tmp/cck1itw9.s
  GNU CPP version 3.2.1 (cpplib) (i386 Linux/ELF)
  GNU C version 3.2.1 (i686-pc-linux-gnu)
	  compiled by GNU C version 3.2.1.
  ignoring nonexistent directory "/berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/i686-pc-linux-gnu/include"
  #include "..." search starts here:
  #include <...> search starts here:
   /usr/local/include
   /berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/include
   /berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/lib/gcc-lib/i686-pc-linux-gnu/3.2.1/include
   /usr/include
  End of search list.
   /berman/migchain/install/target/native/binutils-2.13.2.1/bin/as -V -Qy -o z1.o /tmp/cck1itw9.s
  GNU assembler version 2.13.2.1 (i686-pc-linux-gnu) using BFD version 2.13.2.1

  bash-2.05b$ gcc -v -g -o z1 z1.o ./install/lib/libreadline.a -ltermcap
  Reading specs from /berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/lib/gcc-lib/i686-pc-linux-gnu/3.2.1/specs
  Configured with: /berman/migchain/source/gcc-3.2.1/configure --prefix=/berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1 --disable-shared --with-gnu-as --with-as=/berman/migchain/install/target/native/binutils-2.13.2.1/bin/as --with-gnu-ld --with-ld=/berman/migchain/install/target/native/binutils-2.13.2.1/bin/ld
  Thread model: posix
  gcc version 3.2.1
   /berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/lib/gcc-lib/i686-pc-linux-gnu/3.2.1/collect2 --eh-frame-hdr -m elf_i386 -dynamic-linker /lib/ld-linux.so.2 -o z1 /usr/lib/crt1.o /usr/lib/crti.o /berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/lib/gcc-lib/i686-pc-linux-gnu/3.2.1/crtbegin.o -L/berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/lib/gcc-lib/i686-pc-linux-gnu/3.2.1 -L/berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/lib/gcc-lib/i686-pc-linux-gnu/3.2.1/../../.. z1.o ./install/lib/libreadline.a -ltermcap -lgcc -lc -lgcc /berman/migchain/install/target/native/gcc-3.2.1-binutils-2.13.2.1/lib/gcc-lib/i686-pc-linux-gnu/3.2.1/crtend.o /usr/lib/crtn.o

  bash-2.05b$ INPUTRC=/dev/null LANG=en_US ./z1
  (prompt) foo
  foo
  (prompt) 

  bash-2.05b$ INPUTRC=/dev/null LANG=en_US.UTF-8 ./z1
  \r\r(prompt) \r(prompt) \r(prompt) f\r(prompt) f\r(prompt) fo\r(prompt) fo\r(prompt) foo\r(prompt) foo
  foo
  \r\r(prompt) \r(prompt) 
  bash-2.05b$ exit

  Script done on Sun Jan  5 19:58:24 2003

Analysis

  It looks like readline/display.c:_rl_move_cursor_relative
  has a multibyte bug (MB_CUR_MAX != 1 &&d rl_byte_oriented == 0).
  The first optimization "_rl_last_c_pos == new" is disabled
  because "_rl_last_c_pos" is a display position but "new"
  is an index in a a multibyte string, so they are not comparable.
  That is okay.  But later on, this block appears:
 
   #if defined (HANDLE_MULTIBYTE)
   /* NEW points to the buffer point, but _rl_last_c_pos is the display point.
        The byte length of the string is probably bigger than the column width
        of the string, which means that if NEW == _rl_last_c_pos, then NEW's
        display point is less than _rl_last_c_pos. */
     else if (_rl_last_c_pos >= new)
   #else
     else if (_rl_last_c_pos > new)
   #endif
 
  Not so fast, cowboy!  _rl_last_c_pos == new here, but all the
  characters are plain, so they might match just fine.  But readline
  pessimizes out and issues a CR and reprints the line every time.
  There should be logic here to compare the strings rather than just
  assume that a left-move is needed.

Fix

  I don't have one.

Workaround

  Avoid UTF-8 encoding: "export LANG=en_US" (for en_US people).

Spurious testsuite failures due to multibyte locale

[Andreas Schwab]

I'm getting numerous spurious testsuite failures when running in a
multibyte locale like utf8 because the version of readline included in
gdb forces the command line to be always redrawn completely when
running in a multibyte locale.  The output of the prompt will look
like "\r\n\r\r(gdb) \r(gdb) " which won't be matched by $gdb_prompt.

Andreas.

2003-01-05  Andreas Schwab  <schwab@suse.de>

	* lib/gdb.exp: Set LC_ALL to "C" to avoid spurious testsuite
	failures.

--- gdb/testsuite/lib/gdb.exp.~1.30.~	2002-12-16 21:06:23.000000000 +0100
+++ gdb/testsuite/lib/gdb.exp	2003-01-05 16:58:06.000000000 +0100
@@ -1,4 +1,4 @@
-# Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000
+# Copyright 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2003
 # Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
@@ -63,6 +63,9 @@ if ![info exists env(EXEEXT)] {
     set EXEEXT $env(EXEEXT)
 }
 
+# Make sure we are using the C locale.
+set env(LC_ALL) "C"
+
 ### Only procedures should come after this point.
 
 #