[RFA/mips] Fix crash trying to print long double float

[Joel Brobecker <brobecker@gnat.com>]

Restarting the discussion that started with:

        http://sources.redhat.com/ml/gdb-patches/2004-07/msg00290.html

And then continued in:

        http://sources.redhat.com/ml/gdb-patches/2004-07/msg00310.html

I noticed the following SEGV in our testsuite. Here is below how to
reproduce it using the store.c sources in testsuite/gdb.base:

        % gcc -c -g store.c
        % gcc -o store store.o

The following transcript shows how to cause the SEGV:

        (gdb) b wack_doublest
        Breakpoint 1 at 0x1000256c: file store.c, line 125.
        (gdb) run
        Starting program: /[...]/gdb.base/store

        Breakpoint 1, wack_doublest (u=Unhandled dwarf expression opcode 0x93
        ) at store.c:125
        125       register doublest l = u, r = v;
        (gdb) n
        126       l = add_doublest (l, r);
        (gdb) p l
        zsh: 6356790 segmentation fault (core dumped)  ../../gdb store

The problem is that GDB is currently "configured" via the gdbarch
mechanism to think that "long double" types are 64bits long, when
they are in fact 128 bits long.

When we try to print the value of "l", a 128bit long variable (the
size is deduced from the debugging info), GDB does a floatformat lookup
based on type size. At some point, GDB calls the following function with
a length of 16 bytes:

        static const struct floatformat *
        floatformat_from_length (int len)
        {
          if (len * TARGET_CHAR_BIT == TARGET_FLOAT_BIT)
            return TARGET_FLOAT_FORMAT;
          else if (len * TARGET_CHAR_BIT == TARGET_DOUBLE_BIT)
            return TARGET_DOUBLE_FORMAT;
          else if (len * TARGET_CHAR_BIT == TARGET_LONG_DOUBLE_BIT)
            return TARGET_LONG_DOUBLE_FORMAT;
          /* On i386 the 'long double' type takes 96 bits,
             while the real number of used bits is only 80,
             both in processor and in memory.  
             The code below accepts the real bit size.  */
          else if ((TARGET_LONG_DOUBLE_FORMAT != NULL)
                   && (len * TARGET_CHAR_BIT ==
                       TARGET_LONG_DOUBLE_FORMAT->totalsize))
            return TARGET_LONG_DOUBLE_FORMAT;
        
          return NULL;
        }

Because the gdbarch vector tells GDB that TARGET_LONG_DOUBLE_BIT = 64bit,
we end up returning a NULL floatformat. And unfortunately for us, we
immediatly use that NULL floatformat to feed it to floatformat_is_valid(),
which dereferences it without checking that it's not NULL before hand.
This causes the SEGV.  See values.c:unpack_double():

        if (!floatformat_is_valid (floatformat_from_type (type), valaddr))
          {
            *invp = 1;
            return 0.0;
          }

The first thing that needs to be fixed in the size of long doubles.
The attached patch fixes this, and prevents the crash from happening.
There is a discussion regarding the long double floatformat used, which
happens to work, but only by chance. This will be the subject of another
patch.

2004-08-06  Joel Brobecker  <brobecker@gnat.com>

        * mips-tdep.c (mips_gdbarch_init): Set size of long double
        to 128 bits for N32 and N64 ABIs.

Tested on mips-irix. OK to apply?

Thanks,
-- 
Joel