Re: RFA: shrink main_type

[Ken Werner <ken@linux.vnet.ibm.com>]

On Thursday, September 30, 2010 7:55:18 pm Joel Brobecker wrote:
> I'm sorry about the delay in getting to this. I hope it's something
> temporary that we all seem to be busier than usual.

Thanks for taking a look at this.

> > One approach to restore that functionality would be to move the
> > nottext flag into to the instance_flags of the type. Attached is an
> > untested patch of what I have in mind. Comments are welcome.

> It took me a while to figure out why this is necessary. Initially,
> I thought that the vector type should have the NOTTEXT bit set,
> but that wouldn't be sufficient for the case where we just print
> one element of the vector (because we'd end up checking type of
> the element and not find the NOTTEXT bit set, and thus print it
> as a character rather than an integer.
>
> Do I understand the situation correctly? If yes, can we add a test
> that checks that, if not already there?

I think you are right because c_textual_element_type queries the type of the 
element to be printed. In case a single element of the vector is printed it is 
the type of the vectors target type. Printing of character vectors is only 
tested indirectly through gdb.arch/altivec-abi.exp. I think it would be good 
to add a few tests to gdb.base/gnu_vector.exp.

> Based on that understanding, then I agree that the NOTTEXT flag
> seems to be more of an instance flag than a type flag.  The code
> in make_vector_type seems to be confirming that.
> 
> If you make a proper submission for this patch, I will officially
> review it.

When I first realized that printing of char vectors is awkward I attempted to 
fix this at c-valprint (http://sourceware.org/ml/gdb-
patches/2010-06/msg00573.html) but overlooked that there already is a 
mechanism for that. Then I learned about the NOTTEXT flag and thought Toms 
shrink-main_type-patch broke the char vector printing and revived this mail 
thread. Now I think this never worked because even the initial patch from Jan  
(http://sourceware.org/ml/gdb-patches/2007-08/msg00467.html) writes the 
nottext bit into the instance flags (make_vector_type) but  the 
textual_element_type function checks for a type flag. I think that moving 
nottext flag into to the instance_flags of the type is a good way to fix this. 
I'll post a proper patch soon.

Thanks
-ken