[PATCH v2 1/2] Fix function argument and return value locations

[Hannes Domani <ssbssa@yahoo.de>]

Fixes these testsuite fails on Windows:
FAIL: gdb.base/callfuncs.exp: p t_float_complex_values(fc1, fc2)
FAIL: gdb.base/callfuncs.exp: p t_float_complex_many_args(fc1, fc2, fc3, fc4, fc1, fc2, fc3, fc4, fc1, fc2, fc3, fc4, fc1, fc2, fc3, fc4)
FAIL: gdb.base/callfuncs.exp: noproto: p t_float_complex_values(fc1, fc2)
FAIL: gdb.base/callfuncs.exp: noproto: p t_float_complex_many_args(fc1, fc2, fc3, fc4, fc1, fc2, fc3, fc4, fc1, fc2, fc3, fc4, fc1, fc2, fc3, fc4)
FAIL: gdb.base/call-sc.exp: p/c fun(); call call-sc-tld
FAIL: gdb.base/call-sc.exp: advance to fun for return; return call-sc-tld
FAIL: gdb.base/call-sc.exp: zed L for return; return call-sc-tld
FAIL: gdb.base/call-sc.exp: return foo; return call-sc-tld
FAIL: gdb.base/call-sc.exp: return foo; synchronize pc to main() for 'call-sc-tld'
FAIL: gdb.base/call-sc.exp: return foo; synchronize pc to main() for 'call-sc-tld'
FAIL: gdb.base/call-sc.exp: advance to fun for finish; return call-sc-tld
FAIL: gdb.base/call-sc.exp: zed L for finish; return call-sc-tld
FAIL: gdb.base/call-sc.exp: finish foo; return call-sc-tld (the program is no longer running)
FAIL: gdb.base/call-sc.exp: value foo finished; return call-sc-tld

For function arguments (callfuncs.exp), only TYPE_CODE_COMPLEX was
missing in the types passed via integer registers.

For return values, there were a lot more issues:
- TYPE_CODE_DECFLOAT is NOT returned via XMM0.
- long double is NOT returned via XMM0.
- but __int128 IS returned via XMM0.
- the comments for TYPE_CODE_FLT state that __m128, __m128i and __m128d are
  returned by XMM0, and this is correct, but it doesn't actually check for
  them, because they are TYPE_CODE_ARRAY with TYPE_VECTOR

So I had to add TYPE_CODE_DECFLOAT to the arguments passed via XMM register,
but I had to remove it from the values returned via XMM0 register.

gdb/ChangeLog:

2020-05-29  Hannes Domani  <ssbssa@yahoo.de>

	* amd64-windows-tdep.c (amd64_windows_passed_by_integer_register):
	Add TYPE_CODE_COMPLEX.
	(amd64_windows_return_value): Fix types returned via XMM0.

gdb/testsuite/ChangeLog:

2020-05-29  Hannes Domani  <ssbssa@yahoo.de>

	* gdb.base/call-sc.c: Fix return struct on stack test case.
	* gdb.base/call-sc.exp: Likewise.
---
v2:
- TYPE_CODE_COMPLEX is actually passed via integer register, not XMM
  register.
---
 gdb/amd64-windows-tdep.c           | 21 +++++++++++++++++----
 gdb/testsuite/gdb.base/call-sc.c   |  6 +++++-
 gdb/testsuite/gdb.base/call-sc.exp | 13 +++++++++++++
 3 files changed, 35 insertions(+), 5 deletions(-)

diff --git a/gdb/amd64-windows-tdep.c b/gdb/amd64-windows-tdep.c
index 487dfd45fc..e83d76257f 100644
--- a/gdb/amd64-windows-tdep.c
+++ b/gdb/amd64-windows-tdep.c
@@ -60,6 +60,7 @@ amd64_windows_passed_by_integer_register (struct type *type)
       case TYPE_CODE_RVALUE_REF:
       case TYPE_CODE_STRUCT:
       case TYPE_CODE_UNION:
+      case TYPE_CODE_COMPLEX:
 	return (TYPE_LENGTH (type) == 1
 		|| TYPE_LENGTH (type) == 2
 		|| TYPE_LENGTH (type) == 4
@@ -299,17 +300,29 @@ amd64_windows_return_value (struct gdbarch *gdbarch, struct value *function,
   switch (type->code ())
     {
       case TYPE_CODE_FLT:
-      case TYPE_CODE_DECFLOAT:
-        /* __m128, __m128i, __m128d, floats, and doubles are returned
-           via XMM0.  */
-        if (len == 4 || len == 8 || len == 16)
+        /* floats, and doubles are returned via XMM0.  */
+        if (len == 4 || len == 8)
           regnum = AMD64_XMM0_REGNUM;
         break;
+      case TYPE_CODE_ARRAY:
+        /* __m128, __m128i and __m128d are returned via XMM0.  */
+	if (TYPE_VECTOR (type) && len == 16)
+	  {
+	    enum type_code code = TYPE_TARGET_TYPE (type)->code ();
+	    if (code == TYPE_CODE_INT || code == TYPE_CODE_FLT)
+	      {
+		regnum = AMD64_XMM0_REGNUM;
+		break;
+	      }
+	  }
+	/* fall through */
       default:
         /* All other values that are 1, 2, 4 or 8 bytes long are returned
            via RAX.  */
         if (len == 1 || len == 2 || len == 4 || len == 8)
           regnum = AMD64_RAX_REGNUM;
+	else if (len == 16 && type->code () == TYPE_CODE_INT)
+	  regnum = AMD64_XMM0_REGNUM;
         break;
     }
 
diff --git a/gdb/testsuite/gdb.base/call-sc.c b/gdb/testsuite/gdb.base/call-sc.c
index ba80576ac3..eb140cd9cf 100644
--- a/gdb/testsuite/gdb.base/call-sc.c
+++ b/gdb/testsuite/gdb.base/call-sc.c
@@ -35,6 +35,7 @@ typedef t T;
 #endif
 
 T foo = '1', L;
+T init = '9';
 
 T fun()
 {
@@ -55,7 +56,10 @@ int main()
 {
   int i;
 
-  Fun(foo);	
+  /* Use a different initial value then is later used in the
+     "value foo returned" test, so in case the struct is then returned
+     on the stack, it doesn't have the correct value by accident.  */
+  Fun(init);
 
   /* An infinite loop that first clears all the variables and then
      calls the function.  This "hack" is to make re-testing easier -
diff --git a/gdb/testsuite/gdb.base/call-sc.exp b/gdb/testsuite/gdb.base/call-sc.exp
index 719000435a..9697c5ac24 100644
--- a/gdb/testsuite/gdb.base/call-sc.exp
+++ b/gdb/testsuite/gdb.base/call-sc.exp
@@ -291,6 +291,19 @@ proc test_scalar_returns { } {
 		fail "${test}"
 	    }
 	}
+	-re " = 57 .*${gdb_prompt} $" {
+	    if $return_value_unknown {
+		# The struct return case.
+		# The return value is stored on the stack, and since GDB
+		# didn't override it, it still has value that was stored
+		# there in the earlier Foo(init) call.
+		pass "${test}"
+	    } else {
+		# This contradicts the above claim that GDB knew
+		# the location of the return-value.
+		fail "${test}"
+	    }
+	}
 	-re ".*${gdb_prompt} $" {
 	    if $return_value_unimplemented {
 		# What a suprize.  The architecture hasn't implemented
-- 
2.26.2