[PATCH] cc-with-tweaks: show dwz stderr and check exit code

[PATCH] cc-with-tweaks: show dwz stderr and check exit code

[Simon Marchi]

When running the gdb.base/index-cache.exp test case with the
cc-with-dwz-m board, I noticed that the final executable didn't actually
contain a .gnu_debugaltlink section with the name of the external dwz
file:

    $ readelf --debug-dump=links testsuite/outputs/gdb.base/index-cache/index-cache
    * empty *

Running dwz by hand, I realized it's because dwz complains that the
output .debug_info section is empty and fails:

    $ gcc ~/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.c -g3 -O0 -o a && cp a b
    $ dwz -m foo a b
    dwz: foo: .debug_info section not present
    $ echo $?
    1

This is because index-cache.c is trivial (just an empty main) and dwz
doesn't find anything to factor out to the dwz file. [1]

I think that cc-with-tweaks should fail in this scenario: if the user
asks for an external dwz file to be generated (the -m flag), then it
should be an error if cc-with-tweaks doesn't manage to produce an
executable with the proper link to this external dwz file.  Otherwise,
the test runs with a regular non-dwzified executable, which gives a
false sense of security about whether the feature under test works with
dwzified executables.

So this patch adds checks for that after invoking dwz.  It also removes
the 2>&1 to allow the error message to be printed like so:

    Running /home/smarchi/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.exp ...
    gdb compile failed, dwz: /home/smarchi/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/index-cache/index-cache.dwz: .debug_info section not present

With this patch, fewer tests will pass than before with the
cc-with-dwz and cc-with-dwz-m boards, but those were false positives
anyway.

[1] Note that dwz has been patched by Tom de Vries to work correctly in
this case, so we can use dwz master to run the test:

https://sourceware.org/git/?p=dwz.git;a=commit;h=08becc8b33453b6d013a65e7eeae57fc1881e801

gdb/ChangeLog:

	* contrib/cc-with-tweaks.sh: Check for return value of dwz.
---
 gdb/contrib/cc-with-tweaks.sh | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/gdb/contrib/cc-with-tweaks.sh b/gdb/contrib/cc-with-tweaks.sh
index 47379cc15875..366b90918c18 100755
--- a/gdb/contrib/cc-with-tweaks.sh
+++ b/gdb/contrib/cc-with-tweaks.sh
@@ -180,10 +180,14 @@ if [ "$want_index" = true ]; then
 fi
 
 if [ "$want_dwz" = true ]; then
-    $DWZ "$output_file" > /dev/null 2>&1
+    $DWZ "$output_file" > /dev/null
+    rc=$?
+    [ $rc != 0 ] && exit $rc
 elif [ "$want_multi" = true ]; then
     cp $output_file ${output_file}.alt
-    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null 2>&1
+    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null
+    rc=$?
+    [ $rc != 0 ] && exit $rc
     rm -f ${output_file}.alt
 fi
 
-- 
2.21.0

Re: [PATCH] cc-with-tweaks: show dwz stderr and check exit code

[Tom de Vries]

On 08-05-19 18:00, Simon Marchi wrote:
> When running the gdb.base/index-cache.exp test case with the
> cc-with-dwz-m board, I noticed that the final executable didn't actually
> contain a .gnu_debugaltlink section with the name of the external dwz
> file:
> 
>     $ readelf --debug-dump=links testsuite/outputs/gdb.base/index-cache/index-cache
>     * empty *
> 
> Running dwz by hand, I realized it's because dwz complains that the
> output .debug_info section is empty and fails:
> 
>     $ gcc ~/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.c -g3 -O0 -o a && cp a b
>     $ dwz -m foo a b
>     dwz: foo: .debug_info section not present
>     $ echo $?
>     1
> 
> This is because index-cache.c is trivial (just an empty main) and dwz
> doesn't find anything to factor out to the dwz file. [1]
> 
> I think that cc-with-tweaks should fail in this scenario:


> if the user
> asks for an external dwz file to be generated (the -m flag),

This dwz semantics is open for debate, I think.

In the case of dwz a, the user asks for a to be compressed, and if that
doesn't happen due to size heuristics, then execution is still
considered a success.

In the case of dwz a -o b, the user asks for b to be generated, and
failure to generate b (f.i. due to size heuristics) is considered an
error. [ Which I guess is similar to the -o option of many other Linux
commands. ]

ISTM that the -m option could be interpreted either way. And I prefer
the interpretation of -m as: generate an external dwz file, if beneficial.

> then it
> should be an error if cc-with-tweaks doesn't manage to produce an
> executable with the proper link to this external dwz file.  Otherwise,
> the test runs with a regular non-dwzified executable, which gives a
> false sense of security about whether the feature under test works with
> dwzified executables.
> 

Agreed. Failing in cc-with-tweaks.sh in this case will list the
test-case as UNTESTED, which is a fair representation.

[ And at some point we can add support in cc-with-tweaks.sh for dwz
--devel-ignore-size (a switch only supported if dwz is compiled with
-DDEVEL) which makes it more likely to transform the input file and
generate an -m file, even if it's not beneficial, which will mean fewer
UNTESTED test-cases. ]

[ nitpick: And coming back to the false sense of security: there's no
guarantee that the DWARF that is intended to be tested by a particular
test-case is in fact transformed by dwz, even if the executable as a
whole is indeed transformed. So running the test-suite with dwz does not
guarantee that the feature under test works with dwzified executables. ]

> So this patch adds checks for that after invoking dwz.  It also removes
> the 2>&1 to allow the error message to be printed like so:
> 
>     Running /home/smarchi/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.exp ...
>     gdb compile failed, dwz: /home/smarchi/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/index-cache/index-cache.dwz: .debug_info section not present
> 

Nice. I've used your patch to test, and because of this more verbose
behaviour I already noticed a new error message ("Error mmapping
multi-file temporary files"), and fixed it in dwz.

> With this patch, fewer tests will pass than before with the
> cc-with-dwz and cc-with-dwz-m boards, but those were false positives
> anyway.
> 

Ack.

> [1] Note that dwz has been patched by Tom de Vries to work correctly in
> this case, so we can use dwz master to run the test:
> 
> https://sourceware.org/git/?p=dwz.git;a=commit;h=08becc8b33453b6d013a65e7eeae57fc1881e801
> 
> gdb/ChangeLog:
> 
> 	* contrib/cc-with-tweaks.sh: Check for return value of dwz.
> ---
>  gdb/contrib/cc-with-tweaks.sh | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/gdb/contrib/cc-with-tweaks.sh b/gdb/contrib/cc-with-tweaks.sh
> index 47379cc15875..366b90918c18 100755
> --- a/gdb/contrib/cc-with-tweaks.sh
> +++ b/gdb/contrib/cc-with-tweaks.sh
> @@ -180,10 +180,14 @@ if [ "$want_index" = true ]; then
>  fi
>  
>  if [ "$want_dwz" = true ]; then
> -    $DWZ "$output_file" > /dev/null 2>&1
> +    $DWZ "$output_file" > /dev/null
> +    rc=$?
> +    [ $rc != 0 ] && exit $rc
>  elif [ "$want_multi" = true ]; then
>      cp $output_file ${output_file}.alt
> -    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null 2>&1
> +    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null
> +    rc=$?
> +    [ $rc != 0 ] && exit $rc
>      rm -f ${output_file}.alt
>  fi

I wonder if it's more reliable to test for presence of -m file rather
than exit status (and likewise, we could copy and compare for the
want_dwz case).

Also, I propose to run dwz with -q, to inhibit the countless "not
beneficial" messages. It's good to list these cases as untested, but in
the log we'd rather skip the uninteresting messages, to increase the
likelyhood that interesting messages get spotted.

Thanks,
- Tom

Re: [PATCH] cc-with-tweaks: show dwz stderr and check exit code

[Simon Marchi]

On 2019-05-09 7:25 a.m., Tom de Vries wrote:
> On 08-05-19 18:00, Simon Marchi wrote:
>> When running the gdb.base/index-cache.exp test case with the
>> cc-with-dwz-m board, I noticed that the final executable didn't actually
>> contain a .gnu_debugaltlink section with the name of the external dwz
>> file:
>>
>>     $ readelf --debug-dump=links testsuite/outputs/gdb.base/index-cache/index-cache
>>     * empty *
>>
>> Running dwz by hand, I realized it's because dwz complains that the
>> output .debug_info section is empty and fails:
>>
>>     $ gcc ~/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.c -g3 -O0 -o a && cp a b
>>     $ dwz -m foo a b
>>     dwz: foo: .debug_info section not present
>>     $ echo $?
>>     1
>>
>> This is because index-cache.c is trivial (just an empty main) and dwz
>> doesn't find anything to factor out to the dwz file. [1]
>>
>> I think that cc-with-tweaks should fail in this scenario:
> 
> 
>> if the user
>> asks for an external dwz file to be generated (the -m flag),
> 
> This dwz semantics is open for debate, I think.
> 
> In the case of dwz a, the user asks for a to be compressed, and if that
> doesn't happen due to size heuristics, then execution is still
> considered a success.
>
> In the case of dwz a -o b, the user asks for b to be generated, and
> failure to generate b (f.i. due to size heuristics) is considered an
> error. [ Which I guess is similar to the -o option of many other Linux
> commands. ]>
> ISTM that the -m option could be interpreted either way. And I prefer
> the interpretation of -m as: generate an external dwz file, if beneficial.

It would be interesting to get a particular exit code to mean it worked but
"I chose not to change anything", so the caller can differentiate that from
other kinds of errors.

I just noticed something odd, when specifying input files that don't exist, dwz exits with 0:

$ dwz -m foo hello bonjour
dwz: Failed to open input file hello: No such file or directory
dwz: Failed to open input file bonjour: No such file or directory
dwz: Too few files for multifile optimization
$ echo $?
0

In this case, shouldn't it exit with some non-0 code, as it's clearly an error?

>> then it
>> should be an error if cc-with-tweaks doesn't manage to produce an
>> executable with the proper link to this external dwz file.  Otherwise,
>> the test runs with a regular non-dwzified executable, which gives a
>> false sense of security about whether the feature under test works with
>> dwzified executables.
>>
> 
> Agreed. Failing in cc-with-tweaks.sh in this case will list the
> test-case as UNTESTED, which is a fair representation.
> 
> [ And at some point we can add support in cc-with-tweaks.sh for dwz
> --devel-ignore-size (a switch only supported if dwz is compiled with
> -DDEVEL) which makes it more likely to transform the input file and
> generate an -m file, even if it's not beneficial, which will mean fewer
> UNTESTED test-cases. ]

Indeed, if we can probe that the dwz executable in use has this feature, we
can use it.

> 
> [ nitpick: And coming back to the false sense of security: there's no
> guarantee that the DWARF that is intended to be tested by a particular
> test-case is in fact transformed by dwz, even if the executable as a
> whole is indeed transformed. So running the test-suite with dwz does not
> guarantee that the feature under test works with dwzified executables. ]

So I guess that if dwz had a particular exit code to indicate that it chose not
to change the DWARF because it's not beneficial, we could decide that it's equivalent
to a compilation failure in cc-with-tweaks.sh.  As you said, it will result in UNTESTED,
which we agree is better than letting the user think that the test actually ran with a
dwzified executable.  It would also be pointless to continue running the test, as it
would test the exact same thing as what is tested using the default unix board.

Other users of dwz could consider that exit code of dwz as a success and carry on.

> 
>> So this patch adds checks for that after invoking dwz.  It also removes
>> the 2>&1 to allow the error message to be printed like so:
>>
>>     Running /home/smarchi/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.exp ...
>>     gdb compile failed, dwz: /home/smarchi/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/index-cache/index-cache.dwz: .debug_info section not present
>>
> 
> Nice. I've used your patch to test, and because of this more verbose
> behaviour I already noticed a new error message ("Error mmapping
> multi-file temporary files"), and fixed it in dwz.

Heh, I just saw that for the first time, thanks for fixing it!

>>  gdb/contrib/cc-with-tweaks.sh | 8 ++++++--
>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/gdb/contrib/cc-with-tweaks.sh b/gdb/contrib/cc-with-tweaks.sh
>> index 47379cc15875..366b90918c18 100755
>> --- a/gdb/contrib/cc-with-tweaks.sh
>> +++ b/gdb/contrib/cc-with-tweaks.sh
>> @@ -180,10 +180,14 @@ if [ "$want_index" = true ]; then
>>  fi
>>  
>>  if [ "$want_dwz" = true ]; then
>> -    $DWZ "$output_file" > /dev/null 2>&1
>> +    $DWZ "$output_file" > /dev/null
>> +    rc=$?
>> +    [ $rc != 0 ] && exit $rc
>>  elif [ "$want_multi" = true ]; then
>>      cp $output_file ${output_file}.alt
>> -    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null 2>&1
>> +    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null
>> +    rc=$?
>> +    [ $rc != 0 ] && exit $rc
>>      rm -f ${output_file}.alt
>>  fi
> 
> I wonder if it's more reliable to test for presence of -m file rather
> than exit status (and likewise, we could copy and compare for the
> want_dwz case).

Good point.  Why not check both actually (exit code and presence of the -m file)?
I did this change in the updated patch lower, tell me if you think it makes sense.

What do you mean by "we could copy and compare for the want_dwz case"?  No new
file is created in that case, so what should we test for?

> 
> Also, I propose to run dwz with -q, to inhibit the countless "not
> beneficial" messages. It's good to list these cases as untested, but in
> the log we'd rather skip the uninteresting messages, to increase the
> likelyhood that interesting messages get spotted.

Hmm, since this is the reason the test was not ran (it's akin to a compilation
failure), I think it's important to show it.  Someone could search for a very
long time why the test is not running if there's not error message.


Here's a new version of the patch updated to check for the presence of the dwz file.


From 1e6140468c42aec96058357d64d5557267cface3 Mon Sep 17 00:00:00 2001
From: Simon Marchi <simon.marchi@efficios.com>
Date: Tue, 7 May 2019 14:14:54 -0400
Subject: [PATCH] cc-with-tweaks: show dwz stderr and check exit code

When running the gdb.base/index-cache.exp test case with the
cc-with-dwz-m board, I noticed that the final executable didn't actually
contain a .gnu_debugaltlink section with the name of the external dwz
file:

    $ readelf --debug-dump=links testsuite/outputs/gdb.base/index-cache/index-cache
    * empty *

Running dwz by hand, I realized it's because dwz complains that the
output .debug_info section is empty and fails:

    $ gcc ~/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.c -g3 -O0 -o a && cp a b
    $ dwz -m foo a b
    dwz: foo: .debug_info section not present
    $ echo $?
    1

This is because index-cache.c is trivial (just an empty main) and dwz
doesn't find anything to factor out to the dwz file. [1]

I think that cc-with-tweaks should fail in this scenario: if the user
asks for an external dwz file to be generated (the -m flag), then it
should be an error if cc-with-tweaks doesn't manage to produce an
executable with the proper link to this external dwz file.  Otherwise,
the test runs with a regular non-dwzified executable, which gives a
false sense of security about whether the feature under test works with
dwzified executables.

So this patch adds checks for that after invoking dwz.  It also removes
the 2>&1 to allow the error message to be printed like so:

    Running /home/smarchi/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.exp ...
    gdb compile failed, dwz: /home/smarchi/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/index-cache/index-cache.dwz: .debug_info section not present

We also check for the presence of the expected output file in the dwz -m
case.

With this patch, fewer tests will pass than before with the
cc-with-dwz and cc-with-dwz-m boards, but those were false positives
anyway.

[1] Note that dwz has been patched by Tom de Vries to work correctly in
this case, so we can use dwz master to run the test:

https://sourceware.org/git/?p=dwz.git;a=commit;h=08becc8b33453b6d013a65e7eeae57fc1881e801

gdb/ChangeLog:

	* contrib/cc-with-tweaks.sh: Check for return value of dwz.
---
 gdb/contrib/cc-with-tweaks.sh | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/gdb/contrib/cc-with-tweaks.sh b/gdb/contrib/cc-with-tweaks.sh
index 47379cc15875..fcfdbd0589cf 100755
--- a/gdb/contrib/cc-with-tweaks.sh
+++ b/gdb/contrib/cc-with-tweaks.sh
@@ -180,10 +180,20 @@ if [ "$want_index" = true ]; then
 fi

 if [ "$want_dwz" = true ]; then
-    $DWZ "$output_file" > /dev/null 2>&1
+    $DWZ "$output_file" > /dev/null
+    rc=$?
+    [ $rc != 0 ] && exit $rc
 elif [ "$want_multi" = true ]; then
     cp $output_file ${output_file}.alt
-    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null 2>&1
+    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null
+    rc=$?
+    [ $rc != 0 ] && exit $rc
+
+    if [ ! -f "${output_file}.dwz" ]; then
+	echo "$myname: dwz file ${output_file}.dwz missing."
+	exit 1
+    fi
+
     rm -f ${output_file}.alt
 fi

-- 
2.21.0

Re: [PATCH] cc-with-tweaks: show dwz stderr and check exit code

[Tom de Vries]

On 09-05-19 17:15, Simon Marchi wrote:
> On 2019-05-09 7:25 a.m., Tom de Vries wrote:
>> On 08-05-19 18:00, Simon Marchi wrote:
>>> When running the gdb.base/index-cache.exp test case with the
>>> cc-with-dwz-m board, I noticed that the final executable didn't actually
>>> contain a .gnu_debugaltlink section with the name of the external dwz
>>> file:
>>>
>>>     $ readelf --debug-dump=links testsuite/outputs/gdb.base/index-cache/index-cache
>>>     * empty *
>>>
>>> Running dwz by hand, I realized it's because dwz complains that the
>>> output .debug_info section is empty and fails:
>>>
>>>     $ gcc ~/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.c -g3 -O0 -o a && cp a b
>>>     $ dwz -m foo a b
>>>     dwz: foo: .debug_info section not present
>>>     $ echo $?
>>>     1
>>>
>>> This is because index-cache.c is trivial (just an empty main) and dwz
>>> doesn't find anything to factor out to the dwz file. [1]
>>>
>>> I think that cc-with-tweaks should fail in this scenario:
>>
>>
>>> if the user
>>> asks for an external dwz file to be generated (the -m flag),
>>
>> This dwz semantics is open for debate, I think.
>>
>> In the case of dwz a, the user asks for a to be compressed, and if that
>> doesn't happen due to size heuristics, then execution is still
>> considered a success.
>>
>> In the case of dwz a -o b, the user asks for b to be generated, and
>> failure to generate b (f.i. due to size heuristics) is considered an
>> error. [ Which I guess is similar to the -o option of many other Linux
>> commands. ]>
>> ISTM that the -m option could be interpreted either way. And I prefer
>> the interpretation of -m as: generate an external dwz file, if beneficial.
> 
> It would be interesting to get a particular exit code to mean it worked but
> "I chose not to change anything", so the caller can differentiate that from
> other kinds of errors.
> 
> I just noticed something odd, when specifying input files that don't exist, dwz exits with 0:
> 
> $ dwz -m foo hello bonjour
> dwz: Failed to open input file hello: No such file or directory
> dwz: Failed to open input file bonjour: No such file or directory
> dwz: Too few files for multifile optimization
> $ echo $?
> 0
> 
> In this case, shouldn't it exit with some non-0 code, as it's clearly
> an error?
>

That's PR24301 - "dwz fails to return EXIT_FAILURE when processing two
files" ( https://sourceware.org/bugzilla/show_bug.cgi?id=24301 ). I've
just committed a fix.

>>> then it
>>> should be an error if cc-with-tweaks doesn't manage to produce an
>>> executable with the proper link to this external dwz file.  Otherwise,
>>> the test runs with a regular non-dwzified executable, which gives a
>>> false sense of security about whether the feature under test works with
>>> dwzified executables.
>>>
>>
>> Agreed. Failing in cc-with-tweaks.sh in this case will list the
>> test-case as UNTESTED, which is a fair representation.
>>
>> [ And at some point we can add support in cc-with-tweaks.sh for dwz
>> --devel-ignore-size (a switch only supported if dwz is compiled with
>> -DDEVEL) which makes it more likely to transform the input file and
>> generate an -m file, even if it's not beneficial, which will mean fewer
>> UNTESTED test-cases. ]
> 
> Indeed, if we can probe that the dwz executable in use has this feature, we
> can use it.
> 
>>
>> [ nitpick: And coming back to the false sense of security: there's no
>> guarantee that the DWARF that is intended to be tested by a particular
>> test-case is in fact transformed by dwz, even if the executable as a
>> whole is indeed transformed. So running the test-suite with dwz does not
>> guarantee that the feature under test works with dwzified executables. ]
> 
> So I guess that if dwz had a particular exit code to indicate that it chose not
> to change the DWARF because it's not beneficial, we could decide that it's equivalent
> to a compilation failure in cc-with-tweaks.sh.  As you said, it will result in UNTESTED,
> which we agree is better than letting the user think that the test actually ran with a
> dwzified executable.  It would also be pointless to continue running the test, as it
> would test the exact same thing as what is tested using the default unix board.
> 
> Other users of dwz could consider that exit code of dwz as a success and carry on.
> 

That's a user interface change, I'll leave that for Jakub to comment.

>>
>>> So this patch adds checks for that after invoking dwz.  It also removes
>>> the 2>&1 to allow the error message to be printed like so:
>>>
>>>     Running /home/smarchi/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.exp ...
>>>     gdb compile failed, dwz: /home/smarchi/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/index-cache/index-cache.dwz: .debug_info section not present
>>>
>>
>> Nice. I've used your patch to test, and because of this more verbose
>> behaviour I already noticed a new error message ("Error mmapping
>> multi-file temporary files"), and fixed it in dwz.
> 
> Heh, I just saw that for the first time, thanks for fixing it!
> 
>>>  gdb/contrib/cc-with-tweaks.sh | 8 ++++++--
>>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/gdb/contrib/cc-with-tweaks.sh b/gdb/contrib/cc-with-tweaks.sh
>>> index 47379cc15875..366b90918c18 100755
>>> --- a/gdb/contrib/cc-with-tweaks.sh
>>> +++ b/gdb/contrib/cc-with-tweaks.sh
>>> @@ -180,10 +180,14 @@ if [ "$want_index" = true ]; then
>>>  fi
>>>  
>>>  if [ "$want_dwz" = true ]; then
>>> -    $DWZ "$output_file" > /dev/null 2>&1
>>> +    $DWZ "$output_file" > /dev/null
>>> +    rc=$?
>>> +    [ $rc != 0 ] && exit $rc
>>>  elif [ "$want_multi" = true ]; then
>>>      cp $output_file ${output_file}.alt
>>> -    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null 2>&1
>>> +    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null
>>> +    rc=$?
>>> +    [ $rc != 0 ] && exit $rc
>>>      rm -f ${output_file}.alt
>>>  fi
>>
>> I wonder if it's more reliable to test for presence of -m file rather
>> than exit status (and likewise, we could copy and compare for the
>> want_dwz case).
> 
> Good point.  Why not check both actually (exit code and presence of the -m file)?
> I did this change in the updated patch lower, tell me if you think it makes sense.

The rationale for checking for file changed in want_dwz mode and
multifile presence in want_multi mode is because these are measures that
are more reliable than the exit code, and will get better results for
people using older dwz.

Checking for both goes against that rationale.

> 
> What do you mean by "we could copy and compare for the want_dwz case"?  No new
> file is created in that case, so what should we test for?
> 

I meant:
...
if [ "$want_dwz" = true ]; then
    cp $output_file ${output_file}.copy
    $DWZ "$output_file" > /dev/null
    if cmp "$output_file" "$output_file.copy"; rc=$?; then
	true
    fi
    rm -f ${output_file}.copy
    if [ $rc == 0 ]; then exit 1; fi
...

Thanks,
- Tom

>>
>> Also, I propose to run dwz with -q, to inhibit the countless "not
>> beneficial" messages. It's good to list these cases as untested, but in
>> the log we'd rather skip the uninteresting messages, to increase the
>> likelyhood that interesting messages get spotted.
> 
> Hmm, since this is the reason the test was not ran (it's akin to a compilation
> failure), I think it's important to show it.  Someone could search for a very
> long time why the test is not running if there's not error message.
> 
> 
> Here's a new version of the patch updated to check for the presence of the dwz file.
> 
> 
> From 1e6140468c42aec96058357d64d5557267cface3 Mon Sep 17 00:00:00 2001
> From: Simon Marchi <simon.marchi@efficios.com>
> Date: Tue, 7 May 2019 14:14:54 -0400
> Subject: [PATCH] cc-with-tweaks: show dwz stderr and check exit code
> 
> When running the gdb.base/index-cache.exp test case with the
> cc-with-dwz-m board, I noticed that the final executable didn't actually
> contain a .gnu_debugaltlink section with the name of the external dwz
> file:
> 
>     $ readelf --debug-dump=links testsuite/outputs/gdb.base/index-cache/index-cache
>     * empty *
> 
> Running dwz by hand, I realized it's because dwz complains that the
> output .debug_info section is empty and fails:
> 
>     $ gcc ~/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.c -g3 -O0 -o a && cp a b
>     $ dwz -m foo a b
>     dwz: foo: .debug_info section not present
>     $ echo $?
>     1
> 
> This is because index-cache.c is trivial (just an empty main) and dwz
> doesn't find anything to factor out to the dwz file. [1]
> 
> I think that cc-with-tweaks should fail in this scenario: if the user
> asks for an external dwz file to be generated (the -m flag), then it
> should be an error if cc-with-tweaks doesn't manage to produce an
> executable with the proper link to this external dwz file.  Otherwise,
> the test runs with a regular non-dwzified executable, which gives a
> false sense of security about whether the feature under test works with
> dwzified executables.
> 
> So this patch adds checks for that after invoking dwz.  It also removes
> the 2>&1 to allow the error message to be printed like so:
> 
>     Running /home/smarchi/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.exp ...
>     gdb compile failed, dwz: /home/smarchi/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/index-cache/index-cache.dwz: .debug_info section not present
> 
> We also check for the presence of the expected output file in the dwz -m
> case.
> 
> With this patch, fewer tests will pass than before with the
> cc-with-dwz and cc-with-dwz-m boards, but those were false positives
> anyway.
> 
> [1] Note that dwz has been patched by Tom de Vries to work correctly in
> this case, so we can use dwz master to run the test:
> 
> https://sourceware.org/git/?p=dwz.git;a=commit;h=08becc8b33453b6d013a65e7eeae57fc1881e801
> 
> gdb/ChangeLog:
> 
> 	* contrib/cc-with-tweaks.sh: Check for return value of dwz.
> ---
>  gdb/contrib/cc-with-tweaks.sh | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/gdb/contrib/cc-with-tweaks.sh b/gdb/contrib/cc-with-tweaks.sh
> index 47379cc15875..fcfdbd0589cf 100755
> --- a/gdb/contrib/cc-with-tweaks.sh
> +++ b/gdb/contrib/cc-with-tweaks.sh
> @@ -180,10 +180,20 @@ if [ "$want_index" = true ]; then
>  fi
> 
>  if [ "$want_dwz" = true ]; then
> -    $DWZ "$output_file" > /dev/null 2>&1
> +    $DWZ "$output_file" > /dev/null
> +    rc=$?
> +    [ $rc != 0 ] && exit $rc
>  elif [ "$want_multi" = true ]; then
>      cp $output_file ${output_file}.alt
> -    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null 2>&1
> +    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null
> +    rc=$?
> +    [ $rc != 0 ] && exit $rc
> +
> +    if [ ! -f "${output_file}.dwz" ]; then
> +	echo "$myname: dwz file ${output_file}.dwz missing."
> +	exit 1
> +    fi
> +
>      rm -f ${output_file}.alt
>  fi
> 

Re: [PATCH] cc-with-tweaks: show dwz stderr and check exit code

[Simon Marchi]

On 2019-05-10 5:03 a.m., Tom de Vries wrote:
> That's PR24301 - "dwz fails to return EXIT_FAILURE when processing two
> files" ( https://sourceware.org/bugzilla/show_bug.cgi?id=24301 ). I've
> just committed a fix.

Thanks!

>> Good point.  Why not check both actually (exit code and presence of the -m file)?
>> I did this change in the updated patch lower, tell me if you think it makes sense.
> 
> The rationale for checking for file changed in want_dwz mode and
> multifile presence in want_multi mode is because these are measures that
> are more reliable than the exit code, and will get better results for
> people using older dwz.
> 
> Checking for both goes against that rationale.

Ok, I understand your point for older dwzs.

>> What do you mean by "we could copy and compare for the want_dwz case"?  No new
>> file is created in that case, so what should we test for?
>>
> 
> I meant:
> ...
> if [ "$want_dwz" = true ]; then
>     cp $output_file ${output_file}.copy
>     $DWZ "$output_file" > /dev/null
>     if cmp "$output_file" "$output_file.copy"; rc=$?; then
> 	true
>     fi
>     rm -f ${output_file}.copy
>     if [ $rc == 0 ]; then exit 1; fi

Ah, interesting.  As you said earlier, this doesn't guarantee that the DWARF has been
modified, it could be something else in the executable that was changed.  But still, it
should catch a number of cases where dwz doesn't touch the executable at all.

I have included your suggestion in the patch below and made a few adjustments.

By the way, do you have a way to reproduce the case where dwz doesn't optimize anything
in single file mode?  Even with a file with just an empty main, dwz manages to do some
optimization.

Running dwz twice on the same executable is a way to get "DWARF compression not beneficial",
but this shouldn't happen in cc-with-tweaks.

Another question: in multifile mode, we check if the output dwz file exists.  Should we also
check that the original file now contains a .gnu_debugaltlink section?  If dwz succeeded to
generated the external dwz file, but failed to modify the original executable, we could also
end up running a test with a non-dwzified executable.


From ae61facce0dd0a480e265be1728fb1741b886180 Mon Sep 17 00:00:00 2001
From: Simon Marchi <simon.marchi@efficios.com>
Date: Tue, 7 May 2019 14:14:54 -0400
Subject: [PATCH] cc-with-tweaks: show dwz stderr and verify result

When running the gdb.base/index-cache.exp test case with the
cc-with-dwz-m board, I noticed that the final executable didn't actually
contain a .gnu_debugaltlink section with the name of the external dwz
file:

    $ readelf --debug-dump=links testsuite/outputs/gdb.base/index-cache/index-cache
    * empty *

Running dwz by hand, I realized it's because dwz complains that the
output .debug_info section is empty and fails:

    $ gcc ~/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.c -g3 -O0 -o a && cp a b
    $ dwz -m foo a b
    dwz: foo: .debug_info section not present
    $ echo $?
    1

This is because index-cache.c is trivial (just an empty main) and dwz
doesn't find anything to factor out to the dwz file. [1]

I think that cc-with-tweaks should fail in this scenario: if the user
asks for an external dwz file to be generated (the -m flag), then it
should be an error if cc-with-tweaks doesn't manage to produce an
executable with the proper link to this external dwz file.  Otherwise,
the test runs with a regular non-dwzified executable, which gives a
false sense of security about whether the feature under test works with
dwzified executables.

So this patch adds checks for that after invoking dwz.  It also removes
the 2>&1 to allow the error message to be printed like so:

    Running /home/smarchi/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.exp ...
    gdb compile failed, dwz: /home/smarchi/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/index-cache/index-cache.dwz: .debug_info section not present

- In the -m case (multi-file compression), we check if the expected output file
  exists.
- In the -z case (single-file compression), we check if the file
  contents has changed.  This should catch cases where dwz doesn't modify the
  file because it's not worth it.

It was chosen not to check for dwz's exit code, as it is not very
reliable up to dwz 0.12.

With this patch, fewer tests will pass than before with the
cc-with-dwz and cc-with-dwz-m boards, but those were false positives
anyway, as the test ran with regular executables.

[1] Note that dwz has been patched by Tom de Vries to work correctly in
this case, so we can use dwz master to run the test:

https://sourceware.org/git/?p=dwz.git;a=commit;h=08becc8b33453b6d013a65e7eeae57fc1881e801

gdb/ChangeLog:

	* contrib/cc-with-tweaks.sh: Validate dwz's work.
---
 gdb/contrib/cc-with-tweaks.sh | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/gdb/contrib/cc-with-tweaks.sh b/gdb/contrib/cc-with-tweaks.sh
index 47379cc15875..7df16bc6c1ca 100755
--- a/gdb/contrib/cc-with-tweaks.sh
+++ b/gdb/contrib/cc-with-tweaks.sh
@@ -180,11 +180,41 @@ if [ "$want_index" = true ]; then
 fi

 if [ "$want_dwz" = true ]; then
-    $DWZ "$output_file" > /dev/null 2>&1
+    # Validate dwz's result by checking if the executable was modified.
+    cp "$output_file" "${output_file}.copy"
+    $DWZ "$output_file" > /dev/null
+    cmp "$output_file" "$output_file.copy" > /dev/null
+    cmp_rc=$?
+    rm -f "${output_file}.copy"
+
+    case $cmp_rc in
+    0)
+	echo "$myname: dwz did not modify ${output_file}."
+        exit 1
+	;;
+    1)
+	# File was modified, great.
+	;;
+    *)
+	# Other cmp error, it presumably has already printed something on
+	# stderr.
+	exit 1
+	;;
+    esac
 elif [ "$want_multi" = true ]; then
+    # Remove the dwz output file if it exists, so we don't mistake it for a
+    # new file in case dwz fails.
+    rm -f "${output_file}.dwz"
+
     cp $output_file ${output_file}.alt
-    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null 2>&1
+    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null
     rm -f ${output_file}.alt
+
+    # Validate dwz's work by checking if the expected output file exists.
+    if [ ! -f "${output_file}.dwz" ]; then
+	echo "$myname: dwz file ${output_file}.dwz missing."
+	exit 1
+    fi
 fi

 if [ "$want_dwp" = true ]; then
-- 
2.21.0

Re: [PATCH] cc-with-tweaks: show dwz stderr and check exit code

[Tom de Vries]

On 10-05-19 18:05, Simon Marchi wrote:
> On 2019-05-10 5:03 a.m., Tom de Vries wrote:
>> That's PR24301 - "dwz fails to return EXIT_FAILURE when processing two
>> files" ( https://sourceware.org/bugzilla/show_bug.cgi?id=24301 ). I've
>> just committed a fix.
> 
> Thanks!
> 
>>> Good point.  Why not check both actually (exit code and presence of the -m file)?
>>> I did this change in the updated patch lower, tell me if you think it makes sense.
>>
>> The rationale for checking for file changed in want_dwz mode and
>> multifile presence in want_multi mode is because these are measures that
>> are more reliable than the exit code, and will get better results for
>> people using older dwz.
>>
>> Checking for both goes against that rationale.
> 
> Ok, I understand your point for older dwzs.
> 
>>> What do you mean by "we could copy and compare for the want_dwz case"?  No new
>>> file is created in that case, so what should we test for?
>>>
>>
>> I meant:
>> ...
>> if [ "$want_dwz" = true ]; then
>>     cp $output_file ${output_file}.copy
>>     $DWZ "$output_file" > /dev/null
>>     if cmp "$output_file" "$output_file.copy"; rc=$?; then
>> 	true
>>     fi
>>     rm -f ${output_file}.copy
>>     if [ $rc == 0 ]; then exit 1; fi
> 
> Ah, interesting.  As you said earlier, this doesn't guarantee that the DWARF has been
> modified, it could be something else in the executable that was changed.  But still, it
> should catch a number of cases where dwz doesn't touch the executable at all.
> 

The point I was trying to make, is that in a test-case there may be
dwarf related to the test-case, and other, run-of-the-mill dwarf that is
not at all related to the feature being tested, and when dwz has changed
the executable, there's no way of knowing which of the two, or both has
been affected. So, testing a test-case in combination with dwz, and
seeing that the dwarf has been changed by dwz, does not guarantee you
that the dwarf related to the feature being tested has in fact been changed.

> I have included your suggestion in the patch below and made a few adjustments.
> 
> By the way, do you have a way to reproduce the case where dwz doesn't optimize anything
> in single file mode?  Even with a file with just an empty main, dwz manages to do some
> optimization.
> 

I guess if you make a start.c with an empty _start function and compile
with -stdlib (so, the start testcase in the dwz testsuite), and then
furhter prune down the generated .s file, you'll end up with triggering
that warning.

> Running dwz twice on the same executable is a way to get "DWARF compression not beneficial",
> but this shouldn't happen in cc-with-tweaks.
> 

Ack.

> Another question: in multifile mode, we check if the output dwz file exists.  Should we also
> check that the original file now contains a .gnu_debugaltlink section?  If dwz succeeded to
> generated the external dwz file, but failed to modify the original executable, we could also
> end up running a test with a non-dwzified executable.
> 

Agreed, we could do that, but I haven't seen any problems with dwz
related to that, so I'm not sure it's worth the churn atm.

> 
> From ae61facce0dd0a480e265be1728fb1741b886180 Mon Sep 17 00:00:00 2001
> From: Simon Marchi <simon.marchi@efficios.com>
> Date: Tue, 7 May 2019 14:14:54 -0400
> Subject: [PATCH] cc-with-tweaks: show dwz stderr and verify result
> 
> When running the gdb.base/index-cache.exp test case with the
> cc-with-dwz-m board, I noticed that the final executable didn't actually
> contain a .gnu_debugaltlink section with the name of the external dwz
> file:
> 
>     $ readelf --debug-dump=links testsuite/outputs/gdb.base/index-cache/index-cache
>     * empty *
> 
> Running dwz by hand, I realized it's because dwz complains that the
> output .debug_info section is empty and fails:
> 
>     $ gcc ~/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.c -g3 -O0 -o a && cp a b
>     $ dwz -m foo a b
>     dwz: foo: .debug_info section not present
>     $ echo $?
>     1
> 
> This is because index-cache.c is trivial (just an empty main) and dwz
> doesn't find anything to factor out to the dwz file. [1]
> 
> I think that cc-with-tweaks should fail in this scenario: if the user
> asks for an external dwz file to be generated (the -m flag), then it
> should be an error if cc-with-tweaks doesn't manage to produce an
> executable with the proper link to this external dwz file.  Otherwise,
> the test runs with a regular non-dwzified executable, which gives a
> false sense of security about whether the feature under test works with
> dwzified executables.
> 
> So this patch adds checks for that after invoking dwz.  It also removes
> the 2>&1 to allow the error message to be printed like so:
> 
>     Running /home/smarchi/src/binutils-gdb/gdb/testsuite/gdb.base/index-cache.exp ...
>     gdb compile failed, dwz: /home/smarchi/build/binutils-gdb/gdb/testsuite/outputs/gdb.base/index-cache/index-cache.dwz: .debug_info section not present
> 
> - In the -m case (multi-file compression), we check if the expected output file
>   exists.
> - In the -z case (single-file compression), we check if the file
>   contents has changed.  This should catch cases where dwz doesn't modify the
>   file because it's not worth it.
> 
> It was chosen not to check for dwz's exit code, as it is not very
> reliable up to dwz 0.12.
> 
> With this patch, fewer tests will pass than before with the
> cc-with-dwz and cc-with-dwz-m boards, but those were false positives
> anyway, as the test ran with regular executables.
> 
> [1] Note that dwz has been patched by Tom de Vries to work correctly in
> this case, so we can use dwz master to run the test:
> 
> https://sourceware.org/git/?p=dwz.git;a=commit;h=08becc8b33453b6d013a65e7eeae57fc1881e801
> 
> gdb/ChangeLog:
> 
> 	* contrib/cc-with-tweaks.sh: Validate dwz's work.
> ---
>  gdb/contrib/cc-with-tweaks.sh | 34 ++++++++++++++++++++++++++++++++--
>  1 file changed, 32 insertions(+), 2 deletions(-)
> 
> diff --git a/gdb/contrib/cc-with-tweaks.sh b/gdb/contrib/cc-with-tweaks.sh
> index 47379cc15875..7df16bc6c1ca 100755
> --- a/gdb/contrib/cc-with-tweaks.sh
> +++ b/gdb/contrib/cc-with-tweaks.sh
> @@ -180,11 +180,41 @@ if [ "$want_index" = true ]; then
>  fi
> 
>  if [ "$want_dwz" = true ]; then
> -    $DWZ "$output_file" > /dev/null 2>&1
> +    # Validate dwz's result by checking if the executable was modified.
> +    cp "$output_file" "${output_file}.copy"
> +    $DWZ "$output_file" > /dev/null
> +    cmp "$output_file" "$output_file.copy" > /dev/null
> +    cmp_rc=$?
> +    rm -f "${output_file}.copy"
> +
> +    case $cmp_rc in
> +    0)
> +	echo "$myname: dwz did not modify ${output_file}."
> +        exit 1
> +	;;
> +    1)
> +	# File was modified, great.
> +	;;
> +    *)
> +	# Other cmp error, it presumably has already printed something on
> +	# stderr.
> +	exit 1
> +	;;
> +    esac
>  elif [ "$want_multi" = true ]; then
> +    # Remove the dwz output file if it exists, so we don't mistake it for a
> +    # new file in case dwz fails.
> +    rm -f "${output_file}.dwz"
> +
>      cp $output_file ${output_file}.alt
> -    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null 2>&1
> +    $DWZ -m ${output_file}.dwz "$output_file" ${output_file}.alt > /dev/null
>      rm -f ${output_file}.alt
> +
> +    # Validate dwz's work by checking if the expected output file exists.
> +    if [ ! -f "${output_file}.dwz" ]; then
> +	echo "$myname: dwz file ${output_file}.dwz missing."
> +	exit 1
> +    fi
>  fi
> 
>  if [ "$want_dwp" = true ]; then
> 

LGTM.

Thanks,
- Tom

Re: [PATCH] cc-with-tweaks: show dwz stderr and check exit code

[Simon Marchi]

On 2019-05-10 3:49 p.m., Tom de Vries wrote:
> The point I was trying to make, is that in a test-case there may be
> dwarf related to the test-case, and other, run-of-the-mill dwarf that is
> not at all related to the feature being tested, and when dwz has changed
> the executable, there's no way of knowing which of the two, or both has
> been affected. So, testing a test-case in combination with dwz, and
> seeing that the dwarf has been changed by dwz, does not guarantee you
> that the dwarf related to the feature being tested has in fact been changed.

Indeed.

>> I have included your suggestion in the patch below and made a few adjustments.
>>
>> By the way, do you have a way to reproduce the case where dwz doesn't optimize anything
>> in single file mode?  Even with a file with just an empty main, dwz manages to do some
>> optimization.
>>
> 
> I guess if you make a start.c with an empty _start function and compile
> with -stdlib (so, the start testcase in the dwz testsuite), and then
> furhter prune down the generated .s file, you'll end up with triggering
> that warning.

Ok, that's a bit further than I am willing to go right now to test the script :).

>> Another question: in multifile mode, we check if the output dwz file exists.  Should we also
>> check that the original file now contains a .gnu_debugaltlink section?  If dwz succeeded to
>> generated the external dwz file, but failed to modify the original executable, we could also
>> end up running a test with a non-dwzified executable.
>>
> 
> Agreed, we could do that, but I haven't seen any problems with dwz
> related to that, so I'm not sure it's worth the churn atm.

Ok.

> LGTM.

Thanks for all the comments, I am pushing the patch.

Simon