Re: [PATCH 4/4] gdb/testsuite: track if a caching proc calls gdb_exit or not

[Andrew Burgess <aburgess@redhat.com>]

Luis Machado <luis.machado@arm.com> writes:

> On 8/13/24 17:30, Andrew Burgess wrote:
>> Luis Machado <luis.machado@arm.com> writes:
>> 
>>> On 8/8/24 15:50, Andrew Burgess wrote:
>>>> Luis Machado <luis.machado@arm.com> writes:
>>>>
>>>>> On 8/8/24 11:50, Luis Machado wrote:
>>>>>> On 8/8/24 11:20, Luis Machado wrote:
>>>>>>> On 8/7/24 15:31, Andrew Burgess wrote:
>>>>>>>> Luis Machado <luis.machado@arm.com> writes:
>>>>>>>>
>>>>>>>>> Hi Andrew,
>>>>>>>>>
>>>>>>>>> On 6/3/24 19:16, Andrew Burgess wrote:
>>>>>>>>>> After a recent patch review I asked myself why can_spawn_for_attach
>>>>>>>>>> exists.  This proc currently does some checks, and then calls
>>>>>>>>>> can_spawn_for_attach_1 which is an actual caching proc.
>>>>>>>>>>
>>>>>>>>>> The answer is that can_spawn_for_attach exists in order to call
>>>>>>>>>> gdb_exit the first time can_spawn_for_attach is called within any test
>>>>>>>>>> script.
>>>>>>>>>>
>>>>>>>>>> The reason this is useful is that can_spawn_for_attach_1 calls
>>>>>>>>>> gdb_exit.  If imagine the user calling can_spawn_for_attach_1 directly
>>>>>>>>>> then a problem might exist.  Imagine a test written like this:
>>>>>>>>>>
>>>>>>>>>>   gdb_start
>>>>>>>>>>
>>>>>>>>>>   if { [can_spawn_for_attach_1] } {
>>>>>>>>>>     ... do stuff that assumes GDB is running ...
>>>>>>>>>>   }
>>>>>>>>>>
>>>>>>>>>> If this test is NOT the first test run, and if an earlier test calls
>>>>>>>>>> can_spawn_for_attach_1, then when the above test is run the
>>>>>>>>>> can_spawn_for_attach_1 call will return the cached value and gdb_exit
>>>>>>>>>> will not be called.
>>>>>>>>>>
>>>>>>>>>> But, if the above test IS the first test run then
>>>>>>>>>> can_spawn_for_attach_1 will not returned the cached value, but will
>>>>>>>>>> instead compute the cached value, a process that ends up calling
>>>>>>>>>> gdb_exit.  When the body of the if is executed GDB would no longer be
>>>>>>>>>> running and the test would fail!
>>>>>>>>>>
>>>>>>>>>> So can_spawn_for_attach was added which ensures that we _always_ call
>>>>>>>>>> gdb_exit the first time can_spawn_for_attach is called within a single
>>>>>>>>>> test script, this ensures that in the above case, even if the above is
>>>>>>>>>> not the first test run, gdb_exit will still be called.  This avoids
>>>>>>>>>> some hidden bugs in the testsuite.
>>>>>>>>>>
>>>>>>>>>> However, what I observe is that can_spawn_for_attach is not the only
>>>>>>>>>> caching proc that calls gdb_exit.  Why does can_spawn_for_attach get
>>>>>>>>>> special treatment when surely the same issue exists for any other
>>>>>>>>>> caching proc that calls gdb_exit?
>>>>>>>>>>
>>>>>>>>>> I think a better solution is to move the logic from
>>>>>>>>>> can_spawn_for_attach into cache.exp and generalise it so that it
>>>>>>>>>> applies to all caching procs.
>>>>>>>>>>
>>>>>>>>>> This commit does this by:
>>>>>>>>>>
>>>>>>>>>>  1. When the underlying caching proc is executed we wrap gdb_exit.
>>>>>>>>>>     This wrapper sets a global to true if gdb_exit is called.  The
>>>>>>>>>>     value of this global is stored in gdb_data_cache (using a ',exit'
>>>>>>>>>>     suffix), and also written to the cache file if appropriate.
>>>>>>>>>>
>>>>>>>>>>  2. When a cached value is returned from gdb_do_cache, if the
>>>>>>>>>>     underlying proc would have called gdb_exit, and if this is the
>>>>>>>>>>     first use of the caching proc in this test script, then we call
>>>>>>>>>>     gdb_exit.
>>>>>>>>>>
>>>>>>>>>> When storing the ',exit' value into the on-disk cache file, the flag
>>>>>>>>>> value is stored on a second line.  Currently every cached value only
>>>>>>>>>> occupies a single line, and a check is added to ensure this remains
>>>>>>>>>> true in the future.
>>>>>>>>>>
>>>>>>>>>> One issue did come up in testing, a FAIL in gdb.base/break-interp.exp,
>>>>>>>>>> this was caused by can_spawn_for_attach_1 calling gdb_start without
>>>>>>>>>> first calling gdb_exit.  Under the old way of doing things
>>>>>>>>>> can_spawn_for_attach would call gdb_exit _before_ possibly calling the
>>>>>>>>>> actual caching proc.  Under the new scheme gdb_exit is called _after_
>>>>>>>>>> calling the actual caching proc.  What was happening was that
>>>>>>>>>> break-interp.exp would leave GDB running then call
>>>>>>>>>> can_spawn_for_attach, when the test in can_spawn_for_attach_1 tried to
>>>>>>>>>> attach to the inferior, state left in the running GDB would cause some
>>>>>>>>>> unexpected behaviour.  Fixed by having can_spawn_for_attach_1 call
>>>>>>>>>> gdb_exit before calling gdb_start, this ensures we have a fresh GDB.
>>>>>>>>>>
>>>>>>>>>> With this done can_spawn_for_attach_1 can be renamed to
>>>>>>>>>> can_spawn_for_attach, and the existing can_spawn_for_attach can be
>>>>>>>>>> deleted.
>>>>>>>>>> ---
>>>>>>>>>>  gdb/testsuite/lib/cache.exp | 86 +++++++++++++++++++++++++++++++------
>>>>>>>>>>  gdb/testsuite/lib/gdb.exp   | 83 +++++++++--------------------------
>>>>>>>>>>  2 files changed, 93 insertions(+), 76 deletions(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/gdb/testsuite/lib/cache.exp b/gdb/testsuite/lib/cache.exp
>>>>>>>>>> index e7b9114058b..fef065ec8b0 100644
>>>>>>>>>> --- a/gdb/testsuite/lib/cache.exp
>>>>>>>>>> +++ b/gdb/testsuite/lib/cache.exp
>>>>>>>>>> @@ -46,6 +46,40 @@ proc gdb_do_cache_wrap {real_name args} {
>>>>>>>>>>      return $result
>>>>>>>>>>  }
>>>>>>>>>>  
>>>>>>>>>> +# Global written to by wrap_gdb_exit.  Set to true if wrap_gdb_exit is
>>>>>>>>>> +# called.
>>>>>>>>>> +
>>>>>>>>>> +set gdb_exit_called false
>>>>>>>>>> +
>>>>>>>>>> +# Wrapper around gdb_exit.  Use with_override to replace gdb_exit with
>>>>>>>>>> +# wrap_gdb_exit, the original gdb_exit is renamed to orig_gdb_exit.
>>>>>>>>>> +
>>>>>>>>>> +proc wrap_gdb_exit {} {
>>>>>>>>>> +    set ::gdb_exit_called true
>>>>>>>>>> +    orig_gdb_exit
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>> +# If DO_EXIT is false then this proc does nothing.  If DO_EXIT is true
>>>>>>>>>> +# then call gdb_exit the first time this proc is called for each
>>>>>>>>>> +# unique value of NAME within a single test.  Every subsequent time
>>>>>>>>>> +# this proc is called within a single test (for a given value of
>>>>>>>>>> +# NAME), don't call gdb_exit.
>>>>>>>>>> +
>>>>>>>>>> +proc gdb_cache_maybe_gdb_exit { name do_exit } {
>>>>>>>>>> +    if { !$do_exit } {
>>>>>>>>>> +	return
>>>>>>>>>> +    }
>>>>>>>>>> +
>>>>>>>>>> +    # To track if this proc has been called for NAME we create a
>>>>>>>>>> +    # global variable.  In gdb_cleanup_globals (see gdb.exp) this
>>>>>>>>>> +    # global will be deleted when the test has finished.
>>>>>>>>>> +    set global_name __${name}__cached_gdb_exit_called
>>>>>>>>>> +    if { ![info exists ::${global_name}] } {
>>>>>>>>>> +	gdb_exit
>>>>>>>>>> +	set ::${global_name} true
>>>>>>>>>> +    }
>>>>>>>>>> +}
>>>>>>>>>> +
>>>>>>>>>>  # A helper for gdb_caching_proc that handles the caching.
>>>>>>>>>>  
>>>>>>>>>>  proc gdb_do_cache {name args} {
>>>>>>>>>> @@ -71,10 +105,12 @@ proc gdb_do_cache {name args} {
>>>>>>>>>>  
>>>>>>>>>>      set is_cached 0
>>>>>>>>>>      if {[info exists gdb_data_cache(${cache_name},value)]} {
>>>>>>>>>> -	set cached $gdb_data_cache(${cache_name},value)
>>>>>>>>>> -	verbose "$name: returning '$cached' from cache" 2
>>>>>>>>>> +	set cached_value $gdb_data_cache(${cache_name},value)
>>>>>>>>>> +	set cached_exit $gdb_data_cache(${cache_name},exit)
>>>>>>>>>> +	verbose "$name: returning '$cached_value' from cache" 2
>>>>>>>>>>  	if { $cache_verify == 0 } {
>>>>>>>>>> -	    return $cached
>>>>>>>>>> +	    gdb_cache_maybe_gdb_exit $name $cached_exit
>>>>>>>>>> +	    return $cached_value
>>>>>>>>>>  	}
>>>>>>>>>>  	set is_cached 1
>>>>>>>>>>      }
>>>>>>>>>> @@ -83,24 +119,46 @@ proc gdb_do_cache {name args} {
>>>>>>>>>>  	set cache_filename [make_gdb_parallel_path cache $cache_name]
>>>>>>>>>>  	if {[file exists $cache_filename]} {
>>>>>>>>>>  	    set fd [open $cache_filename]
>>>>>>>>>> -	    set gdb_data_cache(${cache_name},value) [read -nonewline $fd]
>>>>>>>>>> +	    set content [split [read -nonewline $fd] \n]
>>>>>>>>>>  	    close $fd
>>>>>>>>>> -	    set cached $gdb_data_cache(${cache_name},value)
>>>>>>>>>> -	    verbose "$name: returning '$cached' from file cache" 2
>>>>>>>>>> +	    set gdb_data_cache(${cache_name},value) [lindex $content 0]
>>>>>>>>>> +	    set gdb_data_cache(${cache_name},exit) [lindex $content 1]
>>>>>>>>>> +	    set cached_value $gdb_data_cache(${cache_name},value)
>>>>>>>>>> +	    set cached_exit $gdb_data_cache(${cache_name},exit)
>>>>>>>>>> +	    verbose "$name: returning '$cached_value' from file cache" 2
>>>>>>>>>>  	    if { $cache_verify == 0 } {
>>>>>>>>>> -		return $cached
>>>>>>>>>> +		gdb_cache_maybe_gdb_exit $name $cached_exit
>>>>>>>>>> +		return $cached_value
>>>>>>>>>>  	    }
>>>>>>>>>>  	    set is_cached 1
>>>>>>>>>>  	}
>>>>>>>>>>      }
>>>>>>>>>>  
>>>>>>>>>> -    set real_name gdb_real__$name
>>>>>>>>>> -    set gdb_data_cache(${cache_name},value) [gdb_do_cache_wrap $real_name {*}$args]
>>>>>>>>>> +    set ::gdb_exit_called false
>>>>>>>>>> +    with_override gdb_exit wrap_gdb_exit orig_gdb_exit {
>>>>>>>>>> +	set real_name gdb_real__$name
>>>>>>>>>> +	set gdb_data_cache(${cache_name},value) [gdb_do_cache_wrap $real_name {*}$args]
>>>>>>>>>> +    }
>>>>>>>>>> +    set gdb_data_cache(${cache_name},exit) $::gdb_exit_called
>>>>>>>>>> +
>>>>>>>>>> +    # If a value being stored in the cache contains a newline then
>>>>>>>>>> +    # when we try to read the value back from an on-disk cache file
>>>>>>>>>> +    # we'll interpret the second line of the value as the ',exit' value.
>>>>>>>>>> +    if { [regexp "\[\r\n\]" $gdb_data_cache(${cache_name},value)] } {
>>>>>>>>>> +	set computed_value $gdb_data_cache(${cache_name},value)
>>>>>>>>>> +	error "Newline found in value for $cache_name: $computed_value"
>>>>>>>>>> +    }
>>>>>>>>>> +
>>>>>>>>>>      if { $cache_verify == 1 && $is_cached == 1 } {
>>>>>>>>>> -	set computed $gdb_data_cache(${cache_name},value)
>>>>>>>>>> -	if { $cached != $computed } {
>>>>>>>>>> -	    error [join [list "Inconsistent results for $cache_name:"
>>>>>>>>>> -			 "cached: $cached vs. computed: $computed"]]
>>>>>>>>>> +	set computed_value $gdb_data_cache(${cache_name},value)
>>>>>>>>>> +	set computed_exit $gdb_data_cache(${cache_name},exit)
>>>>>>>>>> +	if { $cached_value != $computed_value } {
>>>>>>>>>> +	    error [join [list "Inconsistent value results for $cache_name:"
>>>>>>>>>> +			 "cached: $cached_value vs. computed: $computed_value"]]
>>>>>>>>>> +	}
>>>>>>>>>> +	if { $cached_exit != $computed_exit } {
>>>>>>>>>> +	    error [join [list "Inconsistent exit results for $cache_name:"
>>>>>>>>>> +			 "cached: $cached_exit vs. computed: $computed_exit"]]
>>>>>>>>>>  	}
>>>>>>>>>>      }
>>>>>>>>>>  
>>>>>>>>>> @@ -110,9 +168,11 @@ proc gdb_do_cache {name args} {
>>>>>>>>>>  	# Make sure to write the results file atomically.
>>>>>>>>>>  	set fd [open $cache_filename.[pid] w]
>>>>>>>>>>  	puts $fd $gdb_data_cache(${cache_name},value)
>>>>>>>>>> +	puts $fd $gdb_data_cache(${cache_name},exit)
>>>>>>>>>>  	close $fd
>>>>>>>>>>  	file rename -force -- $cache_filename.[pid] $cache_filename
>>>>>>>>>>      }
>>>>>>>>>> +    gdb_cache_maybe_gdb_exit $name $gdb_data_cache(${cache_name},exit)
>>>>>>>>>>      return $gdb_data_cache(${cache_name},value)
>>>>>>>>>>  }
>>>>>>>>>>  
>>>>>>>>>> diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp
>>>>>>>>>> index 8235d4f28eb..d29fd740f91 100644
>>>>>>>>>> --- a/gdb/testsuite/lib/gdb.exp
>>>>>>>>>> +++ b/gdb/testsuite/lib/gdb.exp
>>>>>>>>>> @@ -6186,14 +6186,23 @@ proc gdb_exit { } {
>>>>>>>>>>      catch default_gdb_exit
>>>>>>>>>>  }
>>>>>>>>>>  
>>>>>>>>>> -# Helper function for can_spawn_for_attach.  Try to spawn and attach, and
>>>>>>>>>> -# return 0 only if we cannot attach because it's unsupported.
>>>>>>>>>> -
>>>>>>>>>> -gdb_caching_proc can_spawn_for_attach_1 {} {
>>>>>>>>>> -    # For the benefit of gdb-caching-proc-consistency.exp, which
>>>>>>>>>> -    # calls can_spawn_for_attach_1 directly.  Keep in sync with
>>>>>>>>>> -    # can_spawn_for_attach.
>>>>>>>>>> -    if { [is_remote target] || [target_info exists use_gdb_stub] } {
>>>>>>>>>> +# Return true if we can spawn a program on the target and attach to
>>>>>>>>>> +# it.
>>>>>>>>>> +
>>>>>>>>>> +gdb_caching_proc can_spawn_for_attach {} {
>>>>>>>>>> +    # We use exp_pid to get the inferior's pid, assuming that gives
>>>>>>>>>> +    # back the pid of the program.  On remote boards, that would give
>>>>>>>>>> +    # us instead the PID of e.g., the ssh client, etc.
>>>>>>>>>> +    if {[is_remote target]} {
>>>>>>>>>> +	verbose -log "can't spawn for attach (target is remote)"
>>>>>>>>>> +	return 0
>>>>>>>>>> +    }
>>>>>>>>>> +
>>>>>>>>>> +    # The "attach" command doesn't make sense when the target is
>>>>>>>>>> +    # stub-like, where GDB finds the program already started on
>>>>>>>>>> +    # initial connection.
>>>>>>>>>> +    if {[target_info exists use_gdb_stub]} {
>>>>>>>>>> +	verbose -log "can't spawn for attach (target is stub)"
>>>>>>>>>>  	return 0
>>>>>>>>>>      }
>>>>>>>>>>  
>>>>>>>>>> @@ -6218,6 +6227,9 @@ gdb_caching_proc can_spawn_for_attach_1 {} {
>>>>>>>>>>      set test_spawn_id [spawn_wait_for_attach_1 $obj]
>>>>>>>>>>      remote_file build delete $obj
>>>>>>>>>>  
>>>>>>>>>> +    # In case GDB is already running.
>>>>>>>>>> +    gdb_exit
>>>>>>>>>> +    
>>>>>>>>>>      gdb_start
>>>>>>>>>>  
>>>>>>>>>>      set test_pid [spawn_id_get_pid $test_spawn_id]
>>>>>>>>>> @@ -6239,61 +6251,6 @@ gdb_caching_proc can_spawn_for_attach_1 {} {
>>>>>>>>>>      return $res
>>>>>>>>>>  }
>>>>>>>>>>  
>>>>>>>>>> -# Return true if we can spawn a program on the target and attach to
>>>>>>>>>> -# it.  Calls gdb_exit for the first call in a test-case.
>>>>>>>>>> -
>>>>>>>>>> -proc can_spawn_for_attach { } {
>>>>>>>>>> -    # We use exp_pid to get the inferior's pid, assuming that gives
>>>>>>>>>> -    # back the pid of the program.  On remote boards, that would give
>>>>>>>>>> -    # us instead the PID of e.g., the ssh client, etc.
>>>>>>>>>> -    if {[is_remote target]} {
>>>>>>>>>> -	verbose -log "can't spawn for attach (target is remote)"
>>>>>>>>>> -	return 0
>>>>>>>>>> -    }
>>>>>>>>>> -
>>>>>>>>>> -    # The "attach" command doesn't make sense when the target is
>>>>>>>>>> -    # stub-like, where GDB finds the program already started on
>>>>>>>>>> -    # initial connection.
>>>>>>>>>> -    if {[target_info exists use_gdb_stub]} {
>>>>>>>>>> -	verbose -log "can't spawn for attach (target is stub)"
>>>>>>>>>> -	return 0
>>>>>>>>>> -    }
>>>>>>>>>> -
>>>>>>>>>> -    # The normal sequence to use for a runtime test like
>>>>>>>>>> -    # can_spawn_for_attach_1 is:
>>>>>>>>>> -    # - gdb_exit (don't use a running gdb, we don't know what state it is in),
>>>>>>>>>> -    # - gdb_start (start a new gdb), and
>>>>>>>>>> -    # - gdb_exit (cleanup).
>>>>>>>>>> -    #
>>>>>>>>>> -    # By making can_spawn_for_attach_1 a gdb_caching_proc, we make it
>>>>>>>>>> -    # unpredictable which test-case will call it first, and consequently a
>>>>>>>>>> -    # test-case may pass in say a full test run, but fail when run
>>>>>>>>>> -    # individually, due to a can_spawn_for_attach call in a location where a
>>>>>>>>>> -    # gdb_exit (as can_spawn_for_attach_1 does) breaks things.
>>>>>>>>>> -    # To avoid this, we move the initial gdb_exit out of
>>>>>>>>>> -    # can_spawn_for_attach_1, guaranteeing that we end up in the same state
>>>>>>>>>> -    # regardless of whether can_spawn_for_attach_1 is called.  However, that
>>>>>>>>>> -    # is only necessary for the first call in a test-case, so cache the result
>>>>>>>>>> -    # in a global (which should be reset after each test-case) to keep track
>>>>>>>>>> -    # of that.
>>>>>>>>>> -    #
>>>>>>>>>> -    # In summary, we distinguish between three cases:
>>>>>>>>>> -    # - first call in first test-case.  Executes can_spawn_for_attach_1.
>>>>>>>>>> -    #   Calls gdb_exit, gdb_start, gdb_exit.
>>>>>>>>>> -    # - first call in following test-cases.  Uses cached result of
>>>>>>>>>> -    #   can_spawn_for_attach_1.  Calls gdb_exit.
>>>>>>>>>> -    # - rest.  Use cached result in cache_can_spawn_for_attach_1. Calls no
>>>>>>>>>> -    #   gdb_start or gdb_exit.
>>>>>>>>>> -    global cache_can_spawn_for_attach_1
>>>>>>>>>> -    if { [info exists cache_can_spawn_for_attach_1] } {
>>>>>>>>>> -	return $cache_can_spawn_for_attach_1
>>>>>>>>>> -    }
>>>>>>>>>> -    gdb_exit
>>>>>>>>>> -
>>>>>>>>>> -    set cache_can_spawn_for_attach_1 [can_spawn_for_attach_1]
>>>>>>>>>> -    return $cache_can_spawn_for_attach_1
>>>>>>>>>> -}
>>>>>>>>>> -
>>>>>>>>>>  # Centralize the failure checking of "attach" command.
>>>>>>>>>>  # Return 0 if attach failed, otherwise return 1.
>>>>>>>>>>  
>>>>>>>>>
>>>>>>>>> This is a bit after the fact, but I tracked down some aarch64 sme test regressions
>>>>>>>>> to this particular patch. I'm still investigating exactly why it stopped working, but I
>>>>>>>>> can tell it only happens if we run 2 or more tests in the same run. It is not
>>>>>>>>> clear if making things parallel has an impact, or if it is just the fact we
>>>>>>>>> run 2+ tests in the same run.
>>>>>>>>>
>>>>>>>>> I suspect we may be calling gdb_exit when we shouldn't, and then things just
>>>>>>>>> stop working.
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> Running target unix
>>>>>>>>> Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.
>>>>>>>>> Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
>>>>>>>>> Using repos/binutils-gdb/gdb/testsuite/config/unix.exp as tool-and-target-specific interface file.
>>>>>>>>> Running repos/binutils-gdb/gdb/testsuite/gdb.arch/aarch64-sme-core-0.exp ...
>>>>>>>>> Running repos/binutils-gdb/gdb/testsuite/gdb.arch/aarch64-sme-regs-unavailable-3.exp ...
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> FAIL: gdb.arch/aarch64-sme-regs-unavailable-3.exp: prctl, vl=32 svl=256: check_regs: incorrect ZA state
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> ERROR: no fileid for ubuntu
>>>>>>>>> FAIL: gdb.arch/aarch64-sme-regs-unavailable-3.exp: gdb, vl=32 svl=256: check_regs: incorrect ZA state
>>>>>>>>
>>>>>>>> Luis,
>>>>>>>>
>>>>>>>> Could you please test the patch below to see if this fixes the issues
>>>>>>>> you are seeing.  This is also running through local testing at my side,
>>>>>>>> but I thought I'd get your feedback early.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>
>>>>>>> Well, it's one of those things I guess. I saw some errors the first time I tried the patch, but then
>>>>>>> I couldn't reproduce it anymore. So far it's been running pretty smoothly for both parallel and
>>>>>>> serialized runs. So I'd say this patch does the job and we should push it.
>>>>>>> Thanks for putting it together.
>>>>>>>
>>>>>>> I'll do a complete run overnight just to make sure, but it will take a little bit before I can report
>>>>>>> it.
>>>>>>
>>>>>> Of course, a short while after sending this, I managed to reproduce the error.
>>>>>>
>>>>>> I'm running the following:
>>>>>>
>>>>>> make check-gdb TESTS=gdb.arch/*.exp -j$(nproc). Let me fetch some more information.
>>>>>
>>>>> It seems we're hitting the same situation with aarch64_initialize_sve_information via aarch64_supports_sve_vl,
>>>>> which causes the testsuite to call gdb_exit.
>>>>>
>>>>> It is as you described, it is the first time we're running gdb.arch/aarch64-sme-core-2.exp, so we go through
>>>>> caching etc.
>>>>>
>>>>> ---
>>>>>
>>>>> Running builds/binutils-gdb/gdb/testsuite/../../../../repos/binutils-gdb/gdb/testsuite/gdb.arch/aarch64-sme-core-2.exp ...
>>>>> gdb_caching_proc allow_aarch64_sve_tests caused gdb_exit to be called
>>>>> stack trace is Stack trace:
>>>>>  gdb_cache_maybe_gdb_exit proc_name='allow_aarch64_sve_tests' cache_name='unix/allow_aarch64_sve_tests'
>>>>>   gdb_do_cache name='allow_aarch64_sve_tests' args=''
>>>>>    allow_aarch64_sve_tests
>>>>
>>>> This makes sense assuming that this is not the first time
>>>> `allow_aarch64_sve_tests` was called in this test run.  This looks like
>>>> its using the previously cached result.
>>>>
>>>> If all had gone as expected then this should have marked `gdb_exit` as
>>>> having been called for both `allow_aarch64_sve_tests` and
>>>> `aarch64_initialize_sve_information`, though given what happens below I
>>>> guess that the marker for `aarch64_initialize_sve_information` isn't
>>>> being created correctly.
>>>>
>>>> You could try applying this patch:
>>>>
>>>> ### START ###
>>>>
>>>> diff --git a/gdb/testsuite/lib/cache.exp b/gdb/testsuite/lib/cache.exp
>>>> index 7e1eae9259e..d6027b352f1 100644
>>>> --- a/gdb/testsuite/lib/cache.exp
>>>> +++ b/gdb/testsuite/lib/cache.exp
>>>> @@ -98,6 +98,7 @@ proc gdb_cache_maybe_gdb_exit { proc_name cache_name } {
>>>>  	set ::${global_name} true
>>>>  
>>>>  	foreach other_name $gdb_data_cache(${cache_name},also_called) {
>>>> +	    verbose -log "  gdb_caching_proc $other_name also called exit"
>>>>  	    set global_name __${other_name}__cached_gdb_exit_called
>>>>  	    set ::${global_name} true
>>>>  	}
>>>> @@ -110,6 +111,8 @@ proc gdb_do_cache {name args} {
>>>>      global gdb_data_cache objdir
>>>>      global GDB_PARALLEL
>>>>  
>>>> +    verbose -log "gdb_do_cache: $name ( $args )"
>>>> +
>>>>      # Normally, if we have a cached value, we skip computation and return
>>>>      # the cached value.  If set to 1, instead don't skip computation and
>>>>      # verify against the cached value.
>>>>
>>>> ### END ###
>>>>
>>>> which will log the "other" caching procs that are recorded as having
>>>> been called.
>>>>
>>>> Also, if you are running `make -j...." then you can look into the cache
>>>> files which will be 'gdb/testsuite/cache/unix/allow_aarch64_sve_tests'
>>>> and 'gdb/testsuite/cache/unix/aarch64_initialize_sve_information' as
>>>> this should also include the information about the nested caching proc
>>>> structure.
>>>>
>>>>>
>>>>> gdb_caching_proc allow_aarch64_sme_tests caused gdb_exit to be called
>>>>> stack trace is Stack trace:
>>>>>  gdb_cache_maybe_gdb_exit proc_name='allow_aarch64_sme_tests' cache_name='unix/allow_aarch64_sme_tests'
>>>>>   gdb_do_cache name='allow_aarch64_sme_tests' args=''
>>>>>    allow_aarch64_sme_tests
>>>>>
>>>>> get_compiler_info: gcc-13-2-0
>>>>> Executing on host: gcc  -fno-stack-protector  -fdiagnostics-color=never -g3 -march=armv8.5-a+sve -c -g  -o builds/binutils-gdb/gdb/testsuite/outputs/gdb.arch/aarch64-sme-core-2/aarch64-sme-core-20.o builds/binutils-gdb/gdb/testsuite/../../../../repos/binutils-gdb/gdb/testsuite/gdb.arch/aarch64-sme-core.c    (timeout = 300)
>>>>> builtin_spawn -ignore SIGHUP gcc -fno-stack-protector -fdiagnostics-color=never -g3 -march=armv8.5-a+sve -c -g -o builds/binutils-gdb/gdb/testsuite/outputs/gdb.arch/aarch64-sme-core-2/aarch64-sme-core-20.o builds/binutils-gdb/gdb/testsuite/../../../../repos/binutils-gdb/gdb/testsuite/gdb.arch/aarch64-sme-core.c
>>>>> Executing on host: gcc  -fno-stack-protector builds/binutils-gdb/gdb/testsuite/outputs/gdb.arch/aarch64-sme-core-2/aarch64-sme-core-20.o  -fdiagnostics-color=never -g3 -march=armv8.5-a+sve -g  -lm   -o builds/binutils-gdb/gdb/testsuite/outputs/gdb.arch/aarch64-sme-core-2/aarch64-sme-core-2    (timeout = 300)
>>>>> builtin_spawn -ignore SIGHUP gcc -fno-stack-protector builds/binutils-gdb/gdb/testsuite/outputs/gdb.arch/aarch64-sme-core-2/aarch64-sme-core-20.o -fdiagnostics-color=never -g3 -march=armv8.5-a+sve -g -lm -o builds/binutils-gdb/gdb/testsuite/outputs/gdb.arch/aarch64-sme-core-2/aarch64-sme-core-2
>>>>> builtin_spawn builds/binutils-gdb/gdb/testsuite/../../gdb/gdb -nw -nx -q -iex set height 0 -iex set width 0 -data-directory builds/binutils-gdb/gdb/data-directory
>>>>> (gdb) set height 0
>>>>> (gdb) set width 0
>>>>> (gdb) dir
>>>>> Reinitialize source path to empty? (y or n) y
>>>>> Source directories searched: $cdir:$cwd
>>>>> (gdb) dir builds/binutils-gdb/gdb/testsuite/../../../../repos/binutils-gdb/gdb/testsuite/gdb.arch
>>>>> Source directories searched: builds/binutils-gdb/gdb/testsuite/../../../../repos/binutils-gdb/gdb/testsuite/gdb.arch:$cdir:$cwd
>>>>> (gdb) kill
>>>>> The program is not being run.
>>>>> (gdb) file builds/binutils-gdb/gdb/testsuite/outputs/gdb.arch/aarch64-sme-core-2/aarch64-sme-core-2
>>>>> Reading symbols from builds/binutils-gdb/gdb/testsuite/outputs/gdb.arch/aarch64-sme-core-2/aarch64-sme-core-2...
>>>>> (gdb) gdb_caching_proc aarch64_initialize_sve_information caused gdb_exit to be called
>>>>> stack trace is Stack trace:
>>>>>  gdb_cache_maybe_gdb_exit proc_name='aarch64_initialize_sve_information' cache_name='unix/aarch64_initialize_sve_information'
>>>>>   gdb_do_cache name='aarch64_initialize_sve_information' args=''
>>>>>    aarch64_initialize_sve_information
>>>>>     aarch64_supports_sve_vl length='16'
>>>>>      test_sme_core_file id_start='50' id_end='74'
>>>>
>>>> This should have been prevented by the earlier allow_aarch64_sve_tests
>>>> call.  Either we failed to correctly spot the relationship between the
>>>> two caching procs, or we're failing to mark this second one as having
>>>> been called for some reason...
>>>>
>>>> I think we need more data to try and debug this...
>>>
>>> I did some more debugging on this, and I think I see what's going on here. I haven't
>>> checked the flow in detail yet, but hopefully it will ring some bells.
>>>
>>> My theory is that your patch's logic is sane, but we're hitting concurrency issues when
>>> running things in parallel, or parallel enough. For instance, I consistently hit issues when
>>> using -j4, but I can't hit it with anything below -j4.
>>>
>>> I added a couple debugging statements in gdb_do_cache, at the end within the if {[info exists GDB_PARALLEL]} block.
>>>
>>> The statements check if the $cache_filename exists and what the contents are, before we rename
>>> $cache_filename.[pid] to $cache_filename, essentially overwriting the old cache file.
>>>
>>> What I saw was the following:
>>>
>>> ---
>>>
>>> gdb_do_cache: aarch64_initialize_sve_information (  )
>>> gdb_caching_proc aarch64_initialize_sve_information caused gdb_exit to be called
>>> Stack trace:
>>>  gdb_cache_maybe_gdb_exit proc_name='aarch64_initialize_sve_information' cache_name='unix/aarch64_initialize_sve_information'
>>>   gdb_do_cache name='aarch64_initialize_sve_information' args=''
>>>    aarch64_initialize_sve_information
>>>     gdb_real__allow_aarch64_sve_tests
>>>      allow_aarch64_sve_tests
>>>
>>> XXXX: File builds/binutils-gdb/gdb/testsuite/cache/unix/allow_aarch64_sve_tests already exists before rename.
>>> XXXX: Old contents: 1 true aarch64_initialize_sve_information
>>> XXXX: New contents: 1 true {}
>>> gdb_caching_proc allow_aarch64_sve_tests caused gdb_exit to be called
>>> Stack trace:
>>>  gdb_cache_maybe_gdb_exit proc_name='allow_aarch64_sve_tests' cache_name='unix/allow_aarch64_sve_tests'
>>>   gdb_do_cache name='allow_aarch64_sve_tests' args=''
>>>    allow_aarch64_sve_tests
>>>
>>> ---
>>>
>>> So we replace a cache file that contains more information with a copy that has less information.
>>>
>>> Eventually we call aarch64_initialize_sve_information again, via a different function, and things go bad.
>>>
>>> I'm guessing the logic of updating the cache files needs to be atomic, and we're having some timing issues
>>> where two (or more) tests try to update the same file and we end up losing information.
>> 
>> Thanks for doing the leg work on this.  The information you provided is
>> spot on, and it was indeed a timing issue as you predicted.  Here's
>> what's happening:
>> 
>> In thread #1 we call 'allow_aarch64_sme_tests', there's no cache file
>> yet we eval the body, this calls 'aarch64_initialize_sme_information'.
>> So we try to create two cache files in this order:
>> 
>>   unix/aarch64_initialize_sme_information
>>     value: ... whatever ...
>>     exit_called?: true
>>     also_called: {}
>> 
>>   unix/allow_aarch64_sme_tests
>>     value: ... whatever ...
>>     exit_called?: true
>>     also_called: aarch64_initialize_sme_information
>> 
>> In thread #2 we also call 'allow_aarch64_sme_tests', however, if we
>> manage to land after the first cache file was created but before the
>> second file was created then we pick up the cached result for
>> 'aarch64_initialize_sme_information'.
>> 
>> And this is where my bug was: I didn't add
>> 'aarch64_initialize_sme_information' to the also_called list for a
>> caching proc if we managed to find a cached value.  As a result, the
>> thread #2 cache file looked like this:
>> 
>>   unix/allow_aarch64_sme_tests
>>     value: ... whatever ...
>>     exit_called?: true
>>     also_called: {}
>> 
>> which then overwrote the first cache file (atomically).
>> 
>> A later test would call 'allow_aarch64_sme_tests' and then separately
>> call 'aarch64_initialize_sme_information'.  Due to the corrupted cache
>> file calling 'allow_aarch64_sme_tests' would not mark
>> 'aarch64_initialize_sme_information' as having been called.  So when we
>> separately call 'aarch64_initialize_sme_information' gdb_exit would be
>> called which broke the test.
>> 
>> The fix, of course, is to ensure that when we get a cache hit we still
>> record the called function as being something that is "also called".
>> 
>> I have a new patch below (discard the previous patch I sent) which I
>> think should resole the problems you're having.  I'm still running the
>> full tests on my end, but initial (limited) testing looks good.
>
> Great. Glad the information was useful. I gave the attached patch a try and
> managed to run all the sme tests in parallel and did not see any FAIL's or
> errors.
>
> I'm doing a few more runs just in case, but I think we're good.
>

That's great news.  If I don't hear anything in a couple of days then
I'll push this fix.

Thanks,
Andrew