[PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[zhiyong.yan]

From: Zhiyong Yan <zhiyong.yan@windriver.com>

Gdb should not assume pending threads always generate “a non-gdbserver trap event”, for example “Signal 17” event could happen. Now that resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point must already exist, resume_one_thread() should ensure the software breaking point is installed although the thread is pending.

Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com
Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387
---
 gdbserver/linux-low.cc | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/gdbserver/linux-low.cc b/gdbserver/linux-low.cc
index e6a39202a98..d29881174db 100644
--- a/gdbserver/linux-low.cc
+++ b/gdbserver/linux-low.cc
@@ -4671,7 +4671,16 @@ linux_process_target::resume_one_thread (thread_info *thread,
       proceed_one_lwp (thread, NULL);
     }
   else
-    threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
+    {
+      threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
+      if (thread->last_resume_kind == resume_step)
+	{
+	  /* If resume_step is required by GDB, 
+	     install single-step breakpoint.  */
+	  if (supports_software_single_step ())
+	    install_software_single_step_breakpoints (lwp);
+	}
+    }
 
   thread->last_status.set_ignore ();
   lwp->resume = NULL;
-- 
2.25.1

[PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[zhiyong.yan]

From: Zhiyong Yan <zhiyong.yan@windriver.com>

Gdb should not assume pending threads always generate “a non-gdbserver trap event”, for example “Signal 17” event could happen. Now that resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point must already exist, resume_one_thread() should ensure the software breaking point is installed although the thread is pending.

Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com
Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387
---
 gdbserver/linux-low.cc | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/gdbserver/linux-low.cc b/gdbserver/linux-low.cc
index e6a39202a98..d29881174db 100644
--- a/gdbserver/linux-low.cc
+++ b/gdbserver/linux-low.cc
@@ -4671,7 +4671,16 @@ linux_process_target::resume_one_thread (thread_info *thread,
       proceed_one_lwp (thread, NULL);
     }
   else
-    threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
+    {
+      threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
+      if (thread->last_resume_kind == resume_step)
+	{
+	  /* If resume_step is required by GDB, 
+	     install single-step breakpoint.  */
+	  if (supports_software_single_step ())
+	    install_software_single_step_breakpoints (lwp);
+	}
+    }
 
   thread->last_status.set_ignore ();
   lwp->resume = NULL;
-- 
2.25.1

Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Kevin Buettner via Gdb-patches]

Hi Zhiyong,

I set up a Raspberry Pi running a recent 32-bit Raspberry Pi OS so
that I could test your patch.  I was able to build and run your test
case, but I could not reproduce the bug on the Pi.

I tested gdb.threads/*.exp using --target_board=native-gdbserver both
with and without your patch.  Some of these tests are racy, but my
conclusion from just looking at the PASSes and FAILs (after many test
runs) is that there are no regressions.

But then I remembered to enable core dumps on the Pi and after running
gdb.threads/pending-fork-event-detach/pending-fork-event-detach-main-vfork
by itself, I saw that it left a core file...

$ make check RUNTESTFLAGS="--target_board=native-gdbserver" TESTS=gdb.threads/pending-fork-event-detach.exp
...
		=== gdb Summary ===

# of unexpected core files	1
# of expected passes		240

The core file was from the running test case, not gdbserver, nor gdb.

Looking at the core file in GDB shows...

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x00010624 in break_here () at /mesquite2/sourceware-git/rpi-gdbserver/bld/../../worktree-gdbserver/gdb/testsuite/gdb.threads/pending-fork-event-detach.c:29
29	  x++;
[Current thread is 1 (Thread 0xf7e10440 (LWP 4835))]
(gdb) x/i $pc
=> 0x10624 <break_here+12>:	udf	#16
(gdb) x/x $pc
0x10624 <break_here+12>:	0xe7f001f0

...and in gdbserver/linux-aarch32-low.cc:

#define arm_eabi_breakpoint 0xe7f001f0UL

I think what's happened here is that the breakpoint added by your
patch is left in place when GDB detaches the test case.  When it
starts running again, it hits the software single step breakpoint
and, since it's no longer under GDB control, it dies with a SIGTRAP.

This core file is not created when I run the test using a gdbserver
without your patch.

I'm suspicious of the assert in linux_process_target::maybe_hw_step. 
Currently, it looks like this:

bool
linux_process_target::maybe_hw_step (thread_info *thread)
{
  if (supports_hardware_single_step ())
    return true;
  else
    {
      /* GDBserver must insert single-step breakpoint for software
	 single step.  */
      gdb_assert (has_single_step_breakpoints (thread));
      return false;
    }
}

But, when Yao Qi introduced it back in June, 2016, it looked like
this:

static int
maybe_hw_step (struct thread_info *thread)
{
  if (can_hardware_single_step ())
    return 1;
  else
    {
      struct process_info *proc = get_thread_process (thread);

      /* GDBserver must insert reinsert breakpoint for software
     single step.  */
      gdb_assert (has_reinsert_breakpoints (proc));
      return 0;
    }
}

So, back is 2016, when it was introduced, it's clear that the assert
was referring to breakpoints which needed to be reinserted.  Now,
that's not at all obvious.

Also, back in 2016, maybe_hw_step() was only called from two
locations; in each case it was in a block in which the condition
lwp->bp_reinsert != 0 was true.  But now there are two other
calls; in one case, the software single step breakpoints have
just been inserted, so that should be okay, but for the other
case, in linux_process_target::resume_stopped_resumed_lwps, I'm
less certain.

In any case, could you comment out (or delete) the assert in a
version of the source without your patch and let me know what
happens?

Also, if possible, I'd like to see a backtrace from where the
assert occurs so that I can see which call to maybe_hw_step
is responsible for triggering the failing assert.

Kevin

RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Yan, Zhiyong]

[-- Attachment #1: Type: text/plain, Size: 4482 bytes --]

Hi Kevin,
    The callstack of assert is attached.
    Please see attached gdbx2 which add more 'n' commands, on arm platform, keep execute 'n' command, this test case can trigger assert error.

    Today, I didn't finish setting up test environments on RaspBerry Pi4. Before I produced this issue on Xilinx arm platform.

Best Regards.
Zhiyong

-----Original Message-----
From: Kevin Buettner <kevinb@redhat.com> 
Sent: Saturday, July 22, 2023 4:50 AM
To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Zhiyong,

I set up a Raspberry Pi running a recent 32-bit Raspberry Pi OS so that I could test your patch.  I was able to build and run your test case, but I could not reproduce the bug on the Pi.

I tested gdb.threads/*.exp using --target_board=native-gdbserver both with and without your patch.  Some of these tests are racy, but my conclusion from just looking at the PASSes and FAILs (after many test
runs) is that there are no regressions.

But then I remembered to enable core dumps on the Pi and after running gdb.threads/pending-fork-event-detach/pending-fork-event-detach-main-vfork
by itself, I saw that it left a core file...

$ make check RUNTESTFLAGS="--target_board=native-gdbserver" TESTS=gdb.threads/pending-fork-event-detach.exp
...
                === gdb Summary ===

# of unexpected core files      1
# of expected passes            240

The core file was from the running test case, not gdbserver, nor gdb.

Looking at the core file in GDB shows...

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  0x00010624 in break_here () at /mesquite2/sourceware-git/rpi-gdbserver/bld/../../worktree-gdbserver/gdb/testsuite/gdb.threads/pending-fork-event-detach.c:29
29        x++;
[Current thread is 1 (Thread 0xf7e10440 (LWP 4835))]
(gdb) x/i $pc
=> 0x10624 <break_here+12>:     udf     #16
(gdb) x/x $pc
0x10624 <break_here+12>:        0xe7f001f0

...and in gdbserver/linux-aarch32-low.cc:

#define arm_eabi_breakpoint 0xe7f001f0UL

I think what's happened here is that the breakpoint added by your patch is left in place when GDB detaches the test case.  When it starts running again, it hits the software single step breakpoint and, since it's no longer under GDB control, it dies with a SIGTRAP.

This core file is not created when I run the test using a gdbserver without your patch.

I'm suspicious of the assert in linux_process_target::maybe_hw_step.
Currently, it looks like this:

bool
linux_process_target::maybe_hw_step (thread_info *thread) {
  if (supports_hardware_single_step ())
    return true;
  else
    {
      /* GDBserver must insert single-step breakpoint for software
         single step.  */
      gdb_assert (has_single_step_breakpoints (thread));
      return false;
    }
}

But, when Yao Qi introduced it back in June, 2016, it looked like
this:

static int
maybe_hw_step (struct thread_info *thread) {
  if (can_hardware_single_step ())
    return 1;
  else
    {
      struct process_info *proc = get_thread_process (thread);

      /* GDBserver must insert reinsert breakpoint for software
     single step.  */
      gdb_assert (has_reinsert_breakpoints (proc));
      return 0;
    }
}

So, back is 2016, when it was introduced, it's clear that the assert was referring to breakpoints which needed to be reinserted.  Now, that's not at all obvious.

Also, back in 2016, maybe_hw_step() was only called from two locations; in each case it was in a block in which the condition
lwp->bp_reinsert != 0 was true.  But now there are two other
calls; in one case, the software single step breakpoints have just been inserted, so that should be okay, but for the other case, in linux_process_target::resume_stopped_resumed_lwps, I'm less certain.

In any case, could you comment out (or delete) the assert in a version of the source without your patch and let me know what happens?

Also, if possible, I'd like to see a backtrace from where the assert occurs so that I can see which call to maybe_hw_step is responsible for triggering the failing assert.

Kevin


[-- Attachment #2: callstack.txt --]
[-- Type: text/plain, Size: 5798 bytes --]

maybe_hw_step: Assertion `has_single_step_breakpoints (thread)' failed.
Aborted (core dumped)
EC@mpr3.1> ls -l
total 7464
-rw------- 1 root root  630784 Jun 28 12:14 core-gdbserver-11475-1656418477
-rwxr-xr-x 1 root root 7265572 Jun 28 12:12 gdbserver
EC@mpr3.1> gdb ./gdbserver ./core-gdbserver-11475-1656418477
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "arm-wrs-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./gdbserver...
[New LWP 11475]

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `./gdbserver --once --attach :1234 325'.
Program terminated with signal SIGABRT, Aborted.
#0  0x76ca6216 in ?? () from /lib/libc.so.6
(gdb) bt
#0  0x76ca6216 in ?? () from /lib/libc.so.6
#1  0x76cb49d4 in raise () from /lib/libc.so.6
#2  0x76ca5ca0 in abort () from /lib/libc.so.6
#3  0x004a8280 in abort_or_exit () at ../../gdb-13.0.50.20221021/gdbserver/utils.cc:39
#4  internal_verror (file=<optimized out>, line=line@entry=2448, fmt=0x0, fmt@entry=0x7ee8c2bc "\210\316N", args=...,
    args@entry=...) at ../../gdb-13.0.50.20221021/gdbserver/utils.cc:108
#5  0x004d83ce in internal_error_loc (file=<optimized out>, line=line@entry=2448, fmt=0x4e4bc8 "%s: Assertion `%s' failed.")
    at ../../gdb-13.0.50.20221021/gdbsupport/errors.cc:58
#6  0x004c5c28 in linux_process_target::maybe_hw_step (this=<optimized out>, thread=<optimized out>)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2448
#7  linux_process_target::maybe_hw_step (this=<optimized out>, thread=<optimized out>)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2440
#8  0x004c660a in linux_process_target::resume_stopped_resumed_lwps (this=this@entry=0x50a3cc <the_arm_target>, thread=0x209fb28)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2466
#9  0x004c7614 in <lambda(thread_info*)>::operator() (__closure=<synthetic pointer>, thread=<optimized out>)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2606
#10 for_each_thread<linux_process_target::wait_for_event_filtered(ptid_t, ptid_t, int*, int)::<lambda(thread_info*)> > (func=...)
    at ../../gdb-13.0.50.20221021/gdbserver/gdbthread.h:159
#11 linux_process_target::wait_for_event_filtered (this=this@entry=0x50a3cc <the_arm_target>, wait_ptid=..., filter_ptid=...,
    wstatp=0x7ee8c544, options=1073741824) at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2604
#12 0x004c9426 in linux_process_target::wait_for_event (options=1079026852, wstatp=0x7ee8c544, ptid=...,
    this=0x50a3cc <the_arm_target>) at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2676
#13 linux_process_target::wait_1 (this=this@entry=0x50a3cc <the_arm_target>, ptid=...,
    ourstatus=ourstatus@entry=0x50c9f8 <g_client_state+1288>, target_options=..., target_options@entry=...)
    at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:2970
#14 0x004cab6c in linux_process_target::wait (this=0x50a3cc <the_arm_target>, ptid=..., ourstatus=0x50c9f8 <g_client_state+1288>,
    target_options=...) at ../../gdb-13.0.50.20221021/gdbserver/linux-low.cc:3624
#15 0x004b9778 in target_wait (options=..., status=0x50c9f8 <g_client_state+1288>, ptid=...)
    at ../../gdb-13.0.50.20221021/gdbserver/target.cc:197
#16 mywait (ptid=..., ourstatus=ourstatus@entry=0x50c9f8 <g_client_state+1288>, options=...,
    connected_wait=connected_wait@entry=1) at ../../gdb-13.0.50.20221021/gdbserver/target.cc:142
#17 0x004b3428 in resume (actions=<optimized out>, num_actions=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--c
    at ../../gdb-13.0.50.20221021/gdbserver/server.cc:2916
#18 resume (actions=0x20a6778, num_actions=<optimized out>) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:2888
#19 0x004b3ed0 in handle_v_cont (own_buf=0x20a6778 "E\001") at ../../gdb-13.0.50.20221021/gdbserver/server.cc:2875
#20 handle_v_requests (own_buf=own_buf@entry=0x208b0a8 "vCont;r45b444,45b44c:p145.18a;c:p145.-1", packet_len=packet_len@entry=39, new_packet_len=new_packet_len@entry=0x7ee8c8ac) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:3135
#21 0x004b6b76 in process_serial_event () at ../../gdb-13.0.50.20221021/gdbserver/server.cc:4481
#22 handle_serial_event (err=<optimized out>, client_data=<optimized out>) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:4513
#23 0x004d8904 in gdb_wait_for_event (block=block@entry=1) at ../../gdb-13.0.50.20221021/gdbsupport/event-loop.cc:694
#24 0x004d9034 in gdb_wait_for_event (block=1) at ../../gdb-13.0.50.20221021/gdbsupport/event-loop.cc:593
#25 gdb_do_one_event (mstimeout=mstimeout@entry=-1) at ../../gdb-13.0.50.20221021/gdbsupport/event-loop.cc:264
#26 0x004a8a6c in start_event_loop () at ../../gdb-13.0.50.20221021/gdbserver/server.cc:3511
#27 captured_main (argv=<optimized out>, argc=5) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:3991
#28 main (argc=5, argv=<optimized out>) at ../../gdb-13.0.50.20221021/gdbserver/server.cc:4077
(gdb)
(gdb)
(gdb) info thread
  Id   Target Id         Frame
* 1    LWP 11475         0x76ca6216 in ?? () from /lib/libc.so.6
(gdb)

[-- Attachment #3: gdbx2 --]
[-- Type: application/octet-stream, Size: 415 bytes --]

# gdbserver --once --attach :1234 $(pgrep eqptCntrl)

set pagination off

#target remote 128.224.176.94:1234
target remote 128.224.178.123:1234

#set logging file pr29742-host.log
#set logging enabled

#set debug remote 1
#set debug remote-packet-max-chars unlimited

handle SIG32 nostop noprint
handle SIGSTOP nostop noprint

b lpudated.c:92

c
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n
n

Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Kevin Buettner via Gdb-patches]

[-- Attachment #1: Type: text/plain, Size: 6177 bytes --]

Hi Zhiyong,

I looked at the backtrace that you provided and see that maybe_hw_step()
is being called from linux_process_target::resume_stopped_resumed_lwps,
which is the one location where I wasn't able to convince myself that
the assert should hold.

I was running your test case executable (osm) as an unprivileged user,
so neither the syslog calls nor the sudo were working.  (Sudo could
perhaps work, but it wanted to prompt for a password and stdin and
stdout were closed.)  I've since modified it so that sudo isn't used
and I'm using 'fprintf(stderr, ...)' instead of syslog - which is how
I discovered that sudo wasn't working.  I've tried next'ing quite a
lot, but so far I haven't reproduced the bug.  (Hopefully, the sudo
isn't required to reproduce the problem.)

If you manage to reproduce the bug on a Raspberry Pi 4 (and tell me
how to do it), that'd be great!

So, what I'm doing, using three separate terminals, in an attempt to
reproduce the bug is:

1) Run osm in terminal 1.  (I didn't want to mess with systemd.)  Once
I start running it, I see a bunch of messages from the dd command.

2) In terminal 2, I run:

   /path/to/gdbserver --debug --debug-format=all --remote-debug --event-loop-debug --once --attach :1234 $(pgrep osm)

3) In terminal 3, I run:

   /path/to/gdb osm -x ./gdbx2

(I've changed the target remote command in gdbx2 to refer to localhost.)

I'm also attaching my hacked lupdated.c.  If you see anything wrong
with what I'm trying, please let me know.

Kevin

On Mon, 24 Jul 2023 13:36:24 +0000
"Yan, Zhiyong" <Zhiyong.Yan@windriver.com> wrote:

> Hi Kevin,
>     The callstack of assert is attached.
>     Please see attached gdbx2 which add more 'n' commands, on arm platform, keep execute 'n' command, this test case can trigger assert error.
> 
>     Today, I didn't finish setting up test environments on RaspBerry Pi4. Before I produced this issue on Xilinx arm platform.
> 
> Best Regards.
> Zhiyong
> 
> -----Original Message-----
> From: Kevin Buettner <kevinb@redhat.com> 
> Sent: Saturday, July 22, 2023 4:50 AM
> To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
> Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
> Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step
> 
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> Hi Zhiyong,
> 
> I set up a Raspberry Pi running a recent 32-bit Raspberry Pi OS so that I could test your patch.  I was able to build and run your test case, but I could not reproduce the bug on the Pi.
> 
> I tested gdb.threads/*.exp using --target_board=native-gdbserver both with and without your patch.  Some of these tests are racy, but my conclusion from just looking at the PASSes and FAILs (after many test
> runs) is that there are no regressions.
> 
> But then I remembered to enable core dumps on the Pi and after running gdb.threads/pending-fork-event-detach/pending-fork-event-detach-main-vfork
> by itself, I saw that it left a core file...
> 
> $ make check RUNTESTFLAGS="--target_board=native-gdbserver" TESTS=gdb.threads/pending-fork-event-detach.exp
> ...
>                 === gdb Summary ===
> 
> # of unexpected core files      1
> # of expected passes            240
> 
> The core file was from the running test case, not gdbserver, nor gdb.
> 
> Looking at the core file in GDB shows...
> 
> Program terminated with signal SIGTRAP, Trace/breakpoint trap.
> #0  0x00010624 in break_here () at /mesquite2/sourceware-git/rpi-gdbserver/bld/../../worktree-gdbserver/gdb/testsuite/gdb.threads/pending-fork-event-detach.c:29
> 29        x++;
> [Current thread is 1 (Thread 0xf7e10440 (LWP 4835))]
> (gdb) x/i $pc
> => 0x10624 <break_here+12>:     udf     #16  
> (gdb) x/x $pc
> 0x10624 <break_here+12>:        0xe7f001f0
> 
> ...and in gdbserver/linux-aarch32-low.cc:
> 
> #define arm_eabi_breakpoint 0xe7f001f0UL
> 
> I think what's happened here is that the breakpoint added by your patch is left in place when GDB detaches the test case.  When it starts running again, it hits the software single step breakpoint and, since it's no longer under GDB control, it dies with a SIGTRAP.
> 
> This core file is not created when I run the test using a gdbserver without your patch.
> 
> I'm suspicious of the assert in linux_process_target::maybe_hw_step.
> Currently, it looks like this:
> 
> bool
> linux_process_target::maybe_hw_step (thread_info *thread) {
>   if (supports_hardware_single_step ())
>     return true;
>   else
>     {
>       /* GDBserver must insert single-step breakpoint for software
>          single step.  */
>       gdb_assert (has_single_step_breakpoints (thread));
>       return false;
>     }
> }
> 
> But, when Yao Qi introduced it back in June, 2016, it looked like
> this:
> 
> static int
> maybe_hw_step (struct thread_info *thread) {
>   if (can_hardware_single_step ())
>     return 1;
>   else
>     {
>       struct process_info *proc = get_thread_process (thread);
> 
>       /* GDBserver must insert reinsert breakpoint for software
>      single step.  */
>       gdb_assert (has_reinsert_breakpoints (proc));
>       return 0;
>     }
> }
> 
> So, back is 2016, when it was introduced, it's clear that the assert was referring to breakpoints which needed to be reinserted.  Now, that's not at all obvious.
> 
> Also, back in 2016, maybe_hw_step() was only called from two locations; in each case it was in a block in which the condition
> lwp->bp_reinsert != 0 was true.  But now there are two other
> calls; in one case, the software single step breakpoints have just been inserted, so that should be okay, but for the other case, in linux_process_target::resume_stopped_resumed_lwps, I'm less certain.
> 
> In any case, could you comment out (or delete) the assert in a version of the source without your patch and let me know what happens?
> 
> Also, if possible, I'd like to see a backtrace from where the assert occurs so that I can see which call to maybe_hw_step is responsible for triggering the failing assert.
> 
> Kevin
> 

[-- Attachment #2: lpudated.c --]
[-- Type: text/x-c++src, Size: 4636 bytes --]

/*
* lpudated.c - Simple timestamping daemon
*/

#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <time.h>
#include <stddef.h>
//
#include <signal.h>
#include <pthread.h>
#include <sys/wait.h>

#define WORKERCOUNT 1
#define WORKERSLEEPSECS 5
sigset_t g_stSigSet;
pthread_t g_stPid;

int bRun = 1;

int _osal_task_pause (unsigned short sleep_time)
{
	struct timespec ts;

	ts.tv_sec = sleep_time / 1000;
	ts.tv_nsec = (sleep_time % 1000) * 1000000;

	if (nanosleep(&ts, NULL) != 0)
    	return -1;

	return 0;
}

static void* sigthread_routine(void *pArg)
{
    sigset_t *pstSigSet = &g_stSigSet;
    int iSigNum = 0;
    pthread_t stPid = pthread_self();
    int iRet = -1;
    /* pthread_detach(stPid); */
    while(1){
        iRet = sigwait(pstSigSet, &iSigNum);
        if(-1 == iRet){
             fprintf(stderr, "Signal error: %s", strerror(errno));
        }
        bRun = 0;
        break;
    }
    return NULL;
}

static void* worker_a(void *pArg)
{
	pid_t sm_cid;
    pthread_t stPid = pthread_self();
    pthread_detach(stPid);
    /* Finally do out work */
    // char *sm_args[] = {"sudo", "swManager", "swpactivation", "activate", "forced", "standby", NULL };
    char *sm_args[] = {"dd", "if=/dev/zero", "of=/dev/null", "bs=1k", "count=1024", NULL };
    while(bRun){
    	sm_cid = fork();
    	if( sm_cid == 0 ) { //child
    		if (execvp(sm_args[0], sm_args) == -1){
//    			LOGGER(LCON|LRAM, SWD, SDL, ERR, "%s() - ERROR activating sw package - errno=%d \n", __FUNCTION__, errno);
    		}
    	}

    	/* wait for the SW Manager activation execution */
    	waitpid(sm_cid, NULL, 0);
        _osal_task_pause(WORKERSLEEPSECS);
     }
}

static void* worker_b(void *pArg)
{
    int num = (int)pArg;
    int fd, len;
    time_t timebuf;
    char fname[1024] = {0};
    sprintf(fname, "/dev/null");
    pthread_t stPid = pthread_self();
    pthread_detach(stPid);
    /* Finally do out work */
    len = strlen(ctime(&timebuf));
    while(bRun){
        char *buf = malloc(sizeof(char)*(len+1));
        if(buf == NULL){
           fprintf(stderr, "malloc");
           exit(EXIT_FAILURE);
        }
        if((fd == open(fname, O_WRONLY|O_APPEND, 0600)) < 0){
            fprintf(stderr, "open");
            exit(EXIT_FAILURE);
        }
        time(&timebuf);
        strncpy(buf, ctime(&timebuf), len+1);
        write(fd, buf, len+1);
        free(buf);
        close(fd);
        _osal_task_pause(WORKERSLEEPSECS);
     }
}

int main(void)
{
    pid_t pid, sid;
    time_t timebuf;
    int fd, len;
    pthread_t worker_stPid[WORKERCOUNT];
    pthread_t worker_b_stPid[WORKERCOUNT];
    void *pRet;
    int i;
    fprintf(stderr, "zyan1 lpudated is starting.\n");
    pid = fork();
    if(pid < 0) {
        fprintf(stderr, "%s\n", strerror(errno));
        exit(EXIT_FAILURE);
    }
    if(pid > 0){
       /* In the parent, let's bail */
       exit(EXIT_SUCCESS);
    }

    /* In the child ... */
    /* Open the system log */
    //openlog("lpudated", LOG_PID, LOG_DAEMON);

    /* First, start a new session */
    if((sid = setsid())<0){
        fprintf(stderr, "%s\n", "setsid");
        exit(EXIT_FAILURE);
    }
#if 0
    if ((pid = fork()) < 0) {
    	fprintf(stderr, "%s %d, %s\n", __func__, __LINE__, strerror(errno));
        exit(EXIT_FAILURE);
    }else if (0!=pid) { //parent
    	exit(EXIT_SUCCESS);
    }
#endif
    /* Next, make / the current directory */
    if(chdir("/")<0){
       fprintf(stderr, "%s\n", "chdir");
       exit(EXIT_FAILURE);
    }
 
    /* Reset the file mode */
    umask(0);

    /* close the unneeded file descriptors */
    close(STDIN_FILENO);
    close(STDOUT_FILENO);
    //close(STDERR_FILENO);

    sigset_t *pstSigSet = &g_stSigSet;
    sigemptyset(pstSigSet);
    sigaddset(pstSigSet, SIGHUP);
    sigaddset(pstSigSet, SIGINT);
    sigaddset(pstSigSet, SIGTERM);
    sigaddset(pstSigSet, SIGKILL);
    if(pthread_sigmask(SIG_BLOCK, pstSigSet, NULL)<0){
	fprintf(stderr, "Cannot set signal mask\n");
        exit(EXIT_FAILURE);
    }

    for(i=0;i < WORKERCOUNT; i++){
    	pthread_create(&worker_stPid[i], NULL, worker_a, (void*)i);
    }

    for(i=0;i < WORKERCOUNT; i++){
        pthread_create(&worker_b_stPid[i], NULL, worker_b, (void*)i);
    }

    if(pthread_create(&g_stPid, NULL, sigthread_routine, NULL)<0){
	fprintf(stderr, "Can not start signal handler thread.\n");
        exit(EXIT_FAILURE);
    }

    pthread_join(g_stPid, &pRet);

    /* Close the system log and scram */
    //closelog();
    exit(EXIT_SUCCESS);
}

Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Kevin Buettner via Gdb-patches]

Hi Zhiyong,

One problem that I encountered on my Pi, which may explain the
behavior that you're seeing, is that recent 32-bit versions of
the Raspberry Pi OS are running a 64-bit/aarch64 kernel, but the
userland is 32-bit.

root@rpi4-2:~# /usr/bin/uname -m
aarch64
root@rpi4-2:~# file /usr/bin/ls
/usr/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV),
dynamically linked, interpreter /lib/ld-linux-armhf.so.3,
BuildID[sha1]=81004d065160807541b79235b23eea0e00a2d44e, for GNU/Linux
3.2.0, stripped

Note that uname -m returns aarch64, but that "ls" and other executables
are "ELF 32-bit ...".

The binutils-gdb configury uses uname -m to figure out for what
gdbserver host/target to build.  (host and target must be the same,
otherwise gdbserver won't build.) So it may be the case that you built
an aarch64 gdbserver instead of an arm gdbserver.  I think you can
check this as follows:

kev@rpi4-2:/mesquite2/sourceware-git/rpi-master/bld/gdbserver$ ls linux-{arm,aarch}*
linux-aarch32-low.o  linux-aarch32-tdesc.o  linux-arm-low.o  linux-arm-tdesc.o

If you also/instead see linux-aarch64-low.o and linux-aarch64-tdesc.o
in that list, then you probably have an aarch64 gdbserver.

When I tried my build with uname -m returning aarch64, the build
errored out because (I think) I was missing certain aarch64 header
files.  But I knew that I didn't want to build for aarch64, so I
abandoned that build.  What I ended up doing was making a wrapper for
uname which substituted 'arm' for 'aarch64'.  I put it in
/usr/local/bin, and /usr/local/bin is early in my PATH, so the
configury finds it first...

root@rpi4-2:~# uname -m
arm

Here's my /usr/local/bin/uname script:
- - - -
root@rpi4-2:~# cat /usr/local/bin/uname
#!/bin/bash

/usr/bin/uname $* | sed -e s/aarch64/arm/
- - - -

[ Yes, this is a hack, but I couldn't think of a cleaner way to do it. 
  I tried a configure line with "--host=arm-linux --target=host-linux",
  but that didn't work because something in the build wanted
  arm-linux-ar to exist and it didn't.  I could have made some
  symlinks, e.g. "ln -s /usr/bin/ar /usr/local/bin/arm-linux-ar", with
  similar symlinks for gcc, g++, ln, ranlib, etc, but that seemed like
  more work than my uname wrapper hack.]

I just checked my gdbserver build.  It's definitely getting into
arm_target::supports_hardware_single_step:

Breakpoint 1, linux_process_target::maybe_hw_step (
    this=0x8645c <the_arm_target>, thread=0x9df38)
    at /mesquite2/sourceware-git/rpi-master/bld/../../worktree-gdbserver/gdbserver/linux-low.cc:2442
2442	  if (supports_hardware_single_step ())
(gdb) s
arm_target::supports_hardware_single_step (this=0x8645c <the_arm_target>)
    at /mesquite2/sourceware-git/rpi-master/bld/../../worktree-gdbserver/gdbserver/linux-arm-low.cc:1042
1042	  return false;

Kevin

On Tue, 25 Jul 2023 04:21:00 +0000
"Yan, Zhiyong" <Zhiyong.Yan@windriver.com> wrote:

> Hi Kevin,
>      I test gdb11 on RaspBerry Pi4.
>      As you said, I can't produce this assert issue.
>      The direct reason is because supports_hardware_single_step () returns on RaspBerry Pi4, not like xilinux-zynq.
>      Please see attached pictures, we can see arm_target::supports_hardware_single_step () is never entered.
>      This assert only happens when supports_hardware_single_step () returns 'false'. On Raspberry Pi4, when I hardcoded supports_hardware_single_step () returns 'false', then assert happened.
>      For more information about " This assert only happens when supports_hardware_single_step () returns 'false'".
>      You can check https://sourceware.org/bugzilla/show_bug.cgi?id=30387
> 
>      So, the new question is why arm_target::supports_hardware_single_step () is never entered on Raspberry Pi4.
> 
> Best Regards.
> Zhiyong
> 
> 
> -----Original Message-----, 
> From: Kevin Buettner <kevinb@redhat.com> 
> Sent: Tuesday, July 25, 2023 11:37 AM
> To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
> Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
> Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step
> 
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> Hi Zhiyong,
> 
> I looked at the backtrace that you provided and see that maybe_hw_step() is being called from linux_process_target::resume_stopped_resumed_lwps,
> which is the one location where I wasn't able to convince myself that the assert should hold.
> 
> I was running your test case executable (osm) as an unprivileged user, so neither the syslog calls nor the sudo were working.  (Sudo could perhaps work, but it wanted to prompt for a password and stdin and stdout were closed.)  I've since modified it so that sudo isn't used and I'm using 'fprintf(stderr, ...)' instead of syslog - which is how I discovered that sudo wasn't working.  I've tried next'ing quite a lot, but so far I haven't reproduced the bug.  (Hopefully, the sudo isn't required to reproduce the problem.)
> 
> If you manage to reproduce the bug on a Raspberry Pi 4 (and tell me how to do it), that'd be great!
> 
> So, what I'm doing, using three separate terminals, in an attempt to reproduce the bug is:
> 
> 1) Run osm in terminal 1.  (I didn't want to mess with systemd.)  Once I start running it, I see a bunch of messages from the dd command.
> 
> 2) In terminal 2, I run:
> 
>    /path/to/gdbserver --debug --debug-format=all --remote-debug --event-loop-debug --once --attach :1234 $(pgrep osm)
> 
> 3) In terminal 3, I run:
> 
>    /path/to/gdb osm -x ./gdbx2
> 
> (I've changed the target remote command in gdbx2 to refer to localhost.)
> 
> I'm also attaching my hacked lupdated.c.  If you see anything wrong with what I'm trying, please let me know.
> 
> Kevin
> 
> On Mon, 24 Jul 2023 13:36:24 +0000
> "Yan, Zhiyong" <Zhiyong.Yan@windriver.com> wrote:
> 
> > Hi Kevin,
> >     The callstack of assert is attached.
> >     Please see attached gdbx2 which add more 'n' commands, on arm platform, keep execute 'n' command, this test case can trigger assert error.
> >
> >     Today, I didn't finish setting up test environments on RaspBerry Pi4. Before I produced this issue on Xilinx arm platform.
> >
> > Best Regards.
> > Zhiyong
> >
> > -----Original Message-----
> > From: Kevin Buettner <kevinb@redhat.com>
> > Sent: Saturday, July 22, 2023 4:50 AM
> > To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
> > Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
> > Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a 
> > pending thread whose last_resume_kind is resume_step
> >
> > CAUTION: This email comes from a non Wind River email account!
> > Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> > Hi Zhiyong,
> >
> > I set up a Raspberry Pi running a recent 32-bit Raspberry Pi OS so that I could test your patch.  I was able to build and run your test case, but I could not reproduce the bug on the Pi.
> >
> > I tested gdb.threads/*.exp using --target_board=native-gdbserver both 
> > with and without your patch.  Some of these tests are racy, but my 
> > conclusion from just looking at the PASSes and FAILs (after many test
> > runs) is that there are no regressions.
> >
> > But then I remembered to enable core dumps on the Pi and after running 
> > gdb.threads/pending-fork-event-detach/pending-fork-event-detach-main-v
> > fork by itself, I saw that it left a core file...
> >
> > $ make check RUNTESTFLAGS="--target_board=native-gdbserver" 
> > TESTS=gdb.threads/pending-fork-event-detach.exp
> > ...
> >                 === gdb Summary ===
> >
> > # of unexpected core files      1
> > # of expected passes            240
> >
> > The core file was from the running test case, not gdbserver, nor gdb.
> >
> > Looking at the core file in GDB shows...
> >
> > Program terminated with signal SIGTRAP, Trace/breakpoint trap.
> > #0  0x00010624 in break_here () at /mesquite2/sourceware-git/rpi-gdbserver/bld/../../worktree-gdbserver/gdb/testsuite/gdb.threads/pending-fork-event-detach.c:29
> > 29        x++;
> > [Current thread is 1 (Thread 0xf7e10440 (LWP 4835))]
> > (gdb) x/i $pc  
> > => 0x10624 <break_here+12>:     udf     #16  
> > (gdb) x/x $pc
> > 0x10624 <break_here+12>:        0xe7f001f0
> >
> > ...and in gdbserver/linux-aarch32-low.cc:
> >
> > #define arm_eabi_breakpoint 0xe7f001f0UL
> >
> > I think what's happened here is that the breakpoint added by your patch is left in place when GDB detaches the test case.  When it starts running again, it hits the software single step breakpoint and, since it's no longer under GDB control, it dies with a SIGTRAP.
> >
> > This core file is not created when I run the test using a gdbserver without your patch.
> >
> > I'm suspicious of the assert in linux_process_target::maybe_hw_step.
> > Currently, it looks like this:
> >
> > bool
> > linux_process_target::maybe_hw_step (thread_info *thread) {
> >   if (supports_hardware_single_step ())
> >     return true;
> >   else
> >     {
> >       /* GDBserver must insert single-step breakpoint for software
> >          single step.  */
> >       gdb_assert (has_single_step_breakpoints (thread));
> >       return false;
> >     }
> > }
> >
> > But, when Yao Qi introduced it back in June, 2016, it looked like
> > this:
> >
> > static int
> > maybe_hw_step (struct thread_info *thread) {
> >   if (can_hardware_single_step ())
> >     return 1;
> >   else
> >     {
> >       struct process_info *proc = get_thread_process (thread);
> >
> >       /* GDBserver must insert reinsert breakpoint for software
> >      single step.  */
> >       gdb_assert (has_reinsert_breakpoints (proc));
> >       return 0;
> >     }
> > }
> >
> > So, back is 2016, when it was introduced, it's clear that the assert was referring to breakpoints which needed to be reinserted.  Now, that's not at all obvious.
> >
> > Also, back in 2016, maybe_hw_step() was only called from two 
> > locations; in each case it was in a block in which the condition
> > lwp->bp_reinsert != 0 was true.  But now there are two other
> > calls; in one case, the software single step breakpoints have just been inserted, so that should be okay, but for the other case, in linux_process_target::resume_stopped_resumed_lwps, I'm less certain.
> >
> > In any case, could you comment out (or delete) the assert in a version of the source without your patch and let me know what happens?
> >
> > Also, if possible, I'd like to see a backtrace from where the assert occurs so that I can see which call to maybe_hw_step is responsible for triggering the failing assert.
> >
> > Kevin
> >  

RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Yan, Zhiyong]

Hi Kevin,
     Below is my PI's info:

root@bcm-2xxx-rpi4:~# uname -a
Linux bcm-2xxx-rpi4 5.15.110-yocto-standard #1 SMP PREEMPT Wed May 3 01:43:11 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

root@bcm-2xxx-rpi4:~# file gdbserver
gdbserver: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-aarch64.so.1, BuildID[sha1]=1e6ee58be0809d620fbdf3f86c8c4541f01ad9e9, for GNU/Linux 3.14.0, with debug_info, not stripped

root@bcm-2xxx-rpi4:~# file osm
osm: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=da7ee15aec73080ae3954d803b542bd9e8185c44, for GNU/Linux 3.14.0, with debug_info, not stripped
root@bcm-2xxx-rpi4:~#

     Both user app and OS are aarch64. 
[zyan1] On this, gdbserver supports hardware single step, assert doesn't happen.
-----------------------------------------------------------
Below is my Xilinx-zynq info: 
Last login: Wed Jul 26 03:03:09 2023
root@xilinx-zynq:~# uname -a
Linux xilinx-zynq 5.15.106-yocto-standard #1 SMP PREEMPT Tue Apr 11 03:06:10 UTC 2023 armv7l armv7l armv7l GNU/Linux
root@xilinx-zynq:~# which ls
/bin/ls
root@xilinx-zynq:~# file /bin/ls
/bin/ls: symbolic link to /bin/ls.coreutils
root@xilinx-zynq:~# file /bin/ls.coreutils
/bin/ls.coreutils: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=f8bf6bdad65965d53cdd9fd1ebfd00d191f4cbbc, for GNU/Linux 3.2.0, stripped
root@xilinx-zynq:~#

[zyan1] On this, gdbserver doesn't support hardware single step, assert can happen.

Best Regards.

-----Original Message-----
From: Kevin Buettner <kevinb@redhat.com> 
Sent: Tuesday, July 25, 2023 2:32 PM
To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Zhiyong,

One problem that I encountered on my Pi, which may explain the behavior that you're seeing, is that recent 32-bit versions of the Raspberry Pi OS are running a 64-bit/aarch64 kernel, but the userland is 32-bit.

root@rpi4-2:~# /usr/bin/uname -m
aarch64
root@rpi4-2:~# file /usr/bin/ls
/usr/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=81004d065160807541b79235b23eea0e00a2d44e, for GNU/Linux 3.2.0, stripped

Note that uname -m returns aarch64, but that "ls" and other executables are "ELF 32-bit ...".

The binutils-gdb configury uses uname -m to figure out for what gdbserver host/target to build.  (host and target must be the same, otherwise gdbserver won't build.) So it may be the case that you built an aarch64 gdbserver instead of an arm gdbserver.  I think you can check this as follows:

kev@rpi4-2:/mesquite2/sourceware-git/rpi-master/bld/gdbserver$ ls linux-{arm,aarch}* linux-aarch32-low.o  linux-aarch32-tdesc.o  linux-arm-low.o  linux-arm-tdesc.o

If you also/instead see linux-aarch64-low.o and linux-aarch64-tdesc.o in that list, then you probably have an aarch64 gdbserver.

When I tried my build with uname -m returning aarch64, the build errored out because (I think) I was missing certain aarch64 header files.  But I knew that I didn't want to build for aarch64, so I abandoned that build.  What I ended up doing was making a wrapper for uname which substituted 'arm' for 'aarch64'.  I put it in /usr/local/bin, and /usr/local/bin is early in my PATH, so the configury finds it first...

root@rpi4-2:~# uname -m
arm

Here's my /usr/local/bin/uname script:
- - - -
root@rpi4-2:~# cat /usr/local/bin/uname
#!/bin/bash

/usr/bin/uname $* | sed -e s/aarch64/arm/
- - - -

[ Yes, this is a hack, but I couldn't think of a cleaner way to do it.
  I tried a configure line with "--host=arm-linux --target=host-linux",
  but that didn't work because something in the build wanted
  arm-linux-ar to exist and it didn't.  I could have made some
  symlinks, e.g. "ln -s /usr/bin/ar /usr/local/bin/arm-linux-ar", with
  similar symlinks for gcc, g++, ln, ranlib, etc, but that seemed like
  more work than my uname wrapper hack.]

I just checked my gdbserver build.  It's definitely getting into
arm_target::supports_hardware_single_step:

Breakpoint 1, linux_process_target::maybe_hw_step (
    this=0x8645c <the_arm_target>, thread=0x9df38)
    at /mesquite2/sourceware-git/rpi-master/bld/../../worktree-gdbserver/gdbserver/linux-low.cc:2442
2442      if (supports_hardware_single_step ())
(gdb) s
arm_target::supports_hardware_single_step (this=0x8645c <the_arm_target>)
    at /mesquite2/sourceware-git/rpi-master/bld/../../worktree-gdbserver/gdbserver/linux-arm-low.cc:1042
1042      return false;

Kevin

On Tue, 25 Jul 2023 04:21:00 +0000
"Yan, Zhiyong" <Zhiyong.Yan@windriver.com> wrote:

> Hi Kevin,
>      I test gdb11 on RaspBerry Pi4.
>      As you said, I can't produce this assert issue.
>      The direct reason is because supports_hardware_single_step () returns on RaspBerry Pi4, not like xilinux-zynq.
>      Please see attached pictures, we can see arm_target::supports_hardware_single_step () is never entered.
>      This assert only happens when supports_hardware_single_step () returns 'false'. On Raspberry Pi4, when I hardcoded supports_hardware_single_step () returns 'false', then assert happened.
>      For more information about " This assert only happens when supports_hardware_single_step () returns 'false'".
>      You can check 
> https://sourceware.org/bugzilla/show_bug.cgi?id=30387
>
>      So, the new question is why arm_target::supports_hardware_single_step () is never entered on Raspberry Pi4.
>
> Best Regards.
> Zhiyong
>
>
> -----Original Message-----,
> From: Kevin Buettner <kevinb@redhat.com>
> Sent: Tuesday, July 25, 2023 11:37 AM
> To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
> Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
> Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a 
> pending thread whose last_resume_kind is resume_step
>
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> Hi Zhiyong,
>
> I looked at the backtrace that you provided and see that 
> maybe_hw_step() is being called from 
> linux_process_target::resume_stopped_resumed_lwps,
> which is the one location where I wasn't able to convince myself that the assert should hold.
>
> I was running your test case executable (osm) as an unprivileged user, 
> so neither the syslog calls nor the sudo were working.  (Sudo could 
> perhaps work, but it wanted to prompt for a password and stdin and 
> stdout were closed.)  I've since modified it so that sudo isn't used 
> and I'm using 'fprintf(stderr, ...)' instead of syslog - which is how 
> I discovered that sudo wasn't working.  I've tried next'ing quite a 
> lot, but so far I haven't reproduced the bug.  (Hopefully, the sudo 
> isn't required to reproduce the problem.)
>
> If you manage to reproduce the bug on a Raspberry Pi 4 (and tell me how to do it), that'd be great!
>
> So, what I'm doing, using three separate terminals, in an attempt to reproduce the bug is:
>
> 1) Run osm in terminal 1.  (I didn't want to mess with systemd.)  Once I start running it, I see a bunch of messages from the dd command.
>
> 2) In terminal 2, I run:
>
>    /path/to/gdbserver --debug --debug-format=all --remote-debug 
> --event-loop-debug --once --attach :1234 $(pgrep osm)
>
> 3) In terminal 3, I run:
>
>    /path/to/gdb osm -x ./gdbx2
>
> (I've changed the target remote command in gdbx2 to refer to 
> localhost.)
>
> I'm also attaching my hacked lupdated.c.  If you see anything wrong with what I'm trying, please let me know.
>
> Kevin
>
> On Mon, 24 Jul 2023 13:36:24 +0000
> "Yan, Zhiyong" <Zhiyong.Yan@windriver.com> wrote:
>
> > Hi Kevin,
> >     The callstack of assert is attached.
> >     Please see attached gdbx2 which add more 'n' commands, on arm platform, keep execute 'n' command, this test case can trigger assert error.
> >
> >     Today, I didn't finish setting up test environments on RaspBerry Pi4. Before I produced this issue on Xilinx arm platform.
> >
> > Best Regards.
> > Zhiyong
> >
> > -----Original Message-----
> > From: Kevin Buettner <kevinb@redhat.com>
> > Sent: Saturday, July 22, 2023 4:50 AM
> > To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
> > Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
> > Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a 
> > pending thread whose last_resume_kind is resume_step
> >
> > CAUTION: This email comes from a non Wind River email account!
> > Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> > Hi Zhiyong,
> >
> > I set up a Raspberry Pi running a recent 32-bit Raspberry Pi OS so that I could test your patch.  I was able to build and run your test case, but I could not reproduce the bug on the Pi.
> >
> > I tested gdb.threads/*.exp using --target_board=native-gdbserver 
> > both with and without your patch.  Some of these tests are racy, but 
> > my conclusion from just looking at the PASSes and FAILs (after many 
> > test
> > runs) is that there are no regressions.
> >
> > But then I remembered to enable core dumps on the Pi and after 
> > running 
> > gdb.threads/pending-fork-event-detach/pending-fork-event-detach-main
> > -v fork by itself, I saw that it left a core file...
> >
> > $ make check RUNTESTFLAGS="--target_board=native-gdbserver"
> > TESTS=gdb.threads/pending-fork-event-detach.exp
> > ...
> >                 === gdb Summary ===
> >
> > # of unexpected core files      1
> > # of expected passes            240
> >
> > The core file was from the running test case, not gdbserver, nor gdb.
> >
> > Looking at the core file in GDB shows...
> >
> > Program terminated with signal SIGTRAP, Trace/breakpoint trap.
> > #0  0x00010624 in break_here () at /mesquite2/sourceware-git/rpi-gdbserver/bld/../../worktree-gdbserver/gdb/testsuite/gdb.threads/pending-fork-event-detach.c:29
> > 29        x++;
> > [Current thread is 1 (Thread 0xf7e10440 (LWP 4835))]
> > (gdb) x/i $pc
> > => 0x10624 <break_here+12>:     udf     #16
> > (gdb) x/x $pc
> > 0x10624 <break_here+12>:        0xe7f001f0
> >
> > ...and in gdbserver/linux-aarch32-low.cc:
> >
> > #define arm_eabi_breakpoint 0xe7f001f0UL
> >
> > I think what's happened here is that the breakpoint added by your patch is left in place when GDB detaches the test case.  When it starts running again, it hits the software single step breakpoint and, since it's no longer under GDB control, it dies with a SIGTRAP.
> >
> > This core file is not created when I run the test using a gdbserver without your patch.
> >
> > I'm suspicious of the assert in linux_process_target::maybe_hw_step.
> > Currently, it looks like this:
> >
> > bool
> > linux_process_target::maybe_hw_step (thread_info *thread) {
> >   if (supports_hardware_single_step ())
> >     return true;
> >   else
> >     {
> >       /* GDBserver must insert single-step breakpoint for software
> >          single step.  */
> >       gdb_assert (has_single_step_breakpoints (thread));
> >       return false;
> >     }
> > }
> >
> > But, when Yao Qi introduced it back in June, 2016, it looked like
> > this:
> >
> > static int
> > maybe_hw_step (struct thread_info *thread) {
> >   if (can_hardware_single_step ())
> >     return 1;
> >   else
> >     {
> >       struct process_info *proc = get_thread_process (thread);
> >
> >       /* GDBserver must insert reinsert breakpoint for software
> >      single step.  */
> >       gdb_assert (has_reinsert_breakpoints (proc));
> >       return 0;
> >     }
> > }
> >
> > So, back is 2016, when it was introduced, it's clear that the assert was referring to breakpoints which needed to be reinserted.  Now, that's not at all obvious.
> >
> > Also, back in 2016, maybe_hw_step() was only called from two 
> > locations; in each case it was in a block in which the condition
> > lwp->bp_reinsert != 0 was true.  But now there are two other
> > calls; in one case, the software single step breakpoints have just been inserted, so that should be okay, but for the other case, in linux_process_target::resume_stopped_resumed_lwps, I'm less certain.
> >
> > In any case, could you comment out (or delete) the assert in a version of the source without your patch and let me know what happens?
> >
> > Also, if possible, I'd like to see a backtrace from where the assert occurs so that I can see which call to maybe_hw_step is responsible for triggering the failing assert.
> >
> > Kevin
> >

RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Yan, Zhiyong]

Hi Kevin,
     Possibly the conclusion is Raspberry with aarch64 supports gdb 'hardware single step', you need turn to another arm platform which doesn't supports gdb 'hardware single step' to produce this assert error.

Best Regards.
Zhiyong

-----Original Message-----
From: Yan, Zhiyong 
Sent: Tuesday, July 25, 2023 2:51 PM
To: Kevin Buettner <kevinb@redhat.com>
Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
Subject: RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

Hi Kevin,
     Below is my PI's info:

root@bcm-2xxx-rpi4:~# uname -a
Linux bcm-2xxx-rpi4 5.15.110-yocto-standard #1 SMP PREEMPT Wed May 3 01:43:11 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

root@bcm-2xxx-rpi4:~# file gdbserver
gdbserver: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-aarch64.so.1, BuildID[sha1]=1e6ee58be0809d620fbdf3f86c8c4541f01ad9e9, for GNU/Linux 3.14.0, with debug_info, not stripped

root@bcm-2xxx-rpi4:~# file osm
osm: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=da7ee15aec73080ae3954d803b542bd9e8185c44, for GNU/Linux 3.14.0, with debug_info, not stripped root@bcm-2xxx-rpi4:~#

     Both user app and OS are aarch64. 
[zyan1] On this, gdbserver supports hardware single step, assert doesn't happen.
-----------------------------------------------------------
Below is my Xilinx-zynq info: 
Last login: Wed Jul 26 03:03:09 2023
root@xilinx-zynq:~# uname -a
Linux xilinx-zynq 5.15.106-yocto-standard #1 SMP PREEMPT Tue Apr 11 03:06:10 UTC 2023 armv7l armv7l armv7l GNU/Linux root@xilinx-zynq:~# which ls /bin/ls root@xilinx-zynq:~# file /bin/ls
/bin/ls: symbolic link to /bin/ls.coreutils root@xilinx-zynq:~# file /bin/ls.coreutils
/bin/ls.coreutils: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=f8bf6bdad65965d53cdd9fd1ebfd00d191f4cbbc, for GNU/Linux 3.2.0, stripped root@xilinx-zynq:~#

[zyan1] On this, gdbserver doesn't support hardware single step, assert can happen.

Best Regards.

-----Original Message-----
From: Kevin Buettner <kevinb@redhat.com>
Sent: Tuesday, July 25, 2023 2:32 PM
To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Zhiyong,

One problem that I encountered on my Pi, which may explain the behavior that you're seeing, is that recent 32-bit versions of the Raspberry Pi OS are running a 64-bit/aarch64 kernel, but the userland is 32-bit.

root@rpi4-2:~# /usr/bin/uname -m
aarch64
root@rpi4-2:~# file /usr/bin/ls
/usr/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=81004d065160807541b79235b23eea0e00a2d44e, for GNU/Linux 3.2.0, stripped

Note that uname -m returns aarch64, but that "ls" and other executables are "ELF 32-bit ...".

The binutils-gdb configury uses uname -m to figure out for what gdbserver host/target to build.  (host and target must be the same, otherwise gdbserver won't build.) So it may be the case that you built an aarch64 gdbserver instead of an arm gdbserver.  I think you can check this as follows:

kev@rpi4-2:/mesquite2/sourceware-git/rpi-master/bld/gdbserver$ ls linux-{arm,aarch}* linux-aarch32-low.o  linux-aarch32-tdesc.o  linux-arm-low.o  linux-arm-tdesc.o

If you also/instead see linux-aarch64-low.o and linux-aarch64-tdesc.o in that list, then you probably have an aarch64 gdbserver.

When I tried my build with uname -m returning aarch64, the build errored out because (I think) I was missing certain aarch64 header files.  But I knew that I didn't want to build for aarch64, so I abandoned that build.  What I ended up doing was making a wrapper for uname which substituted 'arm' for 'aarch64'.  I put it in /usr/local/bin, and /usr/local/bin is early in my PATH, so the configury finds it first...

root@rpi4-2:~# uname -m
arm

Here's my /usr/local/bin/uname script:
- - - -
root@rpi4-2:~# cat /usr/local/bin/uname
#!/bin/bash

/usr/bin/uname $* | sed -e s/aarch64/arm/
- - - -

[ Yes, this is a hack, but I couldn't think of a cleaner way to do it.
  I tried a configure line with "--host=arm-linux --target=host-linux",
  but that didn't work because something in the build wanted
  arm-linux-ar to exist and it didn't.  I could have made some
  symlinks, e.g. "ln -s /usr/bin/ar /usr/local/bin/arm-linux-ar", with
  similar symlinks for gcc, g++, ln, ranlib, etc, but that seemed like
  more work than my uname wrapper hack.]

I just checked my gdbserver build.  It's definitely getting into
arm_target::supports_hardware_single_step:

Breakpoint 1, linux_process_target::maybe_hw_step (
    this=0x8645c <the_arm_target>, thread=0x9df38)
    at /mesquite2/sourceware-git/rpi-master/bld/../../worktree-gdbserver/gdbserver/linux-low.cc:2442
2442      if (supports_hardware_single_step ())
(gdb) s
arm_target::supports_hardware_single_step (this=0x8645c <the_arm_target>)
    at /mesquite2/sourceware-git/rpi-master/bld/../../worktree-gdbserver/gdbserver/linux-arm-low.cc:1042
1042      return false;

Kevin

On Tue, 25 Jul 2023 04:21:00 +0000
"Yan, Zhiyong" <Zhiyong.Yan@windriver.com> wrote:

> Hi Kevin,
>      I test gdb11 on RaspBerry Pi4.
>      As you said, I can't produce this assert issue.
>      The direct reason is because supports_hardware_single_step () returns on RaspBerry Pi4, not like xilinux-zynq.
>      Please see attached pictures, we can see arm_target::supports_hardware_single_step () is never entered.
>      This assert only happens when supports_hardware_single_step () returns 'false'. On Raspberry Pi4, when I hardcoded supports_hardware_single_step () returns 'false', then assert happened.
>      For more information about " This assert only happens when supports_hardware_single_step () returns 'false'".
>      You can check
> https://sourceware.org/bugzilla/show_bug.cgi?id=30387
>
>      So, the new question is why arm_target::supports_hardware_single_step () is never entered on Raspberry Pi4.
>
> Best Regards.
> Zhiyong
>
>
> -----Original Message-----,
> From: Kevin Buettner <kevinb@redhat.com>
> Sent: Tuesday, July 25, 2023 11:37 AM
> To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
> Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
> Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a 
> pending thread whose last_resume_kind is resume_step
>
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> Hi Zhiyong,
>
> I looked at the backtrace that you provided and see that
> maybe_hw_step() is being called from
> linux_process_target::resume_stopped_resumed_lwps,
> which is the one location where I wasn't able to convince myself that the assert should hold.
>
> I was running your test case executable (osm) as an unprivileged user, 
> so neither the syslog calls nor the sudo were working.  (Sudo could 
> perhaps work, but it wanted to prompt for a password and stdin and 
> stdout were closed.)  I've since modified it so that sudo isn't used 
> and I'm using 'fprintf(stderr, ...)' instead of syslog - which is how 
> I discovered that sudo wasn't working.  I've tried next'ing quite a 
> lot, but so far I haven't reproduced the bug.  (Hopefully, the sudo 
> isn't required to reproduce the problem.)
>
> If you manage to reproduce the bug on a Raspberry Pi 4 (and tell me how to do it), that'd be great!
>
> So, what I'm doing, using three separate terminals, in an attempt to reproduce the bug is:
>
> 1) Run osm in terminal 1.  (I didn't want to mess with systemd.)  Once I start running it, I see a bunch of messages from the dd command.
>
> 2) In terminal 2, I run:
>
>    /path/to/gdbserver --debug --debug-format=all --remote-debug 
> --event-loop-debug --once --attach :1234 $(pgrep osm)
>
> 3) In terminal 3, I run:
>
>    /path/to/gdb osm -x ./gdbx2
>
> (I've changed the target remote command in gdbx2 to refer to
> localhost.)
>
> I'm also attaching my hacked lupdated.c.  If you see anything wrong with what I'm trying, please let me know.
>
> Kevin
>
> On Mon, 24 Jul 2023 13:36:24 +0000
> "Yan, Zhiyong" <Zhiyong.Yan@windriver.com> wrote:
>
> > Hi Kevin,
> >     The callstack of assert is attached.
> >     Please see attached gdbx2 which add more 'n' commands, on arm platform, keep execute 'n' command, this test case can trigger assert error.
> >
> >     Today, I didn't finish setting up test environments on RaspBerry Pi4. Before I produced this issue on Xilinx arm platform.
> >
> > Best Regards.
> > Zhiyong
> >
> > -----Original Message-----
> > From: Kevin Buettner <kevinb@redhat.com>
> > Sent: Saturday, July 22, 2023 4:50 AM
> > To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
> > Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
> > Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a 
> > pending thread whose last_resume_kind is resume_step
> >
> > CAUTION: This email comes from a non Wind River email account!
> > Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> > Hi Zhiyong,
> >
> > I set up a Raspberry Pi running a recent 32-bit Raspberry Pi OS so that I could test your patch.  I was able to build and run your test case, but I could not reproduce the bug on the Pi.
> >
> > I tested gdb.threads/*.exp using --target_board=native-gdbserver 
> > both with and without your patch.  Some of these tests are racy, but 
> > my conclusion from just looking at the PASSes and FAILs (after many 
> > test
> > runs) is that there are no regressions.
> >
> > But then I remembered to enable core dumps on the Pi and after 
> > running 
> > gdb.threads/pending-fork-event-detach/pending-fork-event-detach-main
> > -v fork by itself, I saw that it left a core file...
> >
> > $ make check RUNTESTFLAGS="--target_board=native-gdbserver"
> > TESTS=gdb.threads/pending-fork-event-detach.exp
> > ...
> >                 === gdb Summary ===
> >
> > # of unexpected core files      1
> > # of expected passes            240
> >
> > The core file was from the running test case, not gdbserver, nor gdb.
> >
> > Looking at the core file in GDB shows...
> >
> > Program terminated with signal SIGTRAP, Trace/breakpoint trap.
> > #0  0x00010624 in break_here () at /mesquite2/sourceware-git/rpi-gdbserver/bld/../../worktree-gdbserver/gdb/testsuite/gdb.threads/pending-fork-event-detach.c:29
> > 29        x++;
> > [Current thread is 1 (Thread 0xf7e10440 (LWP 4835))]
> > (gdb) x/i $pc
> > => 0x10624 <break_here+12>:     udf     #16
> > (gdb) x/x $pc
> > 0x10624 <break_here+12>:        0xe7f001f0
> >
> > ...and in gdbserver/linux-aarch32-low.cc:
> >
> > #define arm_eabi_breakpoint 0xe7f001f0UL
> >
> > I think what's happened here is that the breakpoint added by your patch is left in place when GDB detaches the test case.  When it starts running again, it hits the software single step breakpoint and, since it's no longer under GDB control, it dies with a SIGTRAP.
> >
> > This core file is not created when I run the test using a gdbserver without your patch.
> >
> > I'm suspicious of the assert in linux_process_target::maybe_hw_step.
> > Currently, it looks like this:
> >
> > bool
> > linux_process_target::maybe_hw_step (thread_info *thread) {
> >   if (supports_hardware_single_step ())
> >     return true;
> >   else
> >     {
> >       /* GDBserver must insert single-step breakpoint for software
> >          single step.  */
> >       gdb_assert (has_single_step_breakpoints (thread));
> >       return false;
> >     }
> > }
> >
> > But, when Yao Qi introduced it back in June, 2016, it looked like
> > this:
> >
> > static int
> > maybe_hw_step (struct thread_info *thread) {
> >   if (can_hardware_single_step ())
> >     return 1;
> >   else
> >     {
> >       struct process_info *proc = get_thread_process (thread);
> >
> >       /* GDBserver must insert reinsert breakpoint for software
> >      single step.  */
> >       gdb_assert (has_reinsert_breakpoints (proc));
> >       return 0;
> >     }
> > }
> >
> > So, back is 2016, when it was introduced, it's clear that the assert was referring to breakpoints which needed to be reinserted.  Now, that's not at all obvious.
> >
> > Also, back in 2016, maybe_hw_step() was only called from two 
> > locations; in each case it was in a block in which the condition
> > lwp->bp_reinsert != 0 was true.  But now there are two other
> > calls; in one case, the software single step breakpoints have just been inserted, so that should be okay, but for the other case, in linux_process_target::resume_stopped_resumed_lwps, I'm less certain.
> >
> > In any case, could you comment out (or delete) the assert in a version of the source without your patch and let me know what happens?
> >
> > Also, if possible, I'd like to see a backtrace from where the assert occurs so that I can see which call to maybe_hw_step is responsible for triggering the failing assert.
> >
> > Kevin
> >

Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Kevin Buettner via Gdb-patches]

Hi Zhiyong,

I've finally been able to reproduce the bug on a Raspberry Pi. On a
different SD card, I installed 32-bit Ubunutu server 20.04.5 LTS.  It
seems to have both a 32-bit (arm) kernel + 32-bit userland.  I.e...

kev@rpi4-3:~/Downloads/bz30387$ uname -m
armv7l
kev@rpi4-3:~/Downloads/bz30387$ file ./osm
./osm: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=81e7e2b5dfba0fe35f1f1a6af2ee558efbdafa7f, for GNU/Linux 3.2.0, with debug_info, not stripped

(Compare the above output to that your reported on your Pi; on your
Pi, uname -m reported aarch64 and the 'file' command showed 64-bit
aarch64 binaries.)

The internal error appears to be the same as that described in your
bug report as well as on the gdb-patches list:

/mesquite2/sourceware-git/rpi-arm-master/bld/../../worktree-master/gdbserver/linux-low.cc:2448: A problem internal to GDBserver has been detected.
maybe_hw_step: Assertion `has_single_step_breakpoints (thread)' failed.

Now that I've reproduced it, I want to retest gdb.threads/*.exp to see
if any of those tests show the same failure.  If not, I'll try to adapt
your test case into one suitable for the gdb test suite.

I have an alternate patch in mind, which I'll try out too.  If it
works out, I'll ask you to test it on your hardware...

Kevin

RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Yan, Zhiyong]

Hi Kevin,
     Thanks for your effort.

Best Regards.
Zhiyong

-----Original Message-----
From: Kevin Buettner <kevinb@redhat.com> 
Sent: Wednesday, July 26, 2023 11:59 AM
To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
Cc: gdb-patches@sourceware.org; luis.machado@arm.com; tom@tromey.com
Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Zhiyong,

I've finally been able to reproduce the bug on a Raspberry Pi. On a different SD card, I installed 32-bit Ubunutu server 20.04.5 LTS.  It seems to have both a 32-bit (arm) kernel + 32-bit userland.  I.e...

kev@rpi4-3:~/Downloads/bz30387$ uname -m armv7l kev@rpi4-3:~/Downloads/bz30387$ file ./osm
./osm: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=81e7e2b5dfba0fe35f1f1a6af2ee558efbdafa7f, for GNU/Linux 3.2.0, with debug_info, not stripped

(Compare the above output to that your reported on your Pi; on your Pi, uname -m reported aarch64 and the 'file' command showed 64-bit
aarch64 binaries.)

The internal error appears to be the same as that described in your bug report as well as on the gdb-patches list:

/mesquite2/sourceware-git/rpi-arm-master/bld/../../worktree-master/gdbserver/linux-low.cc:2448: A problem internal to GDBserver has been detected.
maybe_hw_step: Assertion `has_single_step_breakpoints (thread)' failed.

Now that I've reproduced it, I want to retest gdb.threads/*.exp to see if any of those tests show the same failure.  If not, I'll try to adapt your test case into one suitable for the gdb test suite.

I have an alternate patch in mind, which I'll try out too.  If it works out, I'll ask you to test it on your hardware...

Kevin

[PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[zhiyong.yan]

From: Zhiyong Yan <zhiyong.yan@windriver.com>

Gdb should not assume pending threads always generate “a non-gdbserver trap event”, for example “Signal 17” event could happen. Now that resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point must already exist, resume_one_thread() should ensure the software breaking point is installed although the thread is pending.

Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com
Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387
---
 gdbserver/linux-low.cc | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/gdbserver/linux-low.cc b/gdbserver/linux-low.cc
index e6a39202a98..d29881174db 100644
--- a/gdbserver/linux-low.cc
+++ b/gdbserver/linux-low.cc
@@ -4671,7 +4671,16 @@ linux_process_target::resume_one_thread (thread_info *thread,
       proceed_one_lwp (thread, NULL);
     }
   else
-    threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
+    {
+      threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
+      if (thread->last_resume_kind == resume_step)
+	{
+	  /* If resume_step is required by GDB, 
+	     install single-step breakpoint.  */
+	  if (supports_software_single_step ())
+	    install_software_single_step_breakpoints (lwp);
+	}
+    }
 
   thread->last_status.set_ignore ();
   lwp->resume = NULL;
-- 
2.25.1

[PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[zhiyong.yan]

From: Zhiyong Yan <zhiyong.yan@windriver.com>

Gdb should not assume pending threads always generate “a non-gdbserver trap event”, for example “Signal 17” event could happen. Now that resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point must already exist, resume_one_thread() should ensure the software breaking point is installed although the thread is pending.

Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com
Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387
---
 gdbserver/linux-low.cc | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/gdbserver/linux-low.cc b/gdbserver/linux-low.cc
index e6a39202a98..7d8bbf71ddc 100644
--- a/gdbserver/linux-low.cc
+++ b/gdbserver/linux-low.cc
@@ -4671,7 +4671,16 @@ linux_process_target::resume_one_thread (thread_info *thread,
       proceed_one_lwp (thread, NULL);
     }
   else
-    threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
+    {
+      threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
+      if (thread->last_resume_kind == resume_step)
+	{
+	  /* If resume_step is required by GDB, 
+	     install single-step breakpoint.  */
+	  if (supports_software_single_step())
+	    install_software_single_step_breakpoints (lwp);
+	}
+    }
 
   thread->last_status.set_ignore ();
   lwp->resume = NULL;
-- 
2.35.5

Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Alexandra Petlanova Hajkova via Gdb-patches]

On Mon, Jul 3, 2023 at 5:05 AM <zhiyong.yan@windriver.com> wrote:

> From: Zhiyong Yan <zhiyong.yan@windriver.com>
>
> Gdb should not assume pending threads always generate “a non-gdbserver
> trap event”, for example “Signal 17” event could happen. Now that
> resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point
> must already exist, resume_one_thread() should ensure the software breaking
> point is installed although the thread is pending.
>
> Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387


I can confirm this patch causes no regression Fedora-Rawhide, ppc64le.

RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Yan, Zhiyong]

Hi Alexandra,
     Thanks for your feedback.

Best Regards.
Zhiyong

From: Alexandra Petlanova Hajkova <ahajkova@redhat.com>
Sent: Monday, July 10, 2023 8:46 PM
To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
Cc: gdb-patches@sourceware.org; tom@tromey.com
Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.


On Mon, Jul 3, 2023 at 5:05 AM <zhiyong.yan@windriver.com<mailto:zhiyong.yan@windriver.com>> wrote:
From: Zhiyong Yan <zhiyong.yan@windriver.com<mailto:zhiyong.yan@windriver.com>>

Gdb should not assume pending threads always generate “a non-gdbserver trap event”, for example “Signal 17” event could happen. Now that resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point must already exist, resume_one_thread() should ensure the software breaking point is installed although the thread is pending.

Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com<mailto:zhiyong.yan@windriver.com>
Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387<https://urldefense.com/v3/__https:/sourceware.org/bugzilla/show_bug.cgi?id=30387__;!!AjveYdw8EvQ!fkg3Ae-1jPyI75siqABZuT57DvfkxmNtWeqO1A1Jly9PEvrBravJqc2Eou3b6H0Qfc31GRg9fixINWkyyZW8IwY$>

I can confirm this patch causes no regression Fedora-Rawhide, ppc64le.

Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Kevin Buettner via Gdb-patches]

Hi Zhiyong,

See my comments below; there's still one formatting nit, but I also
have a question about whether this is the right place to fix the bug
that you're seeing.

Also, I've added Luis to the CC list since he knows a lot more about
the ARM architecture than I do.

On Mon, 3 Jul 2023 11:04:58 +0800
<zhiyong.yan@windriver.com> wrote:

> From: Zhiyong Yan <zhiyong.yan@windriver.com>
> 
> Gdb should not assume pending threads always generate "a non-gdbserver trap event", for example "Signal 17" event could happen. Now that resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point must already exist, resume_one_thread() should ensure the software breaking point is installed although the thread is pending.
> 
> Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387
> ---
>  gdbserver/linux-low.cc | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/gdbserver/linux-low.cc b/gdbserver/linux-low.cc
> index e6a39202a98..7d8bbf71ddc 100644
> --- a/gdbserver/linux-low.cc
> +++ b/gdbserver/linux-low.cc
> @@ -4671,7 +4671,16 @@ linux_process_target::resume_one_thread (thread_info *thread,
>        proceed_one_lwp (thread, NULL);
>      }
>    else
> -    threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
> +    {
> +      threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
> +      if (thread->last_resume_kind == resume_step)
> +	{
> +	  /* If resume_step is required by GDB, 
> +	     install single-step breakpoint.  */
> +	  if (supports_software_single_step())

Formatting nit: you need a space between 'supports_software_single_step'
and the left paren.

> +	    install_software_single_step_breakpoints (lwp);
> +	}
> +    }
>  
>    thread->last_status.set_ignore ();
>    lwp->resume = NULL;

With regard to the change itself...

If the debugging printf is accurate, the LWP is being left in a
stopped state.  If that's the case, then why are software single step
breakpoints being inserted?  It seems to me that you'd only want
to insert these breakpoints when the thread is actually about to
resume.

That said, I have no doubt that this change works for you, so clearly
there's something going on which I do not understand.  I'd like to
understand the situation better before approving it.  Also, once you
have an explanation, please update the code comments and/or commit log
message as appropriate.  My personal preference is for a concise
explanation to be placed in the code comments with any additional,
longer winded explanations or examples being placed in the commit log
message.

Kevin

RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Yan, Zhiyong]

[-- Attachment #1: Type: text/plain, Size: 4753 bytes --]

Hi Kevin,
     My last mail's attachment is a little big, I am not sure if you can receive it. I zipped the log, and send again.

     -  For "format nit", I will work on it again.

     -  For your concern "I'd like to understand the situation better before approving it.",
        Please check attached " step5-assert.txt".
        Attached step5-assert.txt is a debug log which contains several "gdb step next" output.
        For example, at line 29, resume_one_thread() doesn't call process_one_lwp() because  thread LWP 316.316 is pending, as a result the software breaking point is not installed. Later, if this pending thread makes "wait_1: Hit a non-gdbserver trap event." happen, stop_all_lwps() -> "prepare_resume_reply: Writing resume reply for" ->...
        In such case, "gdb N" can finish without assert error. 
        But in step5-assert.txt, from line 14923 to 14943, we can see the pending thread make below happen,
         "
              wait_for_event_filtered: Got a pending child 316
              362.994099   [threads] wait_for_event_filtered: Got an event from pending child 316 (117f)
              362.994379   [threads] wait_1: Ignored signal 17 for LWP 316
         "
         In this case "resume_stopped_resumed_lwps" will resume every 'stopped-resumed' thread, but a thread (previously pending) has no software break point installed, then resume_stopped_resumed_lwps() asserts failed in maybe_hw_step().
     

Best Regards.
Zhiyong

-----Original Message-----
From: Kevin Buettner <kevinb@redhat.com> 
Sent: Wednesday, July 12, 2023 2:43 AM
To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
Cc: gdb-patches@sourceware.org; tom@tromey.com; luis.machado@arm.com
Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Zhiyong,

See my comments below; there's still one formatting nit, but I also have a question about whether this is the right place to fix the bug that you're seeing.

Also, I've added Luis to the CC list since he knows a lot more about the ARM architecture than I do.

On Mon, 3 Jul 2023 11:04:58 +0800
<zhiyong.yan@windriver.com> wrote:

> From: Zhiyong Yan <zhiyong.yan@windriver.com>
>
> Gdb should not assume pending threads always generate "a non-gdbserver trap event", for example "Signal 17" event could happen. Now that resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point must already exist, resume_one_thread() should ensure the software breaking point is installed although the thread is pending.
>
> Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387
> ---
>  gdbserver/linux-low.cc | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/gdbserver/linux-low.cc b/gdbserver/linux-low.cc index 
> e6a39202a98..7d8bbf71ddc 100644
> --- a/gdbserver/linux-low.cc
> +++ b/gdbserver/linux-low.cc
> @@ -4671,7 +4671,16 @@ linux_process_target::resume_one_thread (thread_info *thread,
>        proceed_one_lwp (thread, NULL);
>      }
>    else
> -    threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
> +    {
> +      threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
> +      if (thread->last_resume_kind == resume_step)
> +     {
> +       /* If resume_step is required by GDB,
> +          install single-step breakpoint.  */
> +       if (supports_software_single_step())

Formatting nit: you need a space between 'supports_software_single_step'
and the left paren.

> +         install_software_single_step_breakpoints (lwp);
> +     }
> +    }
>
>    thread->last_status.set_ignore ();
>    lwp->resume = NULL;

With regard to the change itself...

If the debugging printf is accurate, the LWP is being left in a stopped state.  If that's the case, then why are software single step breakpoints being inserted?  It seems to me that you'd only want to insert these breakpoints when the thread is actually about to resume.

That said, I have no doubt that this change works for you, so clearly there's something going on which I do not understand.  I'd like to understand the situation better before approving it.  Also, once you have an explanation, please update the code comments and/or commit log message as appropriate.  My personal preference is for a concise explanation to be placed in the code comments with any additional, longer winded explanations or examples being placed in the commit log message.

Kevin


[-- Attachment #2: step5-assert.zip --]
[-- Type: application/x-zip-compressed, Size: 126847 bytes --]

RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Yan, Zhiyong]

Hi all,
     For "format nit", I have sent new patch again. Because add more mail CC, I send patch triple today, they same. Sorry for bothering you.

Best Regards.
Zhiyong

-----Original Message-----
From: Yan, Zhiyong 
Sent: Wednesday, July 12, 2023 10:34 AM
To: Kevin Buettner <kevinb@redhat.com>
Cc: gdb-patches@sourceware.org; tom@tromey.com; luis.machado@arm.com
Subject: RE: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

Hi Kevin,
     My last mail's attachment is a little big, I am not sure if you can receive it. I zipped the log, and send again.

     -  For "format nit", I will work on it again.

     -  For your concern "I'd like to understand the situation better before approving it.",
        Please check attached " step5-assert.txt".
        Attached step5-assert.txt is a debug log which contains several "gdb step next" output.
        For example, at line 29, resume_one_thread() doesn't call process_one_lwp() because  thread LWP 316.316 is pending, as a result the software breaking point is not installed. Later, if this pending thread makes "wait_1: Hit a non-gdbserver trap event." happen, stop_all_lwps() -> "prepare_resume_reply: Writing resume reply for" ->...
        In such case, "gdb N" can finish without assert error. 
        But in step5-assert.txt, from line 14923 to 14943, we can see the pending thread make below happen,
         "
              wait_for_event_filtered: Got a pending child 316
              362.994099   [threads] wait_for_event_filtered: Got an event from pending child 316 (117f)
              362.994379   [threads] wait_1: Ignored signal 17 for LWP 316
         "
         In this case "resume_stopped_resumed_lwps" will resume every 'stopped-resumed' thread, but a thread (previously pending) has no software break point installed, then resume_stopped_resumed_lwps() asserts failed in maybe_hw_step().
     

Best Regards.
Zhiyong

-----Original Message-----
From: Kevin Buettner <kevinb@redhat.com>
Sent: Wednesday, July 12, 2023 2:43 AM
To: Yan, Zhiyong <Zhiyong.Yan@windriver.com>
Cc: gdb-patches@sourceware.org; tom@tromey.com; luis.machado@arm.com
Subject: Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Zhiyong,

See my comments below; there's still one formatting nit, but I also have a question about whether this is the right place to fix the bug that you're seeing.

Also, I've added Luis to the CC list since he knows a lot more about the ARM architecture than I do.

On Mon, 3 Jul 2023 11:04:58 +0800
<zhiyong.yan@windriver.com> wrote:

> From: Zhiyong Yan <zhiyong.yan@windriver.com>
>
> Gdb should not assume pending threads always generate "a non-gdbserver trap event", for example "Signal 17" event could happen. Now that resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point must already exist, resume_one_thread() should ensure the software breaking point is installed although the thread is pending.
>
> Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387
> ---
>  gdbserver/linux-low.cc | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/gdbserver/linux-low.cc b/gdbserver/linux-low.cc index 
> e6a39202a98..7d8bbf71ddc 100644
> --- a/gdbserver/linux-low.cc
> +++ b/gdbserver/linux-low.cc
> @@ -4671,7 +4671,16 @@ linux_process_target::resume_one_thread (thread_info *thread,
>        proceed_one_lwp (thread, NULL);
>      }
>    else
> -    threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
> +    {
> +      threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
> +      if (thread->last_resume_kind == resume_step)
> +     {
> +       /* If resume_step is required by GDB,
> +          install single-step breakpoint.  */
> +       if (supports_software_single_step())

Formatting nit: you need a space between 'supports_software_single_step'
and the left paren.

> +         install_software_single_step_breakpoints (lwp);
> +     }
> +    }
>
>    thread->last_status.set_ignore ();
>    lwp->resume = NULL;

With regard to the change itself...

If the debugging printf is accurate, the LWP is being left in a stopped state.  If that's the case, then why are software single step breakpoints being inserted?  It seems to me that you'd only want to insert these breakpoints when the thread is actually about to resume.

That said, I have no doubt that this change works for you, so clearly there's something going on which I do not understand.  I'd like to understand the situation better before approving it.  Also, once you have an explanation, please update the code comments and/or commit log message as appropriate.  My personal preference is for a concise explanation to be placed in the code comments with any additional, longer winded explanations or examples being placed in the commit log message.

Kevin

Re: [PATCH] gdbserver: Install single-step breakpoint for a pending thread whose last_resume_kind is resume_step

[Luis Machado via Gdb-patches]

Hi,

On 7/11/23 19:42, Kevin Buettner wrote:
> Hi Zhiyong,
> 
> See my comments below; there's still one formatting nit, but I also
> have a question about whether this is the right place to fix the bug
> that you're seeing.
> 
> Also, I've added Luis to the CC list since he knows a lot more about
> the ARM architecture than I do.

I was aware of this one, but since it is generic code, I thought I'd defer it to the
global maintainers. It doesn't look like something that can/should be handled in arch-specific
layers. But I can revisit it if people think otherwise.

> 
> On Mon, 3 Jul 2023 11:04:58 +0800
> <zhiyong.yan@windriver.com> wrote:
> 
>> From: Zhiyong Yan <zhiyong.yan@windriver.com>
>>
>> Gdb should not assume pending threads always generate "a non-gdbserver trap event", for example "Signal 17" event could happen. Now that resume_stopped_resumed_lwps() -> may_hw_step() assumes that the break point must already exist, resume_one_thread() should ensure the software breaking point is installed although the thread is pending.
>>
>> Signed-off-by: Zhiyong Yan zhiyong.yan@windriver.com
>> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30387
>> ---
>>  gdbserver/linux-low.cc | 11 ++++++++++-
>>  1 file changed, 10 insertions(+), 1 deletion(-)
>>
>> diff --git a/gdbserver/linux-low.cc b/gdbserver/linux-low.cc
>> index e6a39202a98..7d8bbf71ddc 100644
>> --- a/gdbserver/linux-low.cc
>> +++ b/gdbserver/linux-low.cc
>> @@ -4671,7 +4671,16 @@ linux_process_target::resume_one_thread (thread_info *thread,
>>        proceed_one_lwp (thread, NULL);
>>      }
>>    else
>> -    threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
>> +    {
>> +      threads_debug_printf ("leaving LWP %ld stopped", lwpid_of (thread));
>> +      if (thread->last_resume_kind == resume_step)
>> +	{
>> +	  /* If resume_step is required by GDB, 
>> +	     install single-step breakpoint.  */
>> +	  if (supports_software_single_step())
> 
> Formatting nit: you need a space between 'supports_software_single_step'
> and the left paren.
> 
>> +	    install_software_single_step_breakpoints (lwp);
>> +	}
>> +    }
>>  
>>    thread->last_status.set_ignore ();
>>    lwp->resume = NULL;
> 
> With regard to the change itself...
> 
> If the debugging printf is accurate, the LWP is being left in a
> stopped state.  If that's the case, then why are software single step
> breakpoints being inserted?  It seems to me that you'd only want
> to insert these breakpoints when the thread is actually about to
> resume.
> 
> That said, I have no doubt that this change works for you, so clearly
> there's something going on which I do not understand.  I'd like to
> understand the situation better before approving it.  Also, once you
> have an explanation, please update the code comments and/or commit log
> message as appropriate.  My personal preference is for a concise
> explanation to be placed in the code comments with any additional,
> longer winded explanations or examples being placed in the commit log
> message.
> 
> Kevin
>