Multi-threading support on DragonFlyBSD

Multi-threading support on DragonFlyBSD

[Sergey Zigachev via Gdb]

Hi,
I'm trying to enable debugging of multi-threaded programs on 
DragonFlyBSD using gdb and have several questions.

I checked out gdb 15.2 sources, applied out-of-tree patches with 
native-dependent code implementation for amd64 DragonFlyBSD target. 
These patches have basic support for enumerating registers, accessing 
memory etc. All the OS-specific things. Single-thread debugging works 
well with these patches. However, multiprocess debugging, async and 
non-stop modes are not implemented. As is any notion of multiple threads 
in the inferior. My goal is to add missing bits and pieces both in 
userspace (gdb's nat-dep) and kernelspace (mostly ptrace syscall).

I'm at the stage where I have the basics working. I can report events 
from different threads, suspend/resume single or all threads, inspect 
TLS variables etc. Schedule-locking mode also supported.

My questions concern mostly step-over ("step") command issued to 
multiple threads, not at once of course. Although I see approx. the same 
behavior with the "next" command.
I wrote a little test C program that has main thread spawning two extra 
threads running two separate but mostly identical functions. Each 
function prints message to output, sleeps for several seconds, prints 
another message, and then returns. Main function waits for both threads 
to complete.

First test run.
I put a breakpoint at the start of the first function, run the program 
and input "step" command until process exits. Essentially I'm debbuging 
as-if it was a single-thread program. Everything works as expected. By 
that i mean after each "step" command gdb steps on the next source line 
until the end of the function. It skips instructions that are part of a 
single source line, and skips functions that don't have debugging 
symbols. This is the baseline I'm trying to replicate with stepping over 
multiple threads.

Second test run.
Same as the first run but this time I put a breakpoint at the start of 
both functions. This results in gdb stopping at different threads when I 
enter "step" command.
I observe the following issues:
   - each source line gets hit several times, I think by the number of 
asm instructions it implements
   - gdb stops at functions without debug symbols; sometimes it can step 
over and continue; sometimes it doesn't if it can't find the end of the 
function
Basically, gdb's acting as if it's running "stepi" command.

With trial and error and some printf sprinkled around I came up with a 
theory why that could happen. Consider this part of the function:

 > 23 printf("Running thread thr=%d sleep=%d!\n", d->thr_num, d->sleep);
 >   0x0000000000400b2d <+29>:    mov    0xc(%rdi),%edx
 >   0x0000000000400b30 <+32>:    mov    $0x400cff,%edi
 >   0x0000000000400b35 <+37>:    xor    %eax,%eax
 >   0x0000000000400b37 <+39>:    call   0x4007f0 <printf@plt>
 >
 > 24 sleep(d->sleep);
 >   0x0000000000400b3c <+44>:    mov    0xc(%rbx),%edi

When gdb stops at line 23 and I enter "step" command, it calculates step 
range thread_info->control.step_range_{start,end}. In this case it will 
be [0x400b2d-0x400b3c). Then it uses this info to step through asm 
instructions 4 times transparently to the user.
The happy call chain looks like this (infrun.c):
   - clear_proceed_status_thread *for each thread*
   - prepare_one_step (infcmd.c) -- this one updates the range
   - proceed
   - 4x fetch_inferior_event
   - stop at next line waiting for input
This works for single-thread case because gdb clears thread_info state 
and then loads range for the *current* thread after clearing.
If I return event for a different thread, then its info was cleared and 
haven't been loaded by gdb before that. Gdb can't recognize this signal 
and assumes it's a "random signal". It goes downhill from there, I think 
most consecutive signals become "random".

I don't want to track down heisenbugs caused by my dirty hands touching 
gdb thread state and accidentally braking invariants. Hence this 
message, any advice is appreciated.

My questions:
   1. Does my theory make sense?
   2. Is it a job of native-dependent code to save/restore thread state 
before reporting a signal?
   3. If yes, what else should I store, per-thread, per-process etc?

Thanks,
Sergey

Re: Multi-threading support on DragonFlyBSD

[Sergey Zigachev via Gdb]

Ok, skimmed through linux target I saw this:

   /* In all-stop, give preference to the LWP that is being
      single-stepped.  There will be at most one, and it will be the
      LWP that the core is most interested in.  If we didn't do this,
      then we'd have to handle pending step SIGTRAPs somehow in case
      the core later continues the previously-stepped thread, as
      otherwise we'd report the pending SIGTRAP then, and the core, not
      having stepped the thread, wouldn't understand what the trap was
      for, and therefore would report it to the user as a random
      signal.  */

I guess this answers my question.

Re: Multi-threading support on DragonFlyBSD

[Simon Marchi via Gdb]

On 3/5/25 5:22 AM, Sergey Zigachev via Gdb wrote:
> Hi,
> I'm trying to enable debugging of multi-threaded programs on DragonFlyBSD using gdb and have several questions.
> 
> I checked out gdb 15.2 sources, applied out-of-tree patches with native-dependent code implementation for amd64 DragonFlyBSD target. These patches have basic support for enumerating registers, accessing memory etc. All the OS-specific things. Single-thread debugging works well with these patches. However, multiprocess debugging, async and non-stop modes are not implemented. As is any notion of multiple threads in the inferior. My goal is to add missing bits and pieces both in userspace (gdb's nat-dep) and kernelspace (mostly ptrace syscall).
> 
> I'm at the stage where I have the basics working. I can report events from different threads, suspend/resume single or all threads, inspect TLS variables etc. Schedule-locking mode also supported.
> 
> My questions concern mostly step-over ("step") command issued to multiple threads, not at once of course. Although I see approx. the same behavior with the "next" command.
> I wrote a little test C program that has main thread spawning two extra threads running two separate but mostly identical functions. Each function prints message to output, sleeps for several seconds, prints another message, and then returns. Main function waits for both threads to complete.
> 
> First test run.
> I put a breakpoint at the start of the first function, run the program and input "step" command until process exits. Essentially I'm debbuging as-if it was a single-thread program. Everything works as expected. By that i mean after each "step" command gdb steps on the next source line until the end of the function. It skips instructions that are part of a single source line, and skips functions that don't have debugging symbols. This is the baseline I'm trying to replicate with stepping over multiple threads.
> 
> Second test run.
> Same as the first run but this time I put a breakpoint at the start of both functions. This results in gdb stopping at different threads when I enter "step" command.
> I observe the following issues:
>   - each source line gets hit several times, I think by the number of asm instructions it implements
>   - gdb stops at functions without debug symbols; sometimes it can step over and continue; sometimes it doesn't if it can't find the end of the function
> Basically, gdb's acting as if it's running "stepi" command.
> 
> With trial and error and some printf sprinkled around I came up with a theory why that could happen. Consider this part of the function:
> 
>> 23 printf("Running thread thr=%d sleep=%d!\n", d->thr_num, d->sleep);
>>   0x0000000000400b2d <+29>:    mov    0xc(%rdi),%edx
>>   0x0000000000400b30 <+32>:    mov    $0x400cff,%edi
>>   0x0000000000400b35 <+37>:    xor    %eax,%eax
>>   0x0000000000400b37 <+39>:    call   0x4007f0 <printf@plt>
>>
>> 24 sleep(d->sleep);
>>   0x0000000000400b3c <+44>:    mov    0xc(%rbx),%edi
> 
> When gdb stops at line 23 and I enter "step" command, it calculates step range thread_info->control.step_range_{start,end}. In this case it will be [0x400b2d-0x400b3c). Then it uses this info to step through asm instructions 4 times transparently to the user.
> The happy call chain looks like this (infrun.c):
>   - clear_proceed_status_thread *for each thread*
>   - prepare_one_step (infcmd.c) -- this one updates the range
>   - proceed
>   - 4x fetch_inferior_event
>   - stop at next line waiting for input
> This works for single-thread case because gdb clears thread_info state and then loads range for the *current* thread after clearing.
> If I return event for a different thread, then its info was cleared and haven't been loaded by gdb before that. Gdb can't recognize this signal and assumes it's a "random signal". It goes downhill from there, I think most consecutive signals become "random".
> 
> I don't want to track down heisenbugs caused by my dirty hands touching gdb thread state and accidentally braking invariants. Hence this message, any advice is appreciated.
> 
> My questions:
>   1. Does my theory make sense?
>   2. Is it a job of native-dependent code to save/restore thread state before reporting a signal?
>   3. If yes, what else should I store, per-thread, per-process etc?
> 
> Thanks,
> Sergey

Hi Sergey,

What I don't understand here is: what other event do you report for the
other threads?  When you step one thread, I assume that your resume
method gets called with inferior_ptid (the global variable) equal to the
thread being stepped, and scope_ptid equal to (pid, 0, 0)?  If so, the
threads other than the one being stepped should be resumed freely.  In
your scenario, I don't see what signal these other threads would
receive.

Once your stepping thread is done stepping, it should generate a
SIGTRAP, after which your target should send some signal (SIGSTOP?) to
ask the other threads to stop and wait for these threads to report
"stopped by SIGSTOP" (since your target operates in non-stop mode).  But
these SIGSTOPs should not be reported to the core, as it's an
implementation detail of the target.  That's my understanding of how
things work on Linux, perhaps things are different on DragonflyBSD.

I assume that you have enabled and started at the "set debug infrun"
logs?  This is how you generally debug those issues, look at what infrun
asks your target to do, look at what your target answers, try to see if
that makes sense.

Finally, it's perhaps outside of the scope of your work, but "modern"
targets use non-stop (related to "maint set target-non-stop", not the
user-visible "set non-stop") and target-async.  I'm more used to reason
in terms of non-stop & target-async, and that code path of infrun is
likely more tested.  It's probably no small task, but switching to that
would be more future-proof.

Simon

Re: Multi-threading support on DragonFlyBSD

[Sergey Zigachev via Gdb]

[-- Attachment #1: Type: text/plain, Size: 4616 bytes --]

> What I don't understand here is: what other event do you report for the
> other threads?  When you step one thread, I assume that your resume
> method gets called with inferior_ptid (the global variable) equal to the
> thread being stepped, and scope_ptid equal to (pid, 0, 0)?  If so, the
> threads other than the one being stepped should be resumed freely.  In
> your scenario, I don't see what signal these other threads would
> receive.
> 
> Once your stepping thread is done stepping, it should generate a
> SIGTRAP, after which your target should send some signal (SIGSTOP?) to
> ask the other threads to stop and wait for these threads to report
> "stopped by SIGSTOP" (since your target operates in non-stop mode).  But
> these SIGSTOPs should not be reported to the core, as it's an
> implementation detail of the target.  That's my understanding of how
> things work on Linux, perhaps things are different on DragonflyBSD.
> 

That's overall correct. Other threads can receive SIGTRAP if I put a 
breakpoint, for example. I thought that should work, isn't it? If GDB 
resumes all threads that means any number of threads can stop at a 
breakpoint or SIGTRAP caused by single-stepping a process.

Here's what I think is happening:
   1. put a breakpoint_1 at function_1 for thread_1
   2. put a breakpoint_2 at function_2 for thread_2
   3. -> "run" the process, main thread starts threads approx. at once
   4. breakpoint_1 hit, current thread is thread_1
   5. -> enter "step" command
   6. GDB resumes thread_1, then all threads, every time with step=1
   7. breakpoint_2 hit, GDB switches to thread_2
   8. -> enter "step" command
   9. GDB resumes thread_2, then all threads, every time with step=1
   10. thread_1 generates SIGTRAP
   11. GDB can't tell this SIGTRAP is the result of (5), so can't skip it
   12. GDB reports it as a random signal

If GDB receives events for the current thread at steps (6) and (9), then 
it processes them just fine, skipping as necessary.

WRT implementation details: I don't send SIGSTOP to stop all threads 
when SIGTRAP generated and I don't use waitpid. Natdep target 
communicates with the kernel through ptrace syscall only. Ptrace request 
waits for a event, stops all threads and waits for them to actually stop 
before returning to userspace. Then in the natdep code I pull all 
pending events from all threads, stash them in a list, and return one 
event according to what GDB expects to get: if it resumed just a single 
thread, then I return an event strictly for that thread (if any); if it 
resumed all threads, then I return the next event from the list, which 
may or may not be for the current thread. In my test program, I only get 
SIGTRAPs triggered by single-stepping threads, no other signals generated.
Please see attached debug log output. It's rather verbose though.

Couple notes:
   - for DragonFly, I'm using ptid_t format as (pid, thread_id, 0), 
thread_id is unique for a process, but is NOT globally unique
   - you can ignore lines with [dfly-nat], that's my implementation's 
debug info, it mostly duplicates what infrun writes anyway

I'm sorry, my explanations are hard to understand, hopefully it'll be 
clearer with the log.

Ideally, I would like to get GDB to transparently skip assembly 
instructions which are not relevant to report regardless which thread is 
reporting an event, be it currently selected or not. Like part of the 
already reported source line, dynsym code, missing debug symbols or 
anything else. For that to work I think I need to save and restore GDB's 
thread "context" (in the thread_info struct) before reporting an event. 
If it's not possible or doesn't make sense, then I'll to do what Linux 
is doing -- give currently selected thread higher priority and report it 
first.

> I assume that you have enabled and started at the "set debug infrun"
> logs?  This is how you generally debug those issues, look at what infrun
> asks your target to do, look at what your target answers, try to see if
> that makes sense.

Yes, I enabled infrun debug output and it was invaluable for gathering 
info.

> Finally, it's perhaps outside of the scope of your work, but "modern"
> targets use non-stop (related to "maint set target-non-stop", not the
> user-visible "set non-stop") and target-async.  I'm more used to reason
> in terms of non-stop & target-async, and that code path of infrun is
> likely more tested.  It's probably no small task, but switching to that
> would be more future-proof.

I'll take a look at non-stop and async modes.
Also, I wasn't aware of maintenance command, thanks!

-Sergey

[-- Attachment #2: gdb_output.txt --]
[-- Type: text/plain, Size: 46818 bytes --]

[infrun] scoped_disable_commit_resumed: reason=running
Starting program: /root/gdb-tests/tls_threads-clang
[dfly-nat] wait: [in] ptid = -1.0.0
[dfly-nat] wait: resumed_lwps -1.0.0
[dfly-nat] wait: PT_WAIT err 0 errno 22
[dfly-nat] wait: err 0 errno 22 event status 2 lwpid 1 signal 5
[dfly-nat] wait: store lwp=1 signal=5
[dfly-nat] wait: err 0 errno 22 event status 0 lwpid 1 signal 5
[dfly-nat] wait: no event, exiting
[dfly-nat] wait: return lwp=1 signal=5
[dfly-nat] resume: ptid=-1.0.0 step=0 signal=0
[dfly-nat] resume: ptid=7456.1.0 resuming (all)
[dfly-nat] resume: ptid=7456.1.0 unstopped
[dfly-nat] wait: [in] ptid = -1.0.0
[dfly-nat] wait: resumed_lwps -1.0.0
[dfly-nat] wait: PT_WAIT err 0 errno 0
[dfly-nat] wait: err 0 errno 0 event status 2 lwpid 1 signal 5
[dfly-nat] wait: store lwp=1 signal=5
[dfly-nat] wait: err 0 errno 0 event status 0 lwpid 1 signal 5
[dfly-nat] wait: no event, exiting
[dfly-nat] wait: return lwp=1 signal=5
[infrun] infrun_debug_show_threads: enter
  [infrun] infrun_debug_show_threads: immediately after create_process:
  [infrun] infrun_debug_show_threads:   thread 7456.1.0, executing = 0, resumed = 0, state = STOPPED
[infrun] infrun_debug_show_threads: exit
[infrun] infrun_debug_show_threads: enter
  [infrun] infrun_debug_show_threads: threads in the newly created inferior:
  [infrun] infrun_debug_show_threads:   thread 7456.1.0, executing = 0, resumed = 0, state = STOPPED
[infrun] infrun_debug_show_threads: exit
[infrun] proceed: enter
  [infrun] follow_fork: enter
  [infrun] follow_fork: exit
  [infrun] proceed: cur_thr = 7456.1.0
  [infrun] proceed: addr=0x800403820, signal=GDB_SIGNAL_0, resume_ptid=-1.0.0
  [infrun] scoped_disable_commit_resumed: reason=proceeding
  [infrun] start_step_over: enter
    [infrun] start_step_over: stealing global queue of threads to step, length = 0
    [infrun] operator(): step-over queue now empty
  [infrun] start_step_over: exit
  [infrun] proceed_resume_thread_checked: resuming 7456.1.0
  [infrun] resume_1: step=0, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.1.0] at 0x800403820
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=0, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=0 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.1.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=proceeding
[infrun] proceed: exit
[infrun] reset: reason=running
[infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
[infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 1 signal 5
  [dfly-nat] wait: store lwp=1 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 1 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=1 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.1.0 [LWP 1 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.1.0
  [infrun] handle_signal_stop: stop_pc=0x8004042fb
  [infrun] process_event_stop_test: BPSTAT_WHAT_SINGLE
  [infrun] switch_back_to_stepped_thread: thread [7456.1.0] still needs step-over
  [infrun] should_be_inserted: skipping breakpoint: stepping past insn at: 0x8004042fb
  [infrun] should_be_inserted: skipping breakpoint: stepping past insn at: 0x8004042fb
  [infrun] should_be_inserted: skipping breakpoint: stepping past insn at: 0x8004042fb
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=1, current thread [7456.1.0] at 0x8004042fb
  [infrun] do_target_resume: resume_ptid=7456.1.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=7456.1.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (single)
  [dfly-nat] resume: ptid=7456.1.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps 7456.1.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 1 signal 5
  [dfly-nat] wait: store lwp=1 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 1 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=1 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.1.0 [LWP 1 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] clear_step_over_info: clearing step over info
  [infrun] context_switch: Switching context from 0.0.0 to 7456.1.0
  [infrun] handle_signal_stop: stop_pc=0x80040b105
  [infrun] start_step_over: enter
    [infrun] start_step_over: stealing global queue of threads to step, length = 0
    [infrun] operator(): step-over queue now empty
  [infrun] start_step_over: exit
  [infrun] process_event_stop_test: no stepping, continue
  [infrun] resume_1: step=0, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.1.0] at 0x80040b105
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=0, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=0 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.1.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 4 lwpid 2 signal 0
  [dfly-nat] wait: add new thread 2
[New LWP 2 of process 7456]
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 2 signal 0
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: reached EOF!
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.2.0 [LWP 2 of process 7456],
  [infrun] print_target_wait_results:   status->kind = SPURIOUS
  [infrun] handle_inferior_event: status->kind = SPURIOUS
  [infrun] context_switch: Switching context from 0.0.0 to 7456.2.0
  [infrun] resume_1: step=0, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.2.0] at 0x8006de96e
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=0, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=0 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 4 lwpid 3 signal 0
  [dfly-nat] wait: add new thread 3
[New LWP 3 of process 7456]
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 3 signal 0
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: reached EOF!
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.3.0 [LWP 3 of process 7456],
  [infrun] print_target_wait_results:   status->kind = SPURIOUS
  [infrun] handle_inferior_event: status->kind = SPURIOUS
  [infrun] context_switch: Switching context from 0.0.0 to 7456.3.0
  [infrun] resume_1: step=0, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.3.0] at 0x8006de96e
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=0, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=0 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 2 signal 5
  [dfly-nat] wait: store lwp=2 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 2 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=2 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.2.0 [LWP 2 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.2.0
  [infrun] handle_signal_stop: stop_pc=0x400b1a
  [infrun] bpstat_check_breakpoint_conditions: enter
    [infrun] bpstat_check_breakpoint_conditions: thread = 7456.2.0, breakpoint 1.1
    [infrun] bpstat_check_breakpoint_conditions: stopping at this breakpoint
  [infrun] bpstat_check_breakpoint_conditions: exit
  [infrun] process_event_stop_test: BPSTAT_WHAT_STOP_NOISY
  [infrun] stop_waiting: stop_waiting
[Switching to LWP 2 of process 7456]

Thread 2 hit Breakpoint 1, thread_runner_1 (arg=0x7fffffdfd6e0) at ./tls-threads.c:22
22              counter = data->thr_num;
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: not requesting commit-resumed for target native, no resumed threads
[infrun] fetch_inferior_event: exit



=== ENTER "step" command

[infrun] clear_proceed_status_thread: 7456.1.0
[infrun] clear_proceed_status_thread: 7456.2.0
[infrun] clear_proceed_status_thread: 7456.3.0
[infrun] set_step_info: symtab = ./tls-threads.c, line = 22, step_frame_id = {stack=0x7fdfe0000fd0,code=0x0000000000400b10,!special}, step_stack_frame_id = {stack=0x7fdfe0000fd0,code=0x0000000000400b10,!special}
[infrun] proceed: enter
  [infrun] follow_fork: enter
  [infrun] follow_fork: exit
  [infrun] proceed: cur_thr = 7456.2.0
  [infrun] proceed: addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT, resume_ptid=-1.0.0
  [infrun] global_thread_step_over_chain_enqueue: enqueueing thread 7456.2.0 in global step over chain
  [infrun] scoped_disable_commit_resumed: reason=proceeding
  [infrun] start_step_over: enter
    [infrun] start_step_over: stealing global queue of threads to step, length = 1
    [infrun] start_step_over: resuming [7456.2.0] for step-over
    [infrun] should_be_inserted: skipping breakpoint: stepping past insn at: 0x400b1a
    [infrun] should_be_inserted: skipping breakpoint: stepping past insn at: 0x400b1a
    [infrun] should_be_inserted: skipping breakpoint: stepping past insn at: 0x400b1a
    [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=1, current thread [7456.2.0] at 0x400b1a
    [infrun] do_target_resume: resume_ptid=7456.2.0, step=1, sig=GDB_SIGNAL_0
    [dfly-nat] resume: ptid=7456.2.0 step=1 signal=0
    [dfly-nat] resume: ptid=7456.1.0 suspending
    [dfly-nat] resume: ptid=7456.2.0 resuming (single)
    [dfly-nat] resume: ptid=7456.3.0 suspending
    [dfly-nat] resume: ptid=7456.2.0 unstopped
    [infrun] prepare_to_wait: prepare_to_wait
    [infrun] start_step_over: [7456.2.0] was resumed.
    [infrun] operator(): step-over queue now empty
  [infrun] start_step_over: exit
  [infrun] reset: reason=proceeding
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] proceed: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps 7456.2.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 2 signal 5
  [dfly-nat] wait: store lwp=2 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 2 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=2 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.2.0 [LWP 2 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] clear_step_over_info: clearing step over info
  [infrun] context_switch: Switching context from 0.0.0 to 7456.2.0
  [infrun] handle_signal_stop: stop_pc=0x400b1d
  [infrun] process_event_stop_test: stepping inside range [0x400b1a-0x400b2d]
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.2.0] at 0x400b1d
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 2 signal 5
  [dfly-nat] wait: store lwp=2 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 2 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=2 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.2.0 [LWP 2 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.2.0
  [infrun] handle_signal_stop: stop_pc=0x400b26
  [infrun] process_event_stop_test: stepping inside range [0x400b1a-0x400b2d]
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.2.0] at 0x400b26
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 3 signal 5
  [dfly-nat] wait: store lwp=3 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 3 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=3 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.3.0 [LWP 3 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.3.0
  [infrun] handle_signal_stop: stop_pc=0x400b7a
  [infrun] bpstat_check_breakpoint_conditions: enter
    [infrun] bpstat_check_breakpoint_conditions: thread = 7456.3.0, breakpoint 2.1
    [infrun] bpstat_check_breakpoint_conditions: stopping at this breakpoint
  [infrun] bpstat_check_breakpoint_conditions: exit
  [infrun] process_event_stop_test: BPSTAT_WHAT_STOP_NOISY
  [infrun] stop_waiting: stop_waiting
[Switching to LWP 3 of process 7456]

Thread 3 hit Breakpoint 2, thread_runner_2 (arg=0x7fffffdfd6f0) at ./tls-threads.c:37
37              counter = data->thr_num;
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: not requesting commit-resumed for target native, no resumed threads
[infrun] fetch_inferior_event: exit



=== ENTER "step" command

[infrun] clear_proceed_status_thread: 7456.1.0
[infrun] clear_proceed_status_thread: 7456.2.0
[infrun] clear_proceed_status_thread: 7456.3.0
[infrun] set_step_info: symtab = ./tls-threads.c, line = 37, step_frame_id = {stack=0x7fdfe0201fd0,code=0x0000000000400b70,!special}, step_stack_frame_id = {stack=0x7fdfe0201fd0,code=0x0000000000400b70,!special}
[infrun] proceed: enter
  [infrun] follow_fork: enter
  [infrun] follow_fork: exit
  [infrun] proceed: cur_thr = 7456.3.0
  [infrun] proceed: addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT, resume_ptid=-1.0.0
  [infrun] global_thread_step_over_chain_enqueue: enqueueing thread 7456.3.0 in global step over chain
  [infrun] scoped_disable_commit_resumed: reason=proceeding
  [infrun] start_step_over: enter
    [infrun] start_step_over: stealing global queue of threads to step, length = 1
    [infrun] start_step_over: resuming [7456.3.0] for step-over
    [infrun] should_be_inserted: skipping breakpoint: stepping past insn at: 0x400b7a
    [infrun] should_be_inserted: skipping breakpoint: stepping past insn at: 0x400b7a
    [infrun] should_be_inserted: skipping breakpoint: stepping past insn at: 0x400b7a
    [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=1, current thread [7456.3.0] at 0x400b7a
    [infrun] do_target_resume: resume_ptid=7456.3.0, step=1, sig=GDB_SIGNAL_0
    [dfly-nat] resume: ptid=7456.3.0 step=1 signal=0
    [dfly-nat] resume: ptid=7456.1.0 suspending
    [dfly-nat] resume: ptid=7456.2.0 suspending
    [dfly-nat] resume: ptid=7456.3.0 resuming (single)
    [dfly-nat] resume: ptid=7456.3.0 unstopped
    [infrun] prepare_to_wait: prepare_to_wait
    [infrun] start_step_over: [7456.3.0] was resumed.
    [infrun] operator(): step-over queue now empty
  [infrun] start_step_over: exit
  [infrun] reset: reason=proceeding
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] proceed: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps 7456.3.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 3 signal 5
  [dfly-nat] wait: store lwp=3 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 3 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=3 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.3.0 [LWP 3 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] clear_step_over_info: clearing step over info
  [infrun] context_switch: Switching context from 0.0.0 to 7456.3.0
  [infrun] handle_signal_stop: stop_pc=0x400b7d
  [infrun] process_event_stop_test: stepping inside range [0x400b7a-0x400b8d]
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.3.0] at 0x400b7d
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 3 signal 5
  [dfly-nat] wait: store lwp=3 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 3 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=3 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.3.0 [LWP 3 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.3.0
  [infrun] handle_signal_stop: stop_pc=0x400b86
  [infrun] process_event_stop_test: stepping inside range [0x400b7a-0x400b8d]
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.3.0] at 0x400b86
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 2 signal 5
  [dfly-nat] wait: store lwp=2 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 2 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=2 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.2.0 [LWP 2 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.2.0
  [infrun] handle_signal_stop: stop_pc=0x400b2d
  [infrun] handle_signal_stop: random signal (GDB_SIGNAL_TRAP)
  [infrun] stop_waiting: stop_waiting
  [infrun] print_signal_received_reason: signal = Trace/breakpoint trap

Thread 2 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to LWP 2 of process 7456]
thread_runner_1 (arg=0x7fffffdfd6e0) at ./tls-threads.c:23
23              printf("Running 1 thread thr=%d sleep=%d!\n", data->thr_num, data->sleep_ms);
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: not requesting commit-resumed for target native, no resumed threads
[infrun] fetch_inferior_event: exit



=== ENTER "step" command

[infrun] clear_proceed_status_thread: 7456.1.0
[infrun] clear_proceed_status_thread: 7456.2.0
[infrun] clear_proceed_status_thread: 7456.3.0
[infrun] set_step_info: symtab = ./tls-threads.c, line = 23, step_frame_id = {stack=0x7fdfe0000fd0,code=0x0000000000400b10,!special}, step_stack_frame_id = {stack=0x7fdfe0000fd0,code=0x0000000000400b10,!special}
[infrun] proceed: enter
  [infrun] follow_fork: enter
  [infrun] follow_fork: exit
  [infrun] proceed: cur_thr = 7456.2.0
  [infrun] proceed: addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT, resume_ptid=-1.0.0
  [infrun] scoped_disable_commit_resumed: reason=proceeding
  [infrun] start_step_over: enter
    [infrun] start_step_over: stealing global queue of threads to step, length = 0
    [infrun] operator(): step-over queue now empty
  [infrun] start_step_over: exit
  [infrun] proceed_resume_thread_checked: resuming 7456.2.0
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.2.0] at 0x400b2d
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=proceeding
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] proceed: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 3 signal 5
  [dfly-nat] wait: store lwp=3 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 3 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=3 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.3.0 [LWP 3 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.3.0
  [infrun] handle_signal_stop: stop_pc=0x400b8d
  [infrun] handle_signal_stop: random signal (GDB_SIGNAL_TRAP)
  [infrun] stop_waiting: stop_waiting
  [infrun] print_signal_received_reason: signal = Trace/breakpoint trap

Thread 3 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to LWP 3 of process 7456]
thread_runner_2 (arg=0x7fffffdfd6f0) at ./tls-threads.c:38
38              printf("Running extra thread thr=%d sleep=%d!\n", data->thr_num, data->sleep_ms);
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: not requesting commit-resumed for target native, no resumed threads
[infrun] fetch_inferior_event: exit



=== ENTER "step" command

[infrun] clear_proceed_status_thread: 7456.1.0
[infrun] clear_proceed_status_thread: 7456.2.0
[infrun] clear_proceed_status_thread: 7456.3.0
[infrun] set_step_info: symtab = ./tls-threads.c, line = 38, step_frame_id = {stack=0x7fdfe0201fd0,code=0x0000000000400b70,!special}, step_stack_frame_id = {stack=0x7fdfe0201fd0,code=0x0000000000400b70,!special}
[infrun] proceed: enter
  [infrun] follow_fork: enter
  [infrun] follow_fork: exit
  [infrun] proceed: cur_thr = 7456.3.0
  [infrun] proceed: addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT, resume_ptid=-1.0.0
  [infrun] scoped_disable_commit_resumed: reason=proceeding
  [infrun] start_step_over: enter
    [infrun] start_step_over: stealing global queue of threads to step, length = 0
    [infrun] operator(): step-over queue now empty
  [infrun] start_step_over: exit
  [infrun] proceed_resume_thread_checked: resuming 7456.3.0
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.3.0] at 0x400b8d
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=proceeding
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] proceed: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 2 signal 5
  [dfly-nat] wait: store lwp=2 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 2 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=2 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.2.0 [LWP 2 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.2.0
  [infrun] handle_signal_stop: stop_pc=0x400b30
  [infrun] handle_signal_stop: random signal (GDB_SIGNAL_TRAP)
  [infrun] stop_waiting: stop_waiting
  [infrun] print_signal_received_reason: signal = Trace/breakpoint trap

Thread 2 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to LWP 2 of process 7456]
0x0000000000400b30 in thread_runner_1 (arg=0x7fffffdfd6e0) at ./tls-threads.c:23
23              printf("Running 1 thread thr=%d sleep=%d!\n", data->thr_num, data->sleep_ms);
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: not requesting commit-resumed for target native, no resumed threads
[infrun] fetch_inferior_event: exit



=== ENTER "step" command

[infrun] clear_proceed_status_thread: 7456.1.0
[infrun] clear_proceed_status_thread: 7456.2.0
[infrun] clear_proceed_status_thread: 7456.3.0
[infrun] set_step_info: symtab = ./tls-threads.c, line = 23, step_frame_id = {stack=0x7fdfe0000fd0,code=0x0000000000400b10,!special}, step_stack_frame_id = {stack=0x7fdfe0000fd0,code=0x0000000000400b10,!special}
[infrun] proceed: enter
  [infrun] follow_fork: enter
  [infrun] follow_fork: exit
  [infrun] proceed: cur_thr = 7456.2.0
  [infrun] proceed: addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT, resume_ptid=-1.0.0
  [infrun] scoped_disable_commit_resumed: reason=proceeding
  [infrun] start_step_over: enter
    [infrun] start_step_over: stealing global queue of threads to step, length = 0
    [infrun] operator(): step-over queue now empty
  [infrun] start_step_over: exit
  [infrun] proceed_resume_thread_checked: resuming 7456.2.0
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.2.0] at 0x400b30
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=proceeding
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] proceed: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 2 signal 5
  [dfly-nat] wait: store lwp=2 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 2 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=2 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.2.0 [LWP 2 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.2.0
  [infrun] handle_signal_stop: stop_pc=0x400b35
  [infrun] process_event_stop_test: stepping inside range [0x400b30-0x400b3c]
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.2.0] at 0x400b35
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 3 signal 5
  [dfly-nat] wait: store lwp=3 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 3 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=3 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.3.0 [LWP 3 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.3.0
  [infrun] handle_signal_stop: stop_pc=0x400b90
  [infrun] handle_signal_stop: random signal (GDB_SIGNAL_TRAP)
  [infrun] stop_waiting: stop_waiting
  [infrun] print_signal_received_reason: signal = Trace/breakpoint trap

Thread 3 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to LWP 3 of process 7456]
0x0000000000400b90 in thread_runner_2 (arg=0x7fffffdfd6f0) at ./tls-threads.c:38
38              printf("Running extra thread thr=%d sleep=%d!\n", data->thr_num, data->sleep_ms);
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: not requesting commit-resumed for target native, no resumed threads
[infrun] fetch_inferior_event: exit



=== ENTER "step" command

[infrun] clear_proceed_status_thread: 7456.1.0
[infrun] clear_proceed_status_thread: 7456.2.0
[infrun] clear_proceed_status_thread: 7456.3.0
[infrun] set_step_info: symtab = ./tls-threads.c, line = 38, step_frame_id = {stack=0x7fdfe0201fd0,code=0x0000000000400b70,!special}, step_stack_frame_id = {stack=0x7fdfe0201fd0,code=0x0000000000400b70,!special}
[infrun] proceed: enter
  [infrun] follow_fork: enter
  [infrun] follow_fork: exit
  [infrun] proceed: cur_thr = 7456.3.0
  [infrun] proceed: addr=0xffffffffffffffff, signal=GDB_SIGNAL_DEFAULT, resume_ptid=-1.0.0
  [infrun] scoped_disable_commit_resumed: reason=proceeding
  [infrun] start_step_over: enter
    [infrun] start_step_over: stealing global queue of threads to step, length = 0
    [infrun] operator(): step-over queue now empty
  [infrun] start_step_over: exit
  [infrun] proceed_resume_thread_checked: resuming 7456.3.0
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.3.0] at 0x400b90
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=proceeding
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] proceed: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 3 signal 5
  [dfly-nat] wait: store lwp=3 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 3 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=3 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.3.0 [LWP 3 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.3.0
  [infrun] handle_signal_stop: stop_pc=0x400b95
  [infrun] process_event_stop_test: stepping inside range [0x400b90-0x400b9c]
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.3.0] at 0x400b95
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 3 signal 5
  [dfly-nat] wait: store lwp=3 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 3 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=3 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.3.0 [LWP 3 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.3.0
  [infrun] handle_signal_stop: stop_pc=0x400b97
  [infrun] process_event_stop_test: stepping inside range [0x400b90-0x400b9c]
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.3.0] at 0x400b97
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 3 signal 5
  [dfly-nat] wait: store lwp=3 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 3 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=3 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.3.0 [LWP 3 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.3.0
  [infrun] handle_signal_stop: stop_pc=0x4007f0
  [infrun] process_event_stop_test: stepped into dynsym resolve code
  [infrun] resume_1: step=1, signal=GDB_SIGNAL_0, trap_expected=0, current thread [7456.3.0] at 0x4007f0
  [infrun] do_target_resume: resume_ptid=-1.0.0, step=1, sig=GDB_SIGNAL_0
  [dfly-nat] resume: ptid=-1.0.0 step=1 signal=0
  [dfly-nat] resume: ptid=7456.1.0 resuming (all)
  [dfly-nat] resume: ptid=7456.2.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 resuming (all)
  [dfly-nat] resume: ptid=7456.3.0 unstopped
  [infrun] prepare_to_wait: prepare_to_wait
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: enabling commit-resumed for target native
  [infrun] maybe_call_commit_resumed_all_targets: calling commit_resumed for target native
[infrun] fetch_inferior_event: exit
[infrun] fetch_inferior_event: enter
  [infrun] scoped_disable_commit_resumed: reason=handling event
  [infrun] random_pending_event_thread: None found.
  [dfly-nat] wait: [in] ptid = -1.0.0
  [dfly-nat] wait: resumed_lwps -1.0.0
  [dfly-nat] wait: PT_WAIT err 0 errno 35
  [dfly-nat] wait: err 0 errno 35 event status 2 lwpid 2 signal 5
  [dfly-nat] wait: store lwp=2 signal=5
  [dfly-nat] wait: err 0 errno 35 event status 0 lwpid 2 signal 5
  [dfly-nat] wait: no event, exiting
  [dfly-nat] wait: return lwp=2 signal=5
  [infrun] print_target_wait_results: target_wait (-1.0.0 [LWP 1 of process -1], status) =
  [infrun] print_target_wait_results:   7456.2.0 [LWP 2 of process 7456],
  [infrun] print_target_wait_results:   status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] handle_inferior_event: status->kind = STOPPED, sig = GDB_SIGNAL_TRAP
  [infrun] context_switch: Switching context from 0.0.0 to 7456.2.0
  [infrun] handle_signal_stop: stop_pc=0x400b37
  [infrun] handle_signal_stop: random signal (GDB_SIGNAL_TRAP)
  [infrun] stop_waiting: stop_waiting
  [infrun] print_signal_received_reason: signal = Trace/breakpoint trap

Thread 2 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to LWP 2 of process 7456]
0x0000000000400b37 in thread_runner_1 (arg=0x7fffffdfd6e0) at ./tls-threads.c:23
23              printf("Running 1 thread thr=%d sleep=%d!\n", data->thr_num, data->sleep_ms);
  [infrun] reset: reason=handling event
  [infrun] maybe_set_commit_resumed_all_targets: not requesting commit-resumed for target native, no resumed threads
[infrun] fetch_inferior_event: exit

Re: Multi-threading support on DragonFlyBSD

[Simon Marchi via Gdb]

On 3/11/25 3:59 AM, Sergey Zigachev wrote:
> That's overall correct. Other threads can receive SIGTRAP if I put a
> breakpoint, for example. I thought that should work, isn't it? If GDB
> resumes all threads that means any number of threads can stop at a
> breakpoint or SIGTRAP caused by single-stepping a process.

If another thread (that is being ran freely) happens to hit a breakpoint
while single stepping the selected thread, then indeed you should report
that SIGTRAP.  GDB will present that stop to the user, and the step will
be aborted.  The stepping thread will be asked to stop, and will be
stopped at an arbitrary place.

> Here's what I think is happening:
>   1. put a breakpoint_1 at function_1 for thread_1
>   2. put a breakpoint_2 at function_2 for thread_2
>   3. -> "run" the process, main thread starts threads approx. at once
>   4. breakpoint_1 hit, current thread is thread_1
>   5. -> enter "step" command
>   6. GDB resumes thread_1, then all threads, every time with step=1
>   7. breakpoint_2 hit, GDB switches to thread_2
>   8. -> enter "step" command
>   9. GDB resumes thread_2, then all threads, every time with step=1
>   10. thread_1 generates SIGTRAP
>   11. GDB can't tell this SIGTRAP is the result of (5), so can't skip it
>   12. GDB reports it as a random signal

I think that the SIGTRAP at #10 is spurious.  It is perhaps a leftover
from the step command at #5/#6?

When thread 2 hits breakpoint_2 at #7, then the step from thread 1
aborts and should be forgotten.  When you step again at #8/#9, you start
from a blank slate.

> If GDB receives events for the current thread at steps (6) and (9),
> then it processes them just fine, skipping as necessary.

It's possible that the code path taken in that case clears something
that causes the spurious SIGTRAP to be forgotten.

> WRT implementation details: I don't send SIGSTOP to stop all threads
> when SIGTRAP generated and I don't use waitpid. Natdep target
> communicates with the kernel through ptrace syscall only. Ptrace
> request waits for a event, stops all threads and waits for them to
> actually stop before returning to userspace. Then in the natdep code I
> pull all pending events from all threads, stash them in a list, and
> return one event according to what GDB expects to get: if it resumed
> just a single thread, then I return an event strictly for that thread
> (if any); if it resumed all threads, then I return the next event from
> the list, which may or may not be for the current thread. In my test
> program, I only get SIGTRAPs triggered by single-stepping threads, no
> other signals generated.

Ok, so if I understand correctly, you do some kind of "ptrace resume",
which is blocking, and it unblocks when something happens.  When that
returns, the kernel has stopped all threads for you.  In a sense that
simplifies some things, but it would make async/non-stop harder to
implement.

> Please see attached debug log output. It's rather verbose though.

I'll try to go through it later.

> Couple notes:
>   - for DragonFly, I'm using ptid_t format as (pid, thread_id, 0),
>   thread_id is unique for a process, but is NOT globally unique

I think that's fine.  The core of GDB never assumes that ptid_t::lwp is
globally unique.

>   - you can ignore lines with [dfly-nat], that's my implementation's
>   debug info, it mostly duplicates what infrun writes anyway

That's fine, it's very useful to add such logging.

> I'm sorry, my explanations are hard to understand, hopefully it'll be
> clearer with the log.
> 
> Ideally, I would like to get GDB to transparently skip assembly
> instructions which are not relevant to report regardless which thread
> is reporting an event, be it currently selected or not. Like part of
> the already reported source line, dynsym code, missing debug symbols
> or anything else. For that to work I think I need to save and restore
> GDB's thread "context" (in the thread_info struct) before reporting an
> event. If it's not possible or doesn't make sense, then I'll to do
> what Linux is doing -- give currently selected thread higher priority
> and report it first.

This kind of logic is managed at a higher level (infrun), not by
targets.  I don't think that your target_ops::wait method is the place
for such decision.  I'm not super familiar with this, I just know it's
managed by spaghetti-like functions like `handle_signal_stop`.

>> I assume that you have enabled and started at the "set debug infrun"
>> logs?  This is how you generally debug those issues, look at what infrun
>> asks your target to do, look at what your target answers, try to see if
>> that makes sense.
> 
> Yes, I enabled infrun debug output and it was invaluable for gathering info.
> 
>> Finally, it's perhaps outside of the scope of your work, but "modern"
>> targets use non-stop (related to "maint set target-non-stop", not the
>> user-visible "set non-stop") and target-async.  I'm more used to reason
>> in terms of non-stop & target-async, and that code path of infrun is
>> likely more tested.  It's probably no small task, but switching to that
>> would be more future-proof.
> 
> I'll take a look at non-stop and async modes.
> Also, I wasn't aware of maintenance command, thanks!

Given what you said above, about how your debug interface works,
non-stop/async is probably not a priority.

Simon

Re: Multi-threading support on DragonFlyBSD

[Sergey Zigachev via Gdb]

> If another thread (that is being ran freely) happens to hit a breakpoint
> while single stepping the selected thread, then indeed you should report
> that SIGTRAP.  GDB will present that stop to the user, and the step will
> be aborted.  The stepping thread will be asked to stop, and will be
> stopped at an arbitrary place.

Ok, that means I got the "abort the step" part wrong.

> I think that the SIGTRAP at #10 is spurious.  It is perhaps a leftover
> from the step command at #5/#6?

Yes, it is, I didn't think this was a leftover though. 🙂

> When thread 2 hits breakpoint_2 at #7, then the step from thread 1
> aborts and should be forgotten.  When you step again at #8/#9, you start
> from a blank slate.

Got that, too. Currently, I made sure to report to GDB result of every 
event it requested to get. Apparently, that's not how it should work.

> It's possible that the code path taken in that case clears something
> that causes the spurious SIGTRAP to be forgotten.

I think there's no spurious SIGTRAPs for other threads, for GDB it looks 
like the inferior runs only single thread at this point in time.
I can be wrong on that one.

> Ok, so if I understand correctly, you do some kind of "ptrace resume",
> which is blocking, and it unblocks when something happens.  When that
> returns, the kernel has stopped all threads for you.  In a sense that
> simplifies some things, but it would make async/non-stop harder to
> implement.

Yes, it's a "ptrace wait" request which is blocking, I have "ptrace 
resume", too, which doesn't block. I'm ok to change both user and kernel 
space if that would make GDB usage better for end user.

> This kind of logic is managed at a higher level (infrun), not by
> targets.  I don't think that your target_ops::wait method is the place
> for such decision.  I'm not super familiar with this, I just know it's
> managed by spaghetti-like functions like `handle_signal_stop`.

OK, it makes sense. Thanks for your comments, it's rather eye-opening 
experience. My mental model about how gdb core operates is really wrong.

> Given what you said above, about how your debug interface works,
> non-stop/async is probably not a priority.

Well, I implemented the kernel part myself. So if it doesn't fit GDB 
(and later LLDB), then I can redo that.
Any information about how one should implement target dependent code for 
GDB is appreciated. Wiki pages, articles, existing "canonical" targets 
to learn from (maybe less complex than Linux one?), etc.

-Sergey