Error record and replay target doesn't support syscall number 435

Error record and replay target doesn't support syscall number 435

[Navin P via Gdb]

Hi,
     I tried to record a program calling C++ threads and then did a
break on main , then record and continue it throws the error message
"record and replay target doesn't support syscall number 435" . What
should I do to support syscall number 435 ?

I was thinking of adding amd64_sys_clone3 = 435 to amd64-linux-tdep.h
In amd64-linux-tdep.c add  case amd64_sys_clone3: return gdb_sys_clone3;
In linux-record.h gdb_sys_clone3=541 in enum gdb_syscall
In linux-record.c case gdb_sys_clone3=break;


Is it necessary to do the record_mem_at_reg for clone3 ? Why isn't it
done for clone ?

Below is gdb log and after that is source code

gdb ./a.out
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./a.out...
(gdb) b main
Breakpoint 1 at 0x134b: file race1.cpp, line 30.
(gdb) r
Starting program: /home/navin/cpp/a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main () at race1.cpp:30
30 int main(){
(gdb) record full
(gdb) c
Continuing.
Process record and replay target doesn't support syscall number 435
Process record: failed to record execution log.

Program stopped.
clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:60
60 ../sysdeps/unix/sysv/linux/x86_64/clone3.S: No such file or directory.
(gdb) shell uname -a
Linux Navin-acer-5740 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14
13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
(gdb) shell cat /etc/lsb-release
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=21.2
DISTRIB_CODENAME=victoria
DISTRIB_DESCRIPTION="Linux Mint 21.2 Victoria"
(gdb)


Source Code

#include<bits/stdc++.h>
#include<thread>
using namespace std;

int g=0;
int *p=&g;
bool run=true;

void t1()
{
while(run)
{
g++;
if(g==16)
   p=nullptr;

}
}

void t2()
{
int z;
while(run)
{
g--;
z=*p+1;
}
}

int main(){
thread x1{t1};
thread x2{t2};
sleep(5);
run=false;
x1.join();
x2.join();
cout <<g<<endl;
}



Regards,
Navin

Re: Error record and replay target doesn't support syscall number 435

[Guinevere Larsen via Gdb]

On 5/10/24 05:07, Navin P via Gdb wrote:
> Hi,
>       I tried to record a program calling C++ threads and then did a
> break on main , then record and continue it throws the error message
> "record and replay target doesn't support syscall number 435" . What
> should I do to support syscall number 435 ?

Hi!

I can't answer everything, since I'm not too familiar with syscalls 
specifically, but I'll do my best!

My first road bump with this is that we record that a clone has 
happened, but we can only record the process that GDB is following 
(parent or child, depending on follow-fork-mode). This is 
counter-intuitive to me, since I would have expected the full inferior 
to be replayed. It can be enough for some users, and I think changing 
this behavior would be extremely difficult - if at all possible, but if 
we're updating this area, I'd like to have this limitations mentioned 
either in the manual (section 7 - Recording Inferior's Execution and 
Replaying It) or having an extra warning when GDB detects that a fork 
has happened and we're recording, or both :)

>
> I was thinking of adding amd64_sys_clone3 = 435 to amd64-linux-tdep.h
> In amd64-linux-tdep.c add  case amd64_sys_clone3: return gdb_sys_clone3;
> In linux-record.h gdb_sys_clone3=541 in enum gdb_syscall
these 3 changes make sense to me
> In linux-record.c case gdb_sys_clone3=break;
>
>
> Is it necessary to do the record_mem_at_reg for clone3 ? Why isn't it
> done for clone ?

This is where my lack of syscall (and possibly clone) knowledge comes in.

What did you expect to record with this call? I am guessing the stack 
that you passed to the child, and if that is the case, I don't think we 
should. Imagine that you have stayed on the parent process, and started 
stepping back while the child process is still executing. You would 
fully rewrite the memory of the child, making it read the wrong values, 
and if you move forward again, the new writes that the child did will be 
undone as we reset the stack.

If it is something else, do let me know! (and if you have technical 
descriptions of what's going on that don't include me reading kernel 
source code, I'd be tankful :3 )

-- 
Cheers,
Guinevere Larsen
She/Her/Hers

>
> Below is gdb log and after that is source code
>
> gdb ./a.out
> GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
> Copyright (C) 2022 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <https://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
>      <http://www.gnu.org/software/gdb/documentation/>.
>
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ./a.out...
> (gdb) b main
> Breakpoint 1 at 0x134b: file race1.cpp, line 30.
> (gdb) r
> Starting program: /home/navin/cpp/a.out
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>
> Breakpoint 1, main () at race1.cpp:30
> 30 int main(){
> (gdb) record full
> (gdb) c
> Continuing.
> Process record and replay target doesn't support syscall number 435
> Process record: failed to record execution log.
>
> Program stopped.
> clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:60
> 60 ../sysdeps/unix/sysv/linux/x86_64/clone3.S: No such file or directory.
> (gdb) shell uname -a
> Linux Navin-acer-5740 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14
> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
> (gdb) shell cat /etc/lsb-release
> DISTRIB_ID=LinuxMint
> DISTRIB_RELEASE=21.2
> DISTRIB_CODENAME=victoria
> DISTRIB_DESCRIPTION="Linux Mint 21.2 Victoria"
> (gdb)
>
>
> Source Code
>
> #include<bits/stdc++.h>
> #include<thread>
> using namespace std;
>
> int g=0;
> int *p=&g;
> bool run=true;
>
> void t1()
> {
> while(run)
> {
> g++;
> if(g==16)
>     p=nullptr;
>
> }
> }
>
> void t2()
> {
> int z;
> while(run)
> {
> g--;
> z=*p+1;
> }
> }
>
> int main(){
> thread x1{t1};
> thread x2{t2};
> sleep(5);
> run=false;
> x1.join();
> x2.join();
> cout <<g<<endl;
> }
>
>
>
> Regards,
> Navin
>

Re: Error record and replay target doesn't support syscall number 435

[Navin P via Gdb]

Hi,

On Fri, May 10, 2024 at 7:29 PM Guinevere Larsen <blarsen@redhat.com> wrote:
>
> On 5/10/24 05:07, Navin P via Gdb wrote:
> > Hi,
> >       I tried to record a program calling C++ threads and then did a
> > break on main , then record and continue it throws the error message
> > "record and replay target doesn't support syscall number 435" . What
> > should I do to support syscall number 435 ?
>
> Hi!
>
> I can't answer everything, since I'm not too familiar with syscalls
> specifically, but I'll do my best!
>
> My first road bump with this is that we record that a clone has
> happened, but we can only record the process that GDB is following
> (parent or child, depending on follow-fork-mode). This is
> counter-intuitive to me, since I would have expected the full inferior
> to be replayed.
To get this working you need minimum changes of current_inferior to
for each inferior in all_inferiors() and execute that function for
each inferior.
return values have to be captured in a list. Similarly for each return value
you have to perform the actions in a loop.
It is going to definitely slow down execution by orders of magnitude.
Does any free software or paid versions of software capture segfault in
fork on parent(P) and child (C) or something more nested than this?
P->C->P

int main(){
int i=0;
int *p=&i;
if(fork()==0){
    p=nullptr;
    if(fork()!=0){
        int z=*p+5; //segfault
    } else {
        int x;
        wait(&x);
    }
} else {
   int x;
   wait(&x);
}
}



> >
> > I was thinking of adding amd64_sys_clone3 = 435 to amd64-linux-tdep.h
> > In amd64-linux-tdep.c add  case amd64_sys_clone3: return gdb_sys_clone3;
> > In linux-record.h gdb_sys_clone3=541 in enum gdb_syscall
> these 3 changes make sense to me
> > In linux-record.c case gdb_sys_clone3=break;
> >
> >

I've created a pull request and verified that in record mode it starts
and executes
the threads but it doesn't break on the segfault in the thread.

https://github.com/bminor/binutils-gdb/pull/11


>
> --
> Cheers,
> Guinevere Larsen
> She/Her/Hers
>
> >
> > Below is gdb log and after that is source code
> >
> > gdb ./a.out
> > GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
> > Copyright (C) 2022 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.
> > Type "show copying" and "show warranty" for details.
> > This GDB was configured as "x86_64-linux-gnu".
> > Type "show configuration" for configuration details.
> > For bug reporting instructions, please see:
> > <https://www.gnu.org/software/gdb/bugs/>.
> > Find the GDB manual and other documentation resources online at:
> >      <http://www.gnu.org/software/gdb/documentation/>.
> >
> > For help, type "help".
> > Type "apropos word" to search for commands related to "word"...
> > Reading symbols from ./a.out...
> > (gdb) b main
> > Breakpoint 1 at 0x134b: file race1.cpp, line 30.
> > (gdb) r
> > Starting program: /home/navin/cpp/a.out
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> >
> > Breakpoint 1, main () at race1.cpp:30
> > 30 int main(){
> > (gdb) record full
> > (gdb) c
> > Continuing.
> > Process record and replay target doesn't support syscall number 435
> > Process record: failed to record execution log.
> >
> > Program stopped.
> > clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:60
> > 60 ../sysdeps/unix/sysv/linux/x86_64/clone3.S: No such file or directory.
> > (gdb) shell uname -a
> > Linux Navin-acer-5740 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14
> > 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
> > (gdb) shell cat /etc/lsb-release
> > DISTRIB_ID=LinuxMint
> > DISTRIB_RELEASE=21.2
> > DISTRIB_CODENAME=victoria
> > DISTRIB_DESCRIPTION="Linux Mint 21.2 Victoria"
> > (gdb)
> >
> >
> > Source Code
> >
> > #include<bits/stdc++.h>
> > #include<thread>
> > using namespace std;
> >
> > int g=0;
> > int *p=&g;
> > bool run=true;
> >
> > void t1()
> > {
> > while(run)
> > {
> > g++;
> > if(g==16)
> >     p=nullptr;
> >
> > }
> > }
> >
> > void t2()
> > {
> > int z;
> > while(run)
> > {
> > g--;
> > z=*p+1;
> > }
> > }
> >
> > int main(){
> > thread x1{t1};
> > thread x2{t2};
> > sleep(5);
> > run=false;
> > x1.join();
> > x2.join();
> > cout <<g<<endl;
> > }
> >
> >
> >
> > Regards,
> > Navin
> >
>

Re: Error record and replay target doesn't support syscall number 435

[Guinevere Larsen via Gdb]

On 5/13/24 08:06, Navin P wrote:
> Hi,
Hi! :)
>
> On Fri, May 10, 2024 at 7:29 PM Guinevere Larsen <blarsen@redhat.com> wrote:
>> On 5/10/24 05:07, Navin P via Gdb wrote:
>>> Hi,
>>>        I tried to record a program calling C++ threads and then did a
>>> break on main , then record and continue it throws the error message
>>> "record and replay target doesn't support syscall number 435" . What
>>> should I do to support syscall number 435 ?
>> Hi!
>>
>> I can't answer everything, since I'm not too familiar with syscalls
>> specifically, but I'll do my best!
>>
>> My first road bump with this is that we record that a clone has
>> happened, but we can only record the process that GDB is following
>> (parent or child, depending on follow-fork-mode). This is
>> counter-intuitive to me, since I would have expected the full inferior
>> to be replayed.
> To get this working you need minimum changes of current_inferior to
> for each inferior in all_inferiors() and execute that function for
> each inferior.
> return values have to be captured in a list. Similarly for each return value
> you have to perform the actions in a loop.

There is an extra complication, the recording subsystem was added before 
GDB could handle multiple inferiors, so it works on the assumption that 
you only need one global history. Meaning if we did record for all 
inferiors or all processes, the history would be mangled, jumping from 
one inferior to the next. It is already a problem in handling 
multi-threaded inferiors, so a proper fix would require having a 
per-thread history, and possibly some way to serialize the execution.

I added the explanation more as a motivator to why I think we should 
warn the user of the counter-intuitive behavior, at least in the 
documentation.

> It is going to definitely slow down execution by orders of magnitude.
Absolutely! I'm almost wondering if we should try to spawn a recording 
thread per process or thread, to at least make multi-threaded recording 
not worse. But that is far in the future, unrelated to this change
> Does any free software or paid versions of software capture segfault in
> fork on parent(P) and child (C) or something more nested than this?
> P->C->P
I'm unaware of any, but also GDB is the only debugger I ever really 
used, so there may be something out there.
> int main(){
> int i=0;
> int *p=&i;
> if(fork()==0){
>      p=nullptr;
>      if(fork()!=0){
>          int z=*p+5; //segfault
>      } else {
>          int x;
>          wait(&x);
>      }
> } else {
>     int x;
>     wait(&x);
> }
> }
>
>
>
>>> I was thinking of adding amd64_sys_clone3 = 435 to amd64-linux-tdep.h
>>> In amd64-linux-tdep.c add  case amd64_sys_clone3: return gdb_sys_clone3;
>>> In linux-record.h gdb_sys_clone3=541 in enum gdb_syscall
>> these 3 changes make sense to me
>>> In linux-record.c case gdb_sys_clone3=break;
>>>
>>>
> I've created a pull request and verified that in record mode it starts
> and executes
> the threads but it doesn't break on the segfault in the thread.
>
> https://github.com/bminor/binutils-gdb/pull/11
I can take a look at it as is, but that github repo is just an 
unofficial mirror. The proper way to submit changes is to send your 
patch to gdb-patches@sourceware.org, and we'll do proper review through 
email.

-- 
Cheers,
Guinevere Larsen
She/Her/Hers

>
>
>> --
>> Cheers,
>> Guinevere Larsen
>> She/Her/Hers
>>
>>> Below is gdb log and after that is source code
>>>
>>> gdb ./a.out
>>> GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
>>> Copyright (C) 2022 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.
>>> Type "show copying" and "show warranty" for details.
>>> This GDB was configured as "x86_64-linux-gnu".
>>> Type "show configuration" for configuration details.
>>> For bug reporting instructions, please see:
>>> <https://www.gnu.org/software/gdb/bugs/>.
>>> Find the GDB manual and other documentation resources online at:
>>>       <http://www.gnu.org/software/gdb/documentation/>.
>>>
>>> For help, type "help".
>>> Type "apropos word" to search for commands related to "word"...
>>> Reading symbols from ./a.out...
>>> (gdb) b main
>>> Breakpoint 1 at 0x134b: file race1.cpp, line 30.
>>> (gdb) r
>>> Starting program: /home/navin/cpp/a.out
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>
>>> Breakpoint 1, main () at race1.cpp:30
>>> 30 int main(){
>>> (gdb) record full
>>> (gdb) c
>>> Continuing.
>>> Process record and replay target doesn't support syscall number 435
>>> Process record: failed to record execution log.
>>>
>>> Program stopped.
>>> clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:60
>>> 60 ../sysdeps/unix/sysv/linux/x86_64/clone3.S: No such file or directory.
>>> (gdb) shell uname -a
>>> Linux Navin-acer-5740 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14
>>> 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
>>> (gdb) shell cat /etc/lsb-release
>>> DISTRIB_ID=LinuxMint
>>> DISTRIB_RELEASE=21.2
>>> DISTRIB_CODENAME=victoria
>>> DISTRIB_DESCRIPTION="Linux Mint 21.2 Victoria"
>>> (gdb)
>>>
>>>
>>> Source Code
>>>
>>> #include<bits/stdc++.h>
>>> #include<thread>
>>> using namespace std;
>>>
>>> int g=0;
>>> int *p=&g;
>>> bool run=true;
>>>
>>> void t1()
>>> {
>>> while(run)
>>> {
>>> g++;
>>> if(g==16)
>>>      p=nullptr;
>>>
>>> }
>>> }
>>>
>>> void t2()
>>> {
>>> int z;
>>> while(run)
>>> {
>>> g--;
>>> z=*p+1;
>>> }
>>> }
>>>
>>> int main(){
>>> thread x1{t1};
>>> thread x2{t2};
>>> sleep(5);
>>> run=false;
>>> x1.join();
>>> x2.join();
>>> cout <<g<<endl;
>>> }
>>>
>>>
>>>
>>> Regards,
>>> Navin
>>>

Re: Error record and replay target doesn't support syscall number 435

[Guinevere Larsen via Gdb]

Re-adding gdb list, since you mentioned you dropped on accident
On 5/13/24 12:13, Navin P wrote:
> On Mon, May 13, 2024 at 6:40 PM Guinevere Larsen <blarsen@redhat.com> wrote:
>> On 5/13/24 08:06, Navin P wrote:
>>> Hi,
>> Hi! :)
>>> On Fri, May 10, 2024 at 7:29 PM Guinevere Larsen <blarsen@redhat.com> wrote:
>>>> On 5/10/24 05:07, Navin P via Gdb wrote:
>>>>> Hi,
>>>>>         I tried to record a program calling C++ threads and then did a
>>>>> break on main , then record and continue it throws the error message
>>>>> "record and replay target doesn't support syscall number 435" . What
>>>>> should I do to support syscall number 435 ?
>>>> Hi!
>>>>
>>>> I can't answer everything, since I'm not too familiar with syscalls
>>>> specifically, but I'll do my best!
>>>>
>>>> My first road bump with this is that we record that a clone has
>>>> happened, but we can only record the process that GDB is following
>>>> (parent or child, depending on follow-fork-mode). This is
>>>> counter-intuitive to me, since I would have expected the full inferior
>>>> to be replayed.
>>> To get this working you need minimum changes of current_inferior to
>>> for each inferior in all_inferiors() and execute that function for
>>> each inferior.
>>> return values have to be captured in a list. Similarly for each return value
>>> you have to perform the actions in a loop.
>> There is an extra complication, the recording subsystem was added before
>> GDB could handle multiple inferiors, so it works on the assumption that
>> you only need one global history. Meaning if we did record for all
>> inferiors or all processes, the history would be mangled, jumping from
>> one inferior to the next. It is already a problem in handling
>> multi-threaded inferiors, so a proper fix would require having a
>> per-thread history, and possibly some way to serialize the execution.
>>
>> I added the explanation more as a motivator to why I think we should
>> warn the user of the counter-intuitive behavior, at least in the
>> documentation.
>>
>>> It is going to definitely slow down execution by orders of magnitude.
>> Absolutely! I'm almost wondering if we should try to spawn a recording
>> thread per process or thread, to at least make multi-threaded recording
>> not worse. But that is far in the future, unrelated to this change
> static void
> record_full_core_open_1 ()
> {
>    regcache *regcache = get_thread_regcache (inferior_thread ())
>
> In record-full.c we have inferior_thread() which is the current thread,
>   so only 1 thread is recorded ?  That is my observation too.
I am not sure if only thread 1 is recorded, or if it is mixing 
instructions from all threads, but only changing register values for one 
of the threads. You can disassemble thread 1, then reverse-stepi to see 
if the PC changes wildly or if phantom instructions show up.
> If so, what is required to fix it for all threads.
I haven't fully accounted for everything that is required to make record 
work for multi-threaded inferiors. What I know we strictly need to 
either double-check or improve is:
     * If you check gdb/record-full.c:197, you'll see how we store the 
full history. Its a single global variable for everything that is 
recorded. This needs to be turned into a thread specific member.
     * I think the way GDB handles step means multiple threads will all 
be handled by a single record target, in a single thread. This needs to 
be verified, and if I'm right, we need to find a way to serialize the 
threads, so that we can replay thread in the correct order.
     * There's an open bug for gdb not properly handling (at least) 
sigtrap when an inferior is added: 
https://sourceware.org/bugzilla/show_bug.cgi?id=30025. We need to fix 
that if multiple processes are handled like multiple inferiors as you 
mentioned.

Properly handling multi-threaded has been in my backlog for a bit, but I 
have one task for record that's much more important - adding support to 
AVX instructions - and I have limited time for record stuff, unfortunately.
> Here there is only
> one process
> with multiple threads. I expect it to catch the segfault in thread 2
> but it fails
> and goes into an infinite loop.

Instead of a global variable, try using a counter on the while loops, 
and set it to a small value like 3-4. The current slowdown for execution 
is unbelievable (was also a backlogged thing for me).

Also, I don't know if your system doesn't use a libc++ that's very 
optimized, but I can't record the creation of std::thread objects. 
That's because std::thread::_M_start_thread uses memset somewhere, and 
my glibc has the optimized avx512 version of memset, an instruction gdb 
can't disassemble yet. So we can add "supporting newer x86_64 
instructions" to the list of tasks to do if we want to _properly_ 
support multi-threading. And if you want to double check what I 
mentioned above, I'd also recommend not printing, because that's also 
libc++, so likely to be using optimized instructions.

-- 
Cheers,
Guinevere Larsen
She/Her/Hers

>
> #include<thread>
> #include<iostream>
> #include<unistd.h>
> using namespace std;
> bool run=true;
> int i=6;
> int *p=&i,g;
>
> void t1() {
> while(run){
>       g++;
>       if(g>=16)
>            p=nullptr;
>        cout <<"t1 "<<g<<endl;
>    }
> }
>
> void t2(){
> int t=0;
> while(run){
>         g++;
>         t=*p + 5;
>         cout <<"t2 "<<g<<endl;
>    }
> }
>
>
> int main(){
> thread x1{t1},x2{t2};
> sleep(5);
> run=false;
> x1.join();x2.join();
> }
>
>
> Regards,
> Navin
>