Host GDB disconnect while waiting for tracee status change

Host GDB disconnect while waiting for tracee status change

[Dmitry Antipov]

Hello,

Is there a feature/option/etc. to gracefully handle the situation when gdbserver
sits waiting for tracee status change and the network connection is getting lost?

AFAICS in case of Linux, linux_wait_for_event_filtered() just enters sigsuspend()
waiting for SIGCHLD and ignoring anything else, thus having no opportunity to detect
any socket status changes even if the latter are signaled with SIGIOs requested
in enable_async_notification().

Thanks in advance,
Dmitry

Re: Host GDB disconnect while waiting for tracee status change

[Yao Qi]

Dmitry Antipov <dantipov@nvidia.com> writes:

> Is there a feature/option/etc. to gracefully handle the situation when gdbserver
> sits waiting for tracee status change and the network connection is
> getting lost?
>

I didn't try GDBserver under the case you described, but I find
something related int GDB manual,

https://sourceware.org/gdb/onlinedocs/gdb/Server.html
"Therefore, when the connection drops unexpectedly, and GDB cannot ask
gdbserver to kill its debugged processes, gdbserver stays running even
in the target remote mode."

GDBserver is able to handle this for tracepoint, which is called
disconnected-tracing.
https://sourceware.org/gdb/onlinedocs/gdb/Starting-and-Stopping-Trace-Experiments.html#disconnected-tracing
With tracepoint, GDBserver knows what it should do when tracee status
changed and don't have to communicate with GDB.

> AFAICS in case of Linux, linux_wait_for_event_filtered() just enters
> sigsuspend()
> waiting for SIGCHLD and ignoring anything else, thus having no
> opportunity to detect
> any socket status changes even if the latter are signaled with SIGIOs requested
> in enable_async_notification().

-- 
Yao (齐尧)

Re: Host GDB disconnect while waiting for tracee status change

[Dmitry Antipov]

On 08/23/2017 11:18 AM, Yao Qi wrote:

> I didn't try GDBserver under the case you described, but I find
> something related int GDB manual,
> 
> https://sourceware.org/gdb/onlinedocs/gdb/Server.html
> "Therefore, when the connection drops unexpectedly, and GDB cannot ask
> gdbserver to kill its debugged processes, gdbserver stays running even
> in the target remote mode."

It's definitely stays running, but seems doesn't listen for remote connection
attempts anymore. For example:

$ cat /tmp/t-simple.c
int main (int argc, char *argv[]) { while (1) (void) argc; return 0; }

$ gdbserver --debug --attach tcp:9999 $(pidof t-simple)

$ gdb -q
(gdb) target extended-remote :9999
Remote debugging using :9999
[...skipped...]
0x00000000004004f2 in main (argc=1, argv=0x7ffec6b6b208) at /tmp/t-simple.c:1
1	int main (int argc, char *argv[]) { while (1) (void) argc; return 0; }
(gdb) c
Continuing.
Segmentation fault (core dumped) ;; Due to killall -11 gdb from another TTY

Immediately after killing gdb, gdbserver reports:
...
client connection closed
my_waitpid (-1, 0x40000001)
my_waitpid (-1, 0x40000001): status(ffffffff), 0
LWFE: waitpid(-1, ...) returned 0, ERRNO-OK
leader_pid=21352, leader_lp!=NULL=1, num_lwps=1, zombie=0
sigsuspend'ing
...

Next, a new instance of gdb is unable to reconnect:

$ gdb -q
(gdb) target extended-remote :9999
Remote debugging using :9999
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Remote replied unexpectedly to 'vMustReplyEmpty': timeout

And it seems that there is no similar issue in a non-stop mode. For example:

$ gdb -q
(gdb) set non-stop 1
(gdb) target extended-remote :9999
Remote debugging using :9999
[...skipped...]
Program stopped.
0x00000000004004f2 in main (argc=1, argv=0x7ffc72333088) at /tmp/t-simple.c:1
1	int main (int argc, char *argv[]) { while (1) (void) argc; return 0; }
(gdb) c
Continuing.
Segmentation fault (core dumped) ;; Due to killall -11 gdb from another TTY

Immediately after killing gdb, gdbserver reports:

...
handling possible serial event
Remote side has terminated connection.  GDBserver will reopen the connection.
Listening on port 9999
...

This way, a new instance of gdb can reconnect:

$ gdb -q
(gdb) set non-stop 1
(gdb) target extended-remote :9999
Remote debugging using :9999
[...skipped...]
(gdb) interrupt
Program stopped.
0x00000000004004f2 in main (argc=1, argv=0x7ffc72333088) at /tmp/t-simple.c:1
1	int main (int argc, char *argv[]) { while (1) (void) argc; return 0; }
(gdb)

Dmitry

Re: Host GDB disconnect while waiting for tracee status change

[Pedro Alves]

On 08/23/2017 10:37 AM, Dmitry Antipov wrote:

> Immediately after killing gdb, gdbserver reports:
> ...
> client connection closed
> my_waitpid (-1, 0x40000001)
> my_waitpid (-1, 0x40000001): status(ffffffff), 0
> LWFE: waitpid(-1, ...) returned 0, ERRNO-OK
> leader_pid=21352, leader_lp!=NULL=1, num_lwps=1, zombie=0
> sigsuspend'ing
> ...
> 
> Next, a new instance of gdb is unable to reconnect:

...

> And it seems that there is no similar issue in a non-stop mode. For
> example:

...

> This way, a new instance of gdb can reconnect:

Right, because in non-stop mode, linux-low.c doesn't
usually block inside mywait....sigsuspend.  See server.c:resume.
In non-stop mode, gdbserver waits for both socket events and
target events at the same time in the event loop.  The
non-stop variant of the RSP is asynchronous.

Thanks,
Pedro Alves

Re: Host GDB disconnect while waiting for tracee status change

[Dmitry Antipov]

On 08/23/2017 04:10 PM, Pedro Alves wrote:

> Right, because in non-stop mode, linux-low.c doesn't
> usually block inside mywait....sigsuspend.  See server.c:resume.
> In non-stop mode, gdbserver waits for both socket events and
> target events at the same time in the event loop.  The
> non-stop variant of the RSP is asynchronous.

Thanks. So, is it mandatory to use non-stop mode if I want to be able to
restart gdb without loosing the debugging session controlled by gdbserver?

Dmitry

Re: Host GDB disconnect while waiting for tracee status change

[Pedro Alves]

On 08/23/2017 02:55 PM, Dmitry Antipov wrote:
> On 08/23/2017 04:10 PM, Pedro Alves wrote:
> 
>> Right, because in non-stop mode, linux-low.c doesn't
>> usually block inside mywait....sigsuspend.  See server.c:resume.
>> In non-stop mode, gdbserver waits for both socket events and
>> target events at the same time in the event loop.  The
>> non-stop variant of the RSP is asynchronous.
> 
> Thanks. So, is it mandatory to use non-stop mode if I want to be able to
> restart gdb without loosing the debugging session controlled by gdbserver?

In all-stop/sync mode, you can only exit gdb if the
target is stopped, in which case you can't get into the
problem case.  If you're considering "restarting gdb" manually,
then no.  This is about the server robustly handling the case of
the connection unexpectedly closing, like GDB crashing.
And gdbserver could be fixed, somehow.  This is a quality of
implementation issue, not a fundamental insurmountable design
problem.

Thanks,
Pedro Alves