Re: [PATCH] gdb: make "start" breakpoint inferior-specific

[Andrew Burgess via Gdb-patches <gdb-patches@sourceware.org>]

Simon Marchi via Gdb-patches <gdb-patches@sourceware.org> writes:

> I saw this failure on a CI:
>
>     (gdb) add-inferior
>     [New inferior 2]
>     Added inferior 2
>     (gdb) PASS: gdb.threads/vfork-multi-inferior.exp: method=non-stop: add-inferior
>     inferior 2
>     [Switching to inferior 2 [<null>] (<noexec>)]
>     (gdb) PASS: gdb.threads/vfork-multi-inferior.exp: method=non-stop: inferior 2
>     kill
>     The program is not being run.
>     (gdb) file /home/jenkins/workspace/binutils-gdb_master_linuxbuild/platform/jammy-amd64/target_board/unix/tmp/tmp.GYATAXR8Ku/gdb/testsuite/outputs/gdb.threads/vfork-multi-inferior/vfork-multi-inferior-sleep
>     Reading symbols from /home/jenkins/workspace/binutils-gdb_master_linuxbuild/platform/jammy-amd64/target_board/unix/tmp/tmp.GYATAXR8Ku/gdb/testsuite/outputs/gdb.threads/vfork-multi-inferior/vfork-multi-inferior-sleep...
>     (gdb) run &
>     Starting program: /home/jenkins/workspace/binutils-gdb_master_linuxbuild/platform/jammy-amd64/target_board/unix/tmp/tmp.GYATAXR8Ku/gdb/testsuite/outputs/gdb.threads/vfork-multi-inferior/vfork-multi-inferior-sleep
>     (gdb) PASS: gdb.threads/vfork-multi-inferior.exp: method=non-stop: run inferior 2
>     inferior 1
>     [Switching to inferior 1 [<null>] (<noexec>)]
>     (gdb) PASS: gdb.threads/vfork-multi-inferior.exp: method=non-stop: inferior 1
>     kill
>     The program is not being run.
>     (gdb) file /home/jenkins/workspace/binutils-gdb_master_linuxbuild/platform/jammy-amd64/target_board/unix/tmp/tmp.GYATAXR8Ku/gdb/testsuite/outputs/gdb.threads/vfork-multi-inferior/vfork-multi-inferior
>     Reading symbols from /home/jenkins/workspace/binutils-gdb_master_linuxbuild/platform/jammy-amd64/target_board/unix/tmp/tmp.GYATAXR8Ku/gdb/testsuite/outputs/gdb.threads/vfork-multi-inferior/vfork-multi-inferior...
>     (gdb) break should_break_here
>     Breakpoint 1 at 0x11b1: file /home/jenkins/workspace/binutils-gdb_master_linuxbuild/platform/jammy-amd64/target_board/unix/src/binutils-gdb/gdb/testsuite/gdb.threads/vfork-multi-inferior.c, line 25.
>     (gdb) PASS: gdb.threads/vfork-multi-inferior.exp: method=non-stop: break should_break_here
>     [Thread debugging using libthread_db enabled]
>     Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>     start
>     Temporary breakpoint 2 at 0x11c0: -qualified main. (2 locations)
>     Starting program: /home/jenkins/workspace/binutils-gdb_master_linuxbuild/platform/jammy-amd64/target_board/unix/tmp/tmp.GYATAXR8Ku/gdb/testsuite/outputs/gdb.threads/vfork-multi-inferior/vfork-multi-inferior
>     [Thread debugging using libthread_db enabled]
>     Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>
>     Thread 2.1 "vfork-multi-inf" hit Temporary breakpoint 2, main () at /home/jenkins/workspace/binutils-gdb_master_linuxbuild/platform/jammy-amd64/target_board/unix/src/binutils-gdb/gdb/testsuite/gdb.threads/vfork-multi-inferior-sleep.c:23
>     23	  sleep (30);
>     (gdb) FAIL: gdb.threads/vfork-multi-inferior.exp: method=non-stop: start inferior 1
>
> What happens is:
>
>  1. We start inferior 2 with "run&", it runs very slowly, takes time to
>     get to main
>  2. We switch to inferior 1, and run "start"
>  3. The temporary breakpoint inserted by "start" applies to all inferiors
>  4. Inferior 2 hits that breakpoint and GDB reports that hit
>
> To avoid this, breakpoints inserted by "start" should be
> inferior-specific.  However, we don't have a nice way to make
> inferior-specific breakpoints yet.  It's possible to make
> pspace-specific breakpoints (for example how the internal_breakpoint
> constructor does) by creating a symtab_and_line manually.  However,
> inferiors can share program spaces (usually on particular embedded
> targets), so we could have a situation where two inferiors run the same
> code in the same program space.  In that case, it would just not be
> possible to insert a breakpoint in one inferior but not the other.
>
> A simple solution that should work all the time is to add a condition to
> the breakpoint inserted by "start", to check the inferior reporting the
> hit is the expected one.  This is what this patch implements.
>
> Add a test that does:
>
>  - start a background inferior 1 that calls main in a loop
>  - start aninferior 2 with the "start" command
>  - validate that we hit the breakpoint in inferior 2
>
> Without the fix, we hit the breakpoint in inferior 1 pretty much all the
> time.
>
> Change-Id: Ib0148498a476bfa634ed62353c95f163623c686a
> ---
>  gdb/infcmd.c                                  |  3 +-
>  .../gdb.multi/start-inferior-specific-other.c | 30 ++++++++++
>  .../gdb.multi/start-inferior-specific.c       | 22 +++++++
>  .../gdb.multi/start-inferior-specific.exp     | 58 +++++++++++++++++++
>  4 files changed, 112 insertions(+), 1 deletion(-)
>  create mode 100644 gdb/testsuite/gdb.multi/start-inferior-specific-other.c
>  create mode 100644 gdb/testsuite/gdb.multi/start-inferior-specific.c
>  create mode 100644 gdb/testsuite/gdb.multi/start-inferior-specific.exp
>
> diff --git a/gdb/infcmd.c b/gdb/infcmd.c
> index 69a7896b560..e9107cef520 100644
> --- a/gdb/infcmd.c
> +++ b/gdb/infcmd.c
> @@ -423,7 +423,8 @@ run_command_1 (const char *args, int from_tty, enum run_how run_how)
>    /* Insert temporary breakpoint in main function if requested.  */
>    if (run_how == RUN_STOP_AT_MAIN)
>      {
> -      std::string arg = string_printf ("-qualified %s", main_name ());
> +      std::string arg = string_printf ("-qualified %s if $_inferior == %d", main_name (),
> +				       current_inferior ()->num);
>        tbreak_command (arg.c_str (), 0);
>      }
>  
> diff --git a/gdb/testsuite/gdb.multi/start-inferior-specific-other.c b/gdb/testsuite/gdb.multi/start-inferior-specific-other.c
> new file mode 100644
> index 00000000000..b593c3aa068
> --- /dev/null
> +++ b/gdb/testsuite/gdb.multi/start-inferior-specific-other.c
> @@ -0,0 +1,30 @@
> +/* This testcase is part of GDB, the GNU debugger.
> +
> +   Copyright 2022 Free Software Foundation, Inc.
> +
> +   This program is free software; you can redistribute it and/or modify
> +   it under the terms of the GNU General Public License as published by
> +   the Free Software Foundation; either version 3 of the License, or
> +   (at your option) any later version.
> +
> +   This program is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +   GNU General Public License for more details.
> +
> +   You should have received a copy of the GNU General Public License
> +   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
> +
> +#include <stdlib.h>
> +
> +int
> +main (int argc, const char **argv)
> +{
> +  if (argc == 999)
> +    return 0;
> +
> +  for (;;)
> +    main (999, NULL);
> +
> +  return 0;
> +}

I think test cases like this should have an `alarm` call, just in case
something goes wrong, this'll stop the test running forever.

> diff --git a/gdb/testsuite/gdb.multi/start-inferior-specific.c b/gdb/testsuite/gdb.multi/start-inferior-specific.c
> new file mode 100644
> index 00000000000..b69e218962a
> --- /dev/null
> +++ b/gdb/testsuite/gdb.multi/start-inferior-specific.c
> @@ -0,0 +1,22 @@
> +/* This testcase is part of GDB, the GNU debugger.
> +
> +   Copyright 2022 Free Software Foundation, Inc.
> +
> +   This program is free software; you can redistribute it and/or modify
> +   it under the terms of the GNU General Public License as published by
> +   the Free Software Foundation; either version 3 of the License, or
> +   (at your option) any later version.
> +
> +   This program is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +   GNU General Public License for more details.
> +
> +   You should have received a copy of the GNU General Public License
> +   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
> +
> +int
> +main ()
> +{
> +  return 0;
> +}
> diff --git a/gdb/testsuite/gdb.multi/start-inferior-specific.exp b/gdb/testsuite/gdb.multi/start-inferior-specific.exp
> new file mode 100644
> index 00000000000..5a5572f591a
> --- /dev/null
> +++ b/gdb/testsuite/gdb.multi/start-inferior-specific.exp
> @@ -0,0 +1,58 @@
> +# Copyright 2022 Free Software Foundation, Inc.
> +
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 3 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +# Test that "start"ing an inferior does not inadvertently stop in another
> +# inferior.
> +#
> +# To achieve this, we have an inferior (the  "other") running in background
> +# constantly re-calling main.  We then "start" a second inferior.  A buggy GDB
> +# would report a breakpoint hit in "other".
> +
> +standard_testfile .c -other.c
> +
> +if { [use_gdb_stub] } {
> +    return
> +}
> +
> +set srcfile_other ${srcfile2}
> +set binfile_other ${binfile}-other
> +
> +if { [build_executable ${testfile}.exp ${binfile} "${srcfile}" {debug}] != 0 } {
> +    return -1
> +}
> +
> +if { [build_executable ${testfile}.exp ${binfile_other} "${srcfile_other}" {debug}] != 0 } {
> +    return -1
> +}
> +
> +proc do_test {} {
> +    # With remote, to be able to start an inferior while another one is
> +    # running, we need to use the non-stop variant of the protocol.
> +    save_vars { ::GDBFLAGS } {
> +	if { [target_info gdb_protocol] == "extended-remote"} {
> +	    append ::GDBFLAGS " -ex \"maintenance set target-non-stop on\""
> +	}
> +
> +	clean_restart ${::binfile_other}
> +    }
> +
> +    gdb_test "run&" "Starting program: .*" "start background inferior"
> +    gdb_test "add-inferior" "Added inferior 2.*"
> +    gdb_test "inferior 2" "Switching to inferior 2.*"
> +    gdb_file_cmd ${::binfile}
> +    gdb_test "start" "Thread 2.1 .* hit Temporary breakpoint .*"

I'm pretty sure there's a race here.  We could set the breakpoint for
the 'start', start inferior 2 executing, hit the b/p and remove it, all
within the time it takes inferior 1 to perform a single iteration ... in
theory, though it does seem unlikely.

If, before the `start` you did `break *main` then we should hit the
`*main` breakpoint before the temporary `start` breakpoint, the
temporary breakpoint is left in place, which gives inferior 1 additional
time to not hit the breakpoint.

Of course, this relies on the `*main` and `start` breakpoints being at
different addresses, which should be the case if main has some prologue
to skip over.

I guess, ideally we would also do something to confirm that inferior 1
has gone past main while the temporary breakpoint is in place, right now
we're just relying on inferior 1 running fast enough.

In general I'm happy enough with the actual GDB change you propose here,
but like Bruno, I also wondered if it would be better to add inferior
specific breakpoints as an actual thing, rather than relying on the
condition logic like you do here?

I wasn't sure how you'd feel about this idea, so I didn't spend too
long, but below is a VERY rough proof of concept patch, that appies on
top of yours, that adds inferior specific breakpoints.  You temporary
breakpoint now becomes:

  tbreak -qualified main inferior 1

But this functionality would also be available to a user if they wanted
it, which might be nice.

When I say the code below is rough, I really mean it, here's a list of
things I think still need doing:

  * I initially added a `breakpoint::inferior` field just like we have a
    `breakpoint::thread` field, but I think a better solution would be
    to change `breakpoint::thread` into a ptid_t and store inferior and
    thread information in that one field,

  * Obviously it should give an error if a user specifies both `thread`
    and `inferior` for a breakpoint,

  * The `info breakpoints` output is fine for single location
    breakpoints, but I think multi-location breakpoints would need
    sorting out still,

  * There's a couple of TODO comments in there still to address,

  * Need to write docs, NEWS, and add tests,

Like I said, you might see a reason why this is a bad idea, so I didn't
want to put any more work in without having a discussion about the idea.

Thoughts?

Thanks,
Andrew

---

commit 26a101c678cd14828cbd7f8c15240854a50c91e2
Author: Andrew Burgess <aburgess@redhat.com>
Date:   Thu Sep 1 10:44:27 2022 +0100

    wip: per-inferior breakpoints

diff --git a/gdb/breakpoint.c b/gdb/breakpoint.c
index 63feea9e9cd..2ea453ff2c4 100644
--- a/gdb/breakpoint.c
+++ b/gdb/breakpoint.c
@@ -96,7 +96,7 @@ static void create_breakpoints_sal (struct gdbarch *,
 				    gdb::unique_xmalloc_ptr<char>,
 				    gdb::unique_xmalloc_ptr<char>,
 				    enum bptype,
-				    enum bpdisp, int, int,
+				    enum bpdisp, int, int, int,
 				    int,
 				    int, int, int, unsigned);
 
@@ -3101,6 +3101,17 @@ update_inserted_breakpoint_locations (void)
     }
 }
 
+static bool
+valid_global_inferior_id (int id)
+{
+  for (inferior *inf : all_inferiors ())
+    if (inf->num == id)
+      return true;
+
+  return false;
+}
+
+
 /* Used when starting or continuing the program.  */
 
 static void
@@ -3132,6 +3143,10 @@ insert_breakpoint_locations (void)
 	  && !valid_global_thread_id (bl->owner->thread))
 	continue;
 
+      if (bl->owner->inferior != -1
+	  && !valid_global_inferior_id (bl->owner->inferior))
+	continue;
+
       switch_to_program_space_and_thread (bl->pspace);
 
       /* For targets that support global breakpoints, there's no need
@@ -3236,6 +3251,27 @@ Thread-specific breakpoint %d deleted - thread %s no longer in the thread list.\
     }
 }
 
+/* ... */
+
+static void
+remove_inferior_breakpoints (struct inferior *inf)
+{
+  for (breakpoint *b : all_breakpoints_safe ())
+    {
+      if (b->inferior == inf->num && user_breakpoint_p (b))
+	{
+	  b->disposition = disp_del_at_next_stop;
+
+	  gdb_printf (_("\
+Inferior-specific breakpoint %d deleted - inferior %d has exited.\n"),
+		      b->number, inf->num);
+
+	  /* Hide it from the user.  */
+	  b->number = 0;
+       }
+    }
+}
+
 /* See breakpoint.h.  */
 
 void
@@ -5331,6 +5367,7 @@ bpstat_check_breakpoint_conditions (bpstat *bs, thread_info *thread)
      evaluating the condition if this isn't the specified
      thread/task.  */
   if ((b->thread != -1 && b->thread != thread->global_num)
+      || (b->inferior != -1 && b->inferior != thread->inf->num)
       || (b->task != 0 && b->task != ada_get_task_number (thread)))
     {
       bs->stop = 0;
@@ -6339,6 +6376,11 @@ print_one_breakpoint_location (struct breakpoint *b,
 	  uiout->text (" task ");
 	  uiout->field_signed ("task", b->task);
 	}
+      else if (b->inferior != -1)
+	{
+	  uiout->text (" inferior ");
+	  uiout->field_signed ("inferior", b->inferior);
+	}
     }
 
   uiout->text ("\n");
@@ -6384,16 +6426,34 @@ print_one_breakpoint_location (struct breakpoint *b,
       /* FIXME should make an annotation for this.  */
       uiout->text ("\tstop only in thread ");
       if (uiout->is_mi_like_p ())
-	uiout->field_signed ("thread", b->thread);
+	{
+	  uiout->field_signed ("thread", b->thread);
+	}
       else
 	{
 	  struct thread_info *thr = find_thread_global_id (b->thread);
 
 	  uiout->field_string ("thread", print_thread_id (thr));
+	  uiout->field_signed ("thread", b->inferior);
 	}
       uiout->text ("\n");
     }
-  
+
+  if (!part_of_multiple && b->inferior != -1)
+    {
+      /* FIXME should make an annotation for this.  */
+      uiout->text ("\tstop only in inferior ");
+      if (uiout->is_mi_like_p ())
+	{
+	  uiout->field_signed ("inferior", b->inferior);
+	}
+      else
+	{
+	  uiout->field_signed ("inferior", b->inferior);
+	}
+      uiout->text ("\n");
+    }
+
   if (!part_of_multiple)
     {
       if (b->hit_count)
@@ -7366,7 +7426,10 @@ delete_longjmp_breakpoint (int thread)
     if (b->type == bp_longjmp || b->type == bp_exception)
       {
 	if (b->thread == thread)
-	  delete_breakpoint (b);
+	  {
+	    gdb_assert (b->inferior == -1);
+	    delete_breakpoint (b);
+	  }
       }
 }
 
@@ -7377,7 +7440,10 @@ delete_longjmp_breakpoint_at_next_stop (int thread)
     if (b->type == bp_longjmp || b->type == bp_exception)
       {
 	if (b->thread == thread)
-	  b->disposition = disp_del_at_next_stop;
+	  {
+	    gdb_assert (b->inferior == -1);
+	    b->disposition = disp_del_at_next_stop;
+	  }
       }
 }
 
@@ -7430,6 +7496,7 @@ check_longjmp_breakpoint_for_call_dummy (struct thread_info *tp)
   ALL_BREAKPOINTS_SAFE (b, b_tmp)
     if (b->type == bp_longjmp_call_dummy && b->thread == tp->global_num)
       {
+	gdb_assert (b->inferior == -1);
 	struct breakpoint *dummy_b = b->related_breakpoint;
 
 	/* Find the bp_call_dummy breakpoint in the list of breakpoints
@@ -8263,7 +8330,8 @@ code_breakpoint::code_breakpoint (struct gdbarch *gdbarch_,
 				  gdb::unique_xmalloc_ptr<char> cond_string_,
 				  gdb::unique_xmalloc_ptr<char> extra_string_,
 				  enum bpdisp disposition_,
-				  int thread_, int task_, int ignore_count_,
+				  int thread_, int task_, int inferior_,
+				  int ignore_count_,
 				  int from_tty,
 				  int enabled_, unsigned flags,
 				  int display_canonical_)
@@ -8289,6 +8357,7 @@ code_breakpoint::code_breakpoint (struct gdbarch *gdbarch_,
 
   thread = thread_;
   task = task_;
+  inferior = inferior_;
 
   cond_string = std::move (cond_string_);
   extra_string = std::move (extra_string_);
@@ -8390,7 +8459,7 @@ create_breakpoint_sal (struct gdbarch *gdbarch,
 		       gdb::unique_xmalloc_ptr<char> cond_string,
 		       gdb::unique_xmalloc_ptr<char> extra_string,
 		       enum bptype type, enum bpdisp disposition,
-		       int thread, int task, int ignore_count,
+		       int thread, int task, int inferior, int ignore_count,
 		       int from_tty,
 		       int enabled, int internal, unsigned flags,
 		       int display_canonical)
@@ -8404,7 +8473,7 @@ create_breakpoint_sal (struct gdbarch *gdbarch,
 				std::move (cond_string),
 				std::move (extra_string),
 				disposition,
-				thread, task, ignore_count,
+				thread, task, inferior, ignore_count,
 				from_tty,
 				enabled, flags,
 				display_canonical);
@@ -8433,7 +8502,8 @@ create_breakpoints_sal (struct gdbarch *gdbarch,
 			gdb::unique_xmalloc_ptr<char> cond_string,
 			gdb::unique_xmalloc_ptr<char> extra_string,
 			enum bptype type, enum bpdisp disposition,
-			int thread, int task, int ignore_count,
+			int thread, int task, int inferior,
+			int ignore_count,
 			int from_tty,
 			int enabled, int internal, unsigned flags)
 {
@@ -8457,7 +8527,7 @@ create_breakpoints_sal (struct gdbarch *gdbarch,
 			     std::move (cond_string),
 			     std::move (extra_string),
 			     type, disposition,
-			     thread, task, ignore_count,
+			     thread, task, inferior, ignore_count,
 			     from_tty, enabled, internal, flags,
 			     canonical->special_display);
     }
@@ -8597,11 +8667,12 @@ check_fast_tracepoint_sals (struct gdbarch *gdbarch,
 static void
 find_condition_and_thread (const char *tok, CORE_ADDR pc,
 			   gdb::unique_xmalloc_ptr<char> *cond_string,
-			   int *thread, int *task,
+			   int *thread, int *inferior, int *task,
 			   gdb::unique_xmalloc_ptr<char> *rest)
 {
   cond_string->reset ();
   *thread = -1;
+  *inferior = -1;
   *task = 0;
   rest->reset ();
   bool force = false;
@@ -8659,6 +8730,16 @@ find_condition_and_thread (const char *tok, CORE_ADDR pc,
 	  *thread = thr->global_num;
 	  tok = tmptok;
 	}
+      else if (toklen >= 1 && strncmp (tok, "inferior", toklen) == 0)
+	{
+	  char *tmptok;
+
+	  tok = end_tok + 1;
+	  *inferior = strtol (tok, &tmptok, 0);
+	  if (tok == tmptok)
+	    error (_("Junk after thread keyword."));
+	  tok = tmptok;
+	}
       else if (toklen >= 1 && strncmp (tok, "task", toklen) == 0)
 	{
 	  char *tmptok;
@@ -8690,7 +8771,7 @@ static void
 find_condition_and_thread_for_sals (const std::vector<symtab_and_line> &sals,
 				    const char *input,
 				    gdb::unique_xmalloc_ptr<char> *cond_string,
-				    int *thread, int *task,
+				    int *thread, int *inferior, int *task,
 				    gdb::unique_xmalloc_ptr<char> *rest)
 {
   int num_failures = 0;
@@ -8698,6 +8779,7 @@ find_condition_and_thread_for_sals (const std::vector<symtab_and_line> &sals,
     {
       gdb::unique_xmalloc_ptr<char> cond;
       int thread_id = 0;
+      int inferior_id = 0;
       int task_id = 0;
       gdb::unique_xmalloc_ptr<char> remaining;
 
@@ -8710,9 +8792,10 @@ find_condition_and_thread_for_sals (const std::vector<symtab_and_line> &sals,
       try
 	{
 	  find_condition_and_thread (input, sal.pc, &cond, &thread_id,
-				     &task_id, &remaining);
+				     &inferior_id, &task_id, &remaining);
 	  *cond_string = std::move (cond);
 	  *thread = thread_id;
+	  *inferior = inferior_id;
 	  *task = task_id;
 	  *rest = std::move (remaining);
 	  break;
@@ -8814,6 +8897,7 @@ create_breakpoint (struct gdbarch *gdbarch,
   struct linespec_result canonical;
   int pending = 0;
   int task = 0;
+  int inferior = 0;
   int prev_bkpt_count = breakpoint_count;
 
   gdb_assert (ops != NULL);
@@ -8891,7 +8975,8 @@ create_breakpoint (struct gdbarch *gdbarch,
 	  const linespec_sals &lsal = canonical.lsals[0];
 
 	  find_condition_and_thread_for_sals (lsal.sals, extra_string,
-					      &cond, &thread, &task, &rest);
+					      &cond, &thread, &inferior,
+					      &task, &rest);
 	  cond_string_copy = std::move (cond);
 	  extra_string_copy = std::move (rest);
 	}
@@ -8941,7 +9026,7 @@ create_breakpoint (struct gdbarch *gdbarch,
 				   std::move (extra_string_copy),
 				   type_wanted,
 				   tempflag ? disp_del : disp_donttouch,
-				   thread, task, ignore_count,
+				   thread, task, inferior, ignore_count,
 				   from_tty, enabled, internal, flags);
     }
   else
@@ -9886,6 +9971,7 @@ watch_command_1 (const char *arg, int accessflag, int from_tty,
   const char *cond_end = NULL;
   enum bptype bp_type;
   int thread = -1;
+  int inferior = -1;
   /* Flag to indicate whether we are going to use masks for
      the hardware watchpoint.  */
   bool use_mask = false;
@@ -9956,6 +10042,15 @@ watch_command_1 (const char *arg, int accessflag, int from_tty,
 	      if (!valid_task_id (task))
 		error (_("Unknown task %d."), task);
 	    }
+	  else if (toklen == 8 && startswith (tok, "inferior"))
+	    {
+	      char *tmp;
+
+	      inferior = strtol (value_start, &tmp, 0);
+	      if (tmp == value_start)
+		error (_("Junk after task keyword."));
+	      /* TODO: Validate the inferior number.  */
+	    }
 	  else if (toklen == 4 && startswith (tok, "mask"))
 	    {
 	      /* We've found a "mask" token, which means the user wants to
@@ -10127,6 +10222,7 @@ watch_command_1 (const char *arg, int accessflag, int from_tty,
     w.reset (new watchpoint (nullptr, bp_type));
 
   w->thread = thread;
+  w->inferior = inferior;
   w->task = task;
   w->disposition = disp_donttouch;
   w->pspace = current_program_space;
@@ -12026,7 +12122,8 @@ strace_marker_create_breakpoints_sal (struct gdbarch *gdbarch,
 				      enum bptype type_wanted,
 				      enum bpdisp disposition,
 				      int thread,
-				      int task, int ignore_count,
+				      int task, int inferior,
+				      int ignore_count,
 				      int from_tty, int enabled,
 				      int internal, unsigned flags)
 {
@@ -12052,7 +12149,8 @@ strace_marker_create_breakpoints_sal (struct gdbarch *gdbarch,
 			 std::move (cond_string),
 			 std::move (extra_string),
 			 disposition,
-			 thread, task, ignore_count,
+			 /* TODO: Inferior.  */
+			 thread, task, -1, ignore_count,
 			 from_tty, enabled, flags,
 			 canonical->special_display));
 
@@ -12685,10 +12783,11 @@ code_breakpoint::location_spec_to_sals (location_spec *locspec,
       if (condition_not_parsed && extra_string != NULL)
 	{
 	  gdb::unique_xmalloc_ptr<char> local_cond, local_extra;
-	  int local_thread, local_task;
+	  int local_thread, local_task, local_inferior;
 
 	  find_condition_and_thread_for_sals (sals, extra_string.get (),
 					      &local_cond, &local_thread,
+					      &local_inferior,
 					      &local_task, &local_extra);
 	  gdb_assert (cond_string == nullptr);
 	  if (local_cond != nullptr)
@@ -14806,4 +14905,6 @@ This is useful for formatted output in user-defined commands."));
 					   "breakpoint");
   gdb::observers::thread_exit.attach (remove_threaded_breakpoints,
 				      "breakpoint");
+  gdb::observers::inferior_exit.attach (remove_inferior_breakpoints,
+					"breakpoint");
 }
diff --git a/gdb/breakpoint.h b/gdb/breakpoint.h
index 360fa760577..f1df5128c0e 100644
--- a/gdb/breakpoint.h
+++ b/gdb/breakpoint.h
@@ -584,7 +584,7 @@ struct breakpoint_ops
 				  struct linespec_result *,
 				  gdb::unique_xmalloc_ptr<char>,
 				  gdb::unique_xmalloc_ptr<char>,
-				  enum bptype, enum bpdisp, int, int,
+				  enum bptype, enum bpdisp, int, int, int,
 				  int, int, int, int, unsigned);
 };
 
@@ -801,6 +801,9 @@ struct breakpoint
      care.  */
   int thread = -1;
 
+  /* ... */
+  int inferior = -1;
+
   /* Ada task number for task-specific breakpoint, or 0 if don't
      care.  */
   int task = 0;
@@ -856,7 +859,7 @@ struct code_breakpoint : public breakpoint
 		   gdb::unique_xmalloc_ptr<char> cond_string,
 		   gdb::unique_xmalloc_ptr<char> extra_string,
 		   enum bpdisp disposition,
-		   int thread, int task, int ignore_count,
+		   int thread, int task, int inferior, int ignore_count,
 		   int from_tty,
 		   int enabled, unsigned flags,
 		   int display_canonical);
diff --git a/gdb/dummy-frame.c b/gdb/dummy-frame.c
index 40b455c7e4a..5e48dca2982 100644
--- a/gdb/dummy-frame.c
+++ b/gdb/dummy-frame.c
@@ -132,6 +132,7 @@ pop_dummy_frame_bpt (struct breakpoint *b, struct dummy_frame *dummy)
   if (b->thread == dummy->id.thread->global_num
       && b->disposition == disp_del && frame_id_eq (b->frame_id, dummy->id.id))
     {
+      gdb_assert (b->inferior == -1);
       while (b->related_breakpoint != b)
 	delete_breakpoint (b->related_breakpoint);
 
diff --git a/gdb/elfread.c b/gdb/elfread.c
index 13dbaf0227a..0087b19b44b 100644
--- a/gdb/elfread.c
+++ b/gdb/elfread.c
@@ -947,7 +947,10 @@ elf_gnu_ifunc_resolver_stop (code_breakpoint *b)
       if (b_return->thread == thread_id
 	  && b_return->loc->requested_address == prev_pc
 	  && frame_id_eq (b_return->frame_id, prev_frame_id))
-	break;
+	{
+	  gdb_assert (b_return->inferior == -1);
+	  break;
+	}
     }
 
   if (b_return == b)
diff --git a/gdb/infcmd.c b/gdb/infcmd.c
index e9107cef520..68b1fca7c51 100644
--- a/gdb/infcmd.c
+++ b/gdb/infcmd.c
@@ -423,7 +423,10 @@ run_command_1 (const char *args, int from_tty, enum run_how run_how)
   /* Insert temporary breakpoint in main function if requested.  */
   if (run_how == RUN_STOP_AT_MAIN)
     {
-      std::string arg = string_printf ("-qualified %s if $_inferior == %d", main_name (),
+      //std::string arg = string_printf ("-qualified %s if $_inferior == %d", main_name (),
+      //			       current_inferior ()->num);
+      std::string arg = string_printf ("-qualified %s inferior %d",
+				       main_name (),
 				       current_inferior ()->num);
       tbreak_command (arg.c_str (), 0);
     }
diff --git a/gdb/infrun.c b/gdb/infrun.c
index 033699bc3f7..e23bbd8dfa1 100644
--- a/gdb/infrun.c
+++ b/gdb/infrun.c
@@ -8041,6 +8041,7 @@ insert_exception_resume_breakpoint (struct thread_info *tp,
 	  frame = NULL;
 
 	  bp->thread = tp->global_num;
+	  bp->inferior = -1;
 	  inferior_thread ()->control.exception_resume_breakpoint = bp;
 	}
     }
@@ -8074,6 +8075,7 @@ insert_exception_resume_from_probe (struct thread_info *tp,
   bp = set_momentary_breakpoint_at_pc (get_frame_arch (frame),
 				       handler, bp_exception_resume).release ();
   bp->thread = tp->global_num;
+  bp->inferior = -1;
   inferior_thread ()->control.exception_resume_breakpoint = bp;
 }
 
diff --git a/gdb/linespec.c b/gdb/linespec.c
index 3db35998f7e..ca09f630382 100644
--- a/gdb/linespec.c
+++ b/gdb/linespec.c
@@ -254,7 +254,7 @@ enum linespec_token_type
 
 /* List of keywords.  This is NULL-terminated so that it can be used
    as enum completer.  */
-const char * const linespec_keywords[] = { "if", "thread", "task", "-force-condition", NULL };
+const char * const linespec_keywords[] = { "if", "thread", "task", "inferior", "-force-condition", NULL };
 #define IF_KEYWORD_INDEX 0
 #define FORCE_KEYWORD_INDEX 3