Re: [RFC] Apply compilation dir to source_path lookup

[Andrew Burgess <andrew.burgess@embecosm.com>]

* Mike Gulick <mgulick@mathworks.com> [2019-09-09 22:41:02 +0000]:

> On 9/7/19 7:50 PM, Andrew Burgess wrote:
> > * Mike Gulick <mgulick@mathworks.com> [2019-09-05 22:40:48 +0000]:
>  
> [snip]
> 
> >> Here is a preliminary patch for the first approach.  I will go ahead and
> >> add a test-case and re-send a formal patch review request if this seems
> >> like an acceptable solution.
> > 
> > Do you have a copyright assignment in place?  Any patch over a couple
> > of trivial lines requires an assignment.  Others might correct me, but
> > this patch would seem to be big enough to me.
> > 
> 
> Yes, I do already have a copyright assignment in place.
> 
> >>
> >> Thanks,
> >> Mike
> >>
> >> diff --git a/gdb/source.c b/gdb/source.c
> >> index b27f210802..b7b5741028 100644
> >> --- a/gdb/source.c
> >> +++ b/gdb/source.c
> >> @@ -1033,7 +1033,35 @@ find_and_open_source (const char *filename,
> >>    openp_flags flags = OPF_SEARCH_IN_PATH;
> >>    if (basenames_may_differ)
> >>      flags |= OPF_RETURN_REALPATH;
> >> +
> >> +  /* Try to locate file using filename */
> > 
> > Comments need to end with '.', and there should always be two spaces
> > after a '.' in comments.
> > 
> >>    result = openp (path, flags, filename, OPEN_MODE, fullname);
> >> +  if (result < 0 && dirname != NULL)
> >> +    {
> >> +      /* Try to locate file using compilation dir + filename.  This is helpful
> >> +	 if part of the compilation directory was removed, e.g. using gcc's
> >> +	 -fdebug-prefix-map, and we have added the missing prefix to
> >> +	 source_path. */
> >> +      /* Allocate space for dirname, possibly an extra slash, the filename,
> >> +	 and terminating null */
> > 
> > You need two spaces at the end of your comments, and can you fold
> > these two comments into one.
> > 
> >> +      char * cdir_filename = (char *)
> >> +	alloca (strlen (dirname) + strlen(SLASH_STRING) + strlen (filename) + 1);
> > 
> > Whitespace between function name and opening parenthesis.
> > 
> >> +
> >> +      cdir_filename = strcpy (cdir_filename, dirname);
> >> +      int len = strlen (cdir_filename);    /* last char in cdir_filename */
> > 
> > Comments should be formatted as sentences, so capital 'L' here, and
> > '.  ' and the end.
> > 
> >> +
> >> +      /* Add directory separator if not provided by dirname or filename */
> >> +      if (!(IS_DIR_SEPARATOR (cdir_filename[len]) ||
> >> +	    IS_DIR_SEPARATOR (filename[0])))
> > 
> > Break the line before the '||'.
> > 
> >> +        strcat (cdir_filename, SLASH_STRING);
> >> +      else if (IS_DIR_SEPARATOR (cdir_filename[len]) &&
> >> +	       IS_DIR_SEPARATOR (filename[0]))
> > 
> > Same, line break before '&&'.
> > 
> >> +	/* Both provide a slash, use only one */
> > 
> > Comment formatting.
> > 
> >> +	cdir_filename[len] = '\0';
> >> +      strcat(cdir_filename,filename);
> > 
> > Whitespace before opening parenthesis.
> > 
> >> +
> >> +      result = openp(path, flags, cdir_filename, OPEN_MODE, fullname);
> > 
> > And again.
> > 
> >> +    }
> >>    if (result < 0)
> >>      {
> >>        /* Didn't work.  Try using just the basename.  */
> > 
> > You'll also need to write a short ChangeLog entry for the patch.  I
> > also wonder if the function could be simplified by using std::string
> > to build the new string rather than all of the strlen/strcpy/strcat
> > calls?  Though I don't think that would be a requirement for
> > acceptance.
> > 
> 
> Thanks for the feedback.  I've rewritten the patch to use std::string
> and address the formatting issues.  Using std::string does seem nicer
> since it avoids the tricky alloca calls, at the cost of maybe being a
> little more expensive.  It is also a little clearer to read.
> 
> > You might also consider extending the documentation relating to source
> > path searching to better describe the process - as you discuss above,
> > the existing documentation is not that clear on the subject.
> >
> 
> I'll take a look at this and reply with a separate patch.
> 
> > Finally, you'd probably want to write a test.... however, I was
> > wondering how a test for this might be written, and ended up writing
> > one myself.  Feel free to use this as a starting point, or just
> > include this as is if it helps.
> >
> 
> Thanks for writing a test.  I recall this being the most challenging
> part of my last GDB contribution, so this saved me a ton of time.  Your
> test seems to test the scenario I had in mind.  Please use your patch
> as-is.
> 
> > Thanks,
> > Andrew
> > 
> 
> Here is the new version of my patch.  Please let me know if you have any
> additional feedback!

I think the new version is great.  I've taken the liberty of merging
your change and my test case into one patch, and also writing an
update to the docs to (hopefully) better explain the source finding
algorithm.

You code is maintained unchanged in this patch except for one very
minor nit that I've pointed out below.

We can just wait for a doc review, and then I think this can be
merged.

Thanks,
Andrew

> 
> -Mike
> 
> ---
> 
> From 5f3957aceb6a9390b412c310f626d7ab4fa80b62 Mon Sep 17 00:00:00 2001
> From: Mike Gulick <mgulick@mathworks.com>
> Date: Mon, 9 Sep 2019 18:36:23 -0400
> Subject: [PATCH] gdb: Look for compilation directory relative to directory
>  search path
> 
> The 'directory' command allows the user to provide a list of filesystem
> directories in which to search for source code.  The directories in this
> search path are used as the base directory for the source filename from
> the debug information (DW_AT_name).  Thus the directory search path
> provides alternatives to the existing compilation directory from the
> debug information (DW_AT_comp_dir).  Generally speaking, DW_AT_name
> stores the filename argument passed to the compiler (including any
> directory components), and DW_AT_comp_dir stores the current working
> directory from which the compiler was executed.  For example:
> 
>     $ cd /path/to/project/subdir1
>     $ gcc -c a/test.c -g
> 
> The corresponding debug information will look like this:
> 
>     DW_AT_name      : a/test.c
>     DW_AT_comp_dir  : /path/to/project/subdir1
> 
> When compiling with the -fdebug-prefix-map GCC option, the compilation
> directory can be arbitrarily rewritten.  In the above example, we may
> rewrite the compilation directory as follows:
> 
>     $ gcc -c a/test.c -g -fdebug-prefix-map=/path/to/project=
> 
> In this case, the corresponding debug information will look like:
> 
>     DW_AT_name      : a/test.c
>     DW_AT_comp_dir  : /subdir1
> 
> This prevents GDB from finding the corresponding source code based on
> the debug information alone.  In some cases, a substitute-path command
> can be used to re-map a consistent prefix in the rewritten compilation
> directory to the real filesystem path.  However, there may not be a
> consistent prefix remaining in the debug symbols (for example in a
> project that has source code in many subdirectories under the project's
> root), thereby requiring multiple substitute-path rules.  In this case,
> it is easier to add the missing prefix to the directory search path via
> the 'directory' command.
> 
> The function find_and_open_source currently searches in:
> 
>     SEARCH_PATH/FILENAME
> 
> where SEARCH_PATH corresponds to each individual entry in the directory
> search path (which is guaranteed to contain the compilation directory
> from the debug information, as well as the current working directory).
> FILENAME corresponds to the source filename (DW_AT_name), which may have
> directory components in it.  In addition, GDB searches in:
> 
>     SEARCH_PATH/FILE_BASENAME
> 
> where FILE_BASENAME is the basename of the DW_AT_name entry.
> 
> This change modifies find_and_open_source to additionally search in:
> 
>     SEARCH_PATH/COMP_DIR/FILENAME
> 
> where COMP_DIR is the compilation directory from the debug symbols.  In
> the example given earlier, running:
> 
>     (gdb) directory /path/to/project
> 
> will now allow GDB to correctly locate the source code from the debug
> information.
> 
> gdb/ChangeLog:
> 
> 	* source.c (find_and_open_source): Search for the compilation
> 	directory and source file as a relative path beneath the directory
> 	search path.
> ---
>  gdb/ChangeLog |  6 ++++++
>  gdb/source.c  | 24 ++++++++++++++++++++++++
>  2 files changed, 30 insertions(+)
> 
> diff --git a/gdb/ChangeLog b/gdb/ChangeLog
> index eb6d062812..25e3fc8e5d 100644
> --- a/gdb/ChangeLog
> +++ b/gdb/ChangeLog
> @@ -1,3 +1,9 @@
> +2019-09-09  Mike Gulick  <mgulick@mathworks.com>
> +
> +	* source.c (find_and_open_source): Search for the compilation
> +	directory and source file as a relative path beneath the directory
> +	search path.
> +
>  2019-09-09  Philippe Waroquiers  <philippe.waroquiers@skynet.be>
>  
>  	* python/python.c (do_start_initialization): Make progname_copy static,
> diff --git a/gdb/source.c b/gdb/source.c
> index b27f210802..1635563b20 100644
> --- a/gdb/source.c
> +++ b/gdb/source.c
> @@ -1033,7 +1033,31 @@ find_and_open_source (const char *filename,
>    openp_flags flags = OPF_SEARCH_IN_PATH;
>    if (basenames_may_differ)
>      flags |= OPF_RETURN_REALPATH;
> +
> +  /* Try to locate file using filename.  */
>    result = openp (path, flags, filename, OPEN_MODE, fullname);
> +  if (result < 0 && dirname != NULL)
> +    {
> +      /* Try to locate file using compilation dir + filename.  This is helpful
> +	 if part of the compilation directory was removed, e.g. using gcc's
> +	 -fdebug-prefix-map, and we have added the missing prefix to
> +	 source_path.  */
> +      std::string cdir_filename (dirname);
> +
> +      /* Remove any trailing directory separators.  */
> +      while (IS_DIR_SEPARATOR (cdir_filename.back ()))
> +	cdir_filename.pop_back ();
> +      /* Add our own directory separator.  */
> +      cdir_filename.append (SLASH_STRING);
> +      /* Append filename, without any leading directory separators.  */
> +      const char * filename_start = filename;

The space after the '*' is not needed.

> +      while (IS_DIR_SEPARATOR (filename_start[0]))
> +	filename_start++;
> +      cdir_filename.append (filename_start);
> +
> +      result = openp (path, flags, cdir_filename.c_str (), OPEN_MODE,
> +		      fullname);
> +    }
>    if (result < 0)
>      {
>        /* Didn't work.  Try using just the basename.  */
> -- 
> 2.20.1
> 


----

commit 3fcaf369356f2df4ad8d5e9922f818516bb62887
Author: Mike Gulick <mgulick@mathworks.com>
Date:   Thu Sep 12 11:16:06 2019 -0400

    gdb: Look for compilation directory relative to directory search path
    
    The 'directory' command allows the user to provide a list of filesystem
    directories in which to search for source code.  The directories in this
    search path are used as the base directory for the source filename from
    the debug information (DW_AT_name).  Thus the directory search path
    provides alternatives to the existing compilation directory from the
    debug information (DW_AT_comp_dir).  Generally speaking, DW_AT_name
    stores the filename argument passed to the compiler (including any
    directory components), and DW_AT_comp_dir stores the current working
    directory from which the compiler was executed.  For example:
    
        $ cd /path/to/project/subdir1
        $ gcc -c a/test.c -g
    
    The corresponding debug information will look like this:
    
        DW_AT_name      : a/test.c
        DW_AT_comp_dir  : /path/to/project/subdir1
    
    When compiling with the -fdebug-prefix-map GCC option, the compilation
    directory can be arbitrarily rewritten.  In the above example, we may
    rewrite the compilation directory as follows:
    
        $ gcc -c a/test.c -g -fdebug-prefix-map=/path/to/project=
    
    In this case, the corresponding debug information will look like:
    
        DW_AT_name      : a/test.c
        DW_AT_comp_dir  : /subdir1
    
    This prevents GDB from finding the corresponding source code based on
    the debug information alone.  In some cases, a substitute-path command
    can be used to re-map a consistent prefix in the rewritten compilation
    directory to the real filesystem path.  However, there may not be a
    consistent prefix remaining in the debug symbols (for example in a
    project that has source code in many subdirectories under the project's
    root), thereby requiring multiple substitute-path rules.  In this case,
    it is easier to add the missing prefix to the directory search path via
    the 'directory' command.
    
    The function find_and_open_source currently searches in:
    
        SEARCH_PATH/FILENAME
    
    where SEARCH_PATH corresponds to each individual entry in the directory
    search path (which is guaranteed to contain the compilation directory
    from the debug information, as well as the current working directory).
    FILENAME corresponds to the source filename (DW_AT_name), which may have
    directory components in it.  In addition, GDB searches in:
    
        SEARCH_PATH/FILE_BASENAME
    
    where FILE_BASENAME is the basename of the DW_AT_name entry.
    
    This change modifies find_and_open_source to additionally search in:
    
        SEARCH_PATH/COMP_DIR/FILENAME
    
    where COMP_DIR is the compilation directory from the debug symbols.  In
    the example given earlier, running:
    
        (gdb) directory /path/to/project
    
    will now allow GDB to correctly locate the source code from the debug
    information.
    
    gdb/ChangeLog:
    
            * source.c (find_and_open_source): Search for the compilation
            directory and source file as a relative path beneath the directory
            search path.
    
    gdb/doc/ChangeLog:
    
            * gdb.texinfo (Source Path): Additional text to better describe
            how the source path directory list is used when searching for
            source files.
    
    gdb/testsuite/ChangeLog:
    
            * gdb.base/source-dir.c: New file.
            * gdb.base/source-dir.exp: Add extra test for mapped compilation
            directory.

diff --git a/gdb/doc/gdb.texinfo b/gdb/doc/gdb.texinfo
index 79824a0226a..b6b87b28b9f 100644
--- a/gdb/doc/gdb.texinfo
+++ b/gdb/doc/gdb.texinfo
@@ -8968,9 +8968,23 @@
 Plain file names, relative file names with leading directories, file
 names containing dots, etc.@: are all treated as described above; for
 instance, if the source path is @file{/mnt/cross}, and the source file
-is recorded as @file{../lib/foo.c}, @value{GDBN} would first try
-@file{../lib/foo.c}, then @file{/mnt/cross/../lib/foo.c}, and after
-that---@file{/mnt/cross/foo.c}.
+is recorded as @file{../lib/foo.c} and no compilation directory is
+recorded, @value{GDBN} would first try @file{../lib/foo.c}, then
+@file{/mnt/cross/../lib/foo.c}, and after that
+@file{/mnt/cross/foo.c}.
+
+When there is both a filename and a compilation directory in the debug
+information, and if @value{GDBN} hasn't found the source file using
+the above approach, then @value{GDBN} will append the filename to the
+compilation directory and retry using the above steps; for instance,
+if the source path is @file{/mnt/cross}, and the source file is
+recorded as @file{../lib/foo.c} and the compilation directory is
+recorded as @file{/usr/src/foo-1.0/build}, @value{GDBN} would first
+try @file{../lib/foo.c}, then @file{/mnt/cross/../lib/foo.c}, then
+@file{/mnt/cross/foo.c}, using the approach outlined above.  If the
+source file still hasn't been found then @value{GDBN} will next check
+@file{/usr/src/foo-1.0/build/../lib/foo.c}, then
+@file{/mnt/cross/usr/src/foo-1.0/build/../lib/foo.c}.
 
 Note that the executable search path is @emph{not} used to locate the
 source files.
@@ -9074,6 +9088,12 @@
 session, while the latter is immediately expanded to the current
 directory at the time you add an entry to the source path.
 
+The @samp{$cdir} in the source path causes @value{GDBN} to search for
+the filename in the compilation directory, if one is recorded in the
+debug information.  @value{GDBN} will also append the compilation
+directory to the filename and check this against all other entries in
+the source path.
+
 @item directory
 Reset the source path to its default value (@samp{$cdir:$cwd} on Unix systems).  This requires confirmation.
 
diff --git a/gdb/source.c b/gdb/source.c
index b27f210802c..14746d4cd05 100644
--- a/gdb/source.c
+++ b/gdb/source.c
@@ -1033,7 +1033,31 @@ find_and_open_source (const char *filename,
   openp_flags flags = OPF_SEARCH_IN_PATH;
   if (basenames_may_differ)
     flags |= OPF_RETURN_REALPATH;
+
+  /* Try to locate file using filename.  */
   result = openp (path, flags, filename, OPEN_MODE, fullname);
+  if (result < 0 && dirname != NULL)
+    {
+      /* Try to locate file using compilation dir + filename.  This is helpful
+	 if part of the compilation directory was removed, e.g. using gcc's
+	 -fdebug-prefix-map, and we have added the missing prefix to
+	 source_path.  */
+      std::string cdir_filename (dirname);
+
+      /* Remove any trailing directory separators.  */
+      while (IS_DIR_SEPARATOR (cdir_filename.back ()))
+	cdir_filename.pop_back ();
+      /* Add our own directory separator.  */
+      cdir_filename.append (SLASH_STRING);
+      /* Append filename, without any leading directory separators.  */
+      const char *filename_start = filename;
+      while (IS_DIR_SEPARATOR (filename_start[0]))
+	filename_start++;
+      cdir_filename.append (filename_start);
+
+      result = openp (path, flags, cdir_filename.c_str (), OPEN_MODE,
+		      fullname);
+    }
   if (result < 0)
     {
       /* Didn't work.  Try using just the basename.  */
diff --git a/gdb/testsuite/gdb.base/source-dir.c b/gdb/testsuite/gdb.base/source-dir.c
new file mode 100644
index 00000000000..d94b8074ec0
--- /dev/null
+++ b/gdb/testsuite/gdb.base/source-dir.c
@@ -0,0 +1,22 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright 2019 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+int
+main ()
+{
+  return 0;
+}
diff --git a/gdb/testsuite/gdb.base/source-dir.exp b/gdb/testsuite/gdb.base/source-dir.exp
index 048c0e95161..6fe889b0cdb 100644
--- a/gdb/testsuite/gdb.base/source-dir.exp
+++ b/gdb/testsuite/gdb.base/source-dir.exp
@@ -15,9 +15,132 @@
 
 standard_testfile
 
-gdb_start
+# Take a list of directories DIRS, and return a regular expression
+# that will match against the output of the 'directory' command
+# assuming that DIRS are all of the directories that should appear in
+# the results.
+proc search_dir_list { dirs } {
+    set output "\r\nSource directories searched: "
+    append output [join $dirs "\[:;\]"]
 
-set foo "/nOtExStInG"
+    return ${output}
+}
 
-gdb_test "directory $foo/a $foo/b $foo/c" "\r\nSource directories searched: $foo/a\[:;\]$foo/b\[:;\]$foo/c\[:;\]\\\$cdir\[:;\]\\\$cwd"
-gdb_test "directory $foo/b $foo/d $foo/c" "\r\nSource directories searched: $foo/b\[:;\]$foo/d\[:;\]$foo/c\[:;\]$foo/a\[:;\]\\\$cdir\[:;\]\\\$cwd"
+# Check that adding directories to the search path changes the order
+# in which directories are searched.
+proc test_changing_search_directory {} {
+    gdb_start
+
+    set foo "/nOtExStInG"
+
+    gdb_test "directory $foo/a $foo/b $foo/c" \
+	[search_dir_list [list \
+			      "$foo/a" \
+			      "$foo/b" \
+			      "$foo/c" \
+			      "\\\$cdir" \
+			      "\\\$cwd"]]
+    gdb_test "directory $foo/b $foo/d $foo/c" \
+	[search_dir_list [list \
+			      "$foo/b" \
+			      "$foo/d" \
+			      "$foo/c" \
+			      "$foo/a" \
+			      "\\\$cdir" \
+			      "\\\$cwd"]]
+    gdb_exit
+}
+
+# Test that the compilation directory can also be extended with a
+# prefix from the directory search path in order to find source files.
+proc test_truncated_comp_dir {} {
+    global srcfile srcdir subdir binfile
+
+    # When we run this test the current directory will be something
+    # like this:
+    #     /some/path/to/gdb/build/testsuite/
+    # We are going to copy the source file out of the source tree into
+    # a location like this:
+    #     /some/path/to/gdb/build/testsuite/output/gdb.base/soure-dir/
+    #
+    # We will then switch to this directory and compile the source
+    # file, however, we will ask GCC to remove this prefix from the
+    # compilation directory in the debug info:
+    #     /some/path/to/gdb/build/testsuite/output/
+    #
+    # As a result the debug information will look like this:
+    #
+    #     DW_AT_name        : source-dir.c
+    #     DW_AT_comp_dir    : /gdb.base/source-dir
+    #
+    # Finally we switch back to this directory:
+    #     /some/path/to/gdb/build/testsuite/
+    #
+    # and start GDB.  There was a time when GDB would be unable to
+    # find the source file no matter what we added to the directory
+    # search path, this should now be fixed.
+
+    set original_dir [pwd]
+    set working_dir [standard_output_file ""]
+    cd ${working_dir}
+
+    set strip_dir [file normalize "${working_dir}/../.."]
+
+    set new_srcfile [standard_output_file ${srcfile}]
+    remote_exec host "cp ${srcdir}/${subdir}/${srcfile} ${new_srcfile}"
+
+    set options \
+	"debug additional_flags=-fdebug-prefix-map=${strip_dir}="
+    if  { [gdb_compile "${srcfile}" "${binfile}" \
+	       executable ${options}] != "" } {
+	untested "failed to compile"
+	return -1
+    }
+
+    cd ${original_dir}
+
+    clean_restart ${binfile}
+
+    gdb_test_no_output "set directories \$cdir:\$cwd"
+    gdb_test "show directories" \
+	"\r\nSource directories searched: \\\$cdir\[:;\]\\\$cwd"
+
+    if ![runto_main] then {
+	fail "can't run to main"
+	return 0
+    }
+
+    gdb_test "info source" \
+	[multi_line \
+	     "Current source file is ${srcfile}" \
+	     "Compilation directory is \[^\n\r\]+" \
+	     "${srcfile}: No such file or directory."]
+
+    gdb_test "dir $strip_dir" \
+	[search_dir_list [list \
+			      "$strip_dir" \
+			      "\\\$cdir" \
+			      "\\\$cwd"]]
+    gdb_test "list" [multi_line \
+			 "16\[^\n\r\]+" \
+			 "17\[ \t\]+" \
+			 "18\[ \t\]+int" \
+			 "19\[ \t\]+main \\(\\)" \
+			 "20\[ \t\]+\\{" \
+			 "21\[ \t\]+return 0;" \
+			 "22\[ \t\]+\\}" ]
+
+    gdb_test "info source" \
+	[multi_line \
+	     "Current source file is ${srcfile}" \
+	     "Compilation directory is \[^\n\r\]+" \
+	     "Located in ${new_srcfile}" \
+	     "Contains 22 lines." \
+	     "Source language is c." \
+	     "Producer is \[^\n\r\]+" \
+	     "\[^\n\r\]+" \
+	     "\[^\n\r\]+" ]
+}
+
+test_changing_search_directory
+test_truncated_comp_dir