From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 30656 invoked by alias); 9 Sep 2005 21:28:34 -0000 Mailing-List: contact gdb-patches-help@sources.redhat.com; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sources.redhat.com Received: (qmail 30639 invoked by uid 22791); 9 Sep 2005 21:28:29 -0000 Received: from nevyn.them.org (HELO nevyn.them.org) (66.93.172.17) by sourceware.org (qpsmtpd/0.30-dev) with ESMTP; Fri, 09 Sep 2005 21:28:29 +0000 Received: from drow by nevyn.them.org with local (Exim 4.52) id 1EDqQ6-0002c0-Nq; Fri, 09 Sep 2005 17:28:26 -0400 Date: Fri, 09 Sep 2005 21:28:00 -0000 From: Daniel Jacobowitz To: Mark Kettenis , gdb-patches@sourceware.org Subject: [RFC] Tighten and tweak ptrace argument checks Message-ID: <20050909212825.GA9449@nevyn.them.org> Mail-Followup-To: Mark Kettenis , gdb-patches@sourceware.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.8i X-SW-Source: 2005-09/txt/msg00059.txt.bz2 Hi Mark, I discovered recently that your rewrite of the ptrace argument type checks had busted some (not-yet-contributed, sorry) changes to use long long for ptrace calls on MIPS n32. This made me take a closer look at the tests. It turns out that neither the return type checks nor argument type checks work on GNU/Linux. The return type check will work with oldish versions of GCC, but not with 4.0 (or 3.4, I think). The argument type checks won't work at all. Here's the problem: extern long int ptrace (enum __ptrace_request, ...); () won't match ...; this is important because of different calling conventions on some platforms. On i386-pc-linux-gnu the defaults work so this is merely an annoyance. When the real type is long long instead of long, well, it wasn't pretty... So here's a proposed patch. It handles the GNU/Linux case. It handles long long. It also is violently fatal if it fails, instead of making up something sensible - this is because I wasted several days trying to figure out what was wrong with GDB when it was casting all the ptrace arguments to the wrong type. I'm sure it'll break the build in a lot of places but I think it's worth fixing if you want to use autoconf for this at all! Any comments on the patch? I also noticed considerable PTRACE_ARG3_TYPE vs PTRACE_TYPE_ARG3 confusion in the sources. I think that the last time I looked at this, I convinced myself that it was possible to get them out of sync, and the results would be pretty gruesome. I didn't touch that for now. -- Daniel Jacobowitz CodeSourcery, LLC 2005-09-09 Daniel Jacobowitz * configure.ac: Improve tests for the return type and argument list of ptrace. Exit if the tests fail. * configure: Regenerated. Index: configure.ac =================================================================== RCS file: /big/fsf/rsync/src/src/gdb/configure.ac,v retrieving revision 1.24 diff -u -p -r1.24 configure.ac --- configure.ac 25 Jul 2005 15:08:40 -0000 1.24 +++ configure.ac 9 Sep 2005 21:20:01 -0000 @@ -486,12 +486,28 @@ AC_CHECK_DECLS(ptrace, [], [ : ${gdb_cv_func_ptrace_ret='int'} : ${gdb_cv_func_ptrace_args='int,int,long,long'} ], $gdb_ptrace_headers) -# Check return type. -AC_CACHE_CHECK([return type of ptrace], gdb_cv_func_ptrace_ret, - AC_TRY_COMPILE($gdb_ptrace_headers, - [extern int ptrace ();], - gdb_cv_func_ptrace_ret='int', - gdb_cv_func_ptrace_ret='long')) +# Check return type. First try without an argument list. But +# any argument list containing stdargs is incompatible with a missing +# argument list, and recent (e.g. 4.x) versions of gcc will reject +# them. So list every known stdargs argument list too. +AC_CACHE_CHECK([return type of ptrace], gdb_cv_func_ptrace_ret, [ +for gdb_args in '' 'unsigned int, ...'; do + for gdb_ret in 'int' 'long' 'long long'; do + AC_TRY_COMPILE($gdb_ptrace_headers, + [extern $gdb_ret ptrace ($gdb_args);], + [gdb_cv_func_ptrace_ret='int'; break 2]) + done +done +if test "${gdb_cv_func_ptrace_ret+set}" != set; then + AC_MSG_ERROR(unknown) +fi +]) + +# Currently the only systems known to use stdargs expect four arguments. +if test x"$gdb_args" = x'unsigned int, ...'; then + gdb_cv_func_ptrace_args="unsigned int,int,$gdb_ret,$gdb_ret" +fi + AC_DEFINE_UNQUOTED(PTRACE_TYPE_RET, $gdb_cv_func_ptrace_ret, [Define as the return type of ptrace.]) # Check argument types. @@ -517,8 +533,9 @@ gdb_cv_func_ptrace_args="$gdb_arg1,$gdb_ done done done -# Provide a safe default value. -: ${gdb_cv_func_ptrace_args='int,int,long,long'} +if test "${gdb_cv_func_ptrace_args+set}" != set; then + AC_MSG_ERROR(unknown) +fi ]) ac_save_IFS=$IFS; IFS=',' set dummy `echo "$gdb_cv_func_ptrace_args" | sed 's/\*/\*/g'`