From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by sourceware.org (Postfix) with ESMTPS id 55CCB3898502 for ; Thu, 30 Apr 2020 11:20:17 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 55CCB3898502 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=embecosm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=andrew.burgess@embecosm.com Received: by mail-wr1-x429.google.com with SMTP id x18so6416000wrq.2 for ; Thu, 30 Apr 2020 04:20:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=embecosm.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RJbMG/i7mA9UC/RKr7YukY3CP2fuvJKtnJwhcjVghGc=; b=Odw63xBRd9Uh4TBJv60emN1r6pkTagDpxLFgTSKul1wgi6JW2NjOxVYep+3FHcXyyu 4X4j0phKonJ0m+4o1BP1hVopJ/rViIFIbmP4Im30WNO/zRF+My52tsYY3JC+RQ+7Mn0m wus3V8hcse6Drx/jBB0vua0wDjf4RIz/KKb7DKhw+Ulua/6N+9Sn50LgW6DRSBhzOkgs Kwg20kDbVjfrzIIX748MHGWTpQtvyw/fFSMBanDVd6/DeGULiX7ZHiOPZWDAzLfUUQb5 1CUQCn+/U1bXnfojEShnDj1kPUIO7wTdOcjBGumupOB8a3xrfF/tsuaMGGw0uvQY8lqx 2Ptw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RJbMG/i7mA9UC/RKr7YukY3CP2fuvJKtnJwhcjVghGc=; b=KlPFsWCGdM44/YtTpl4se42KoLHnl59BExxv0Mg8i17sbsruzW8eaaKtRAirL11mmJ xHKMDRc69biBCM4xOlJqsWiz0/Aff9TB14j/QadgvltoJaojlUPEpcxjFDjsBH1yybq9 X5nW/5vPaTRi31lWQ8MKn0xzbG3yH/8cnY1sHC6Hs5NKaPUqz1/tEvUT2QKabbrlkQiN GC4xND6/qJfT8OLBiiklxQzbJzIDXDxbSJMyTKQQMXKHfk6HDX/kkOZEc0y1Zub/RyBv jFebwU7HQDdEOQwHbUzxQQ/iVocDFRsAfKyMlCVQpuqN8JpQPGADnOnK3LcJre8waJX6 sJ/w== X-Gm-Message-State: AGi0PuYATzzmi7QqWGJH67hbe+mePFZpEYBn98CcYxKx4naqCBLyqH8Z v9+pVGm11qChJ+CKnbRBhxNQVnX+BQA= X-Google-Smtp-Source: APiQypLBhBKb6Q719arDPkpy8uNDoGYA01JsdJoWxia531Z1kNZj0wZMobrCTmh0HwPFvwejhMK5PA== X-Received: by 2002:adf:ee4c:: with SMTP id w12mr3640439wro.347.1588245616042; Thu, 30 Apr 2020 04:20:16 -0700 (PDT) Received: from localhost (host81-151-181-184.range81-151.btcentralplus.com. [81.151.181.184]) by smtp.gmail.com with ESMTPSA id t16sm957947wrm.26.2020.04.30.04.20.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Apr 2020 04:20:15 -0700 (PDT) From: Andrew Burgess To: gdb-patches@sourceware.org Subject: [PATCHv3 1/3] gdb/testsuite: Detect and warn if paths are used in test names Date: Thu, 30 Apr 2020 12:20:09 +0100 Message-Id: X-Mailer: git-send-email 2.25.3 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-30.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Apr 2020 11:20:25 -0000 A new library is introduced that hooks into the core of Dejagnu and detects when a test's name includes either the source or build paths. If any offending test names are detected then Dejagnu will print a new result type, '# of paths in test names'. Users should treat this result type just like other bad results types, and aim not to increase this number. As well as displaying the total number of offending tests as part of the final results, a new marker is included in both the gdb.log and gdb.sum files, this marker starts with 'PATH: ', so an offending test would be expected to appear like this: PASS: gdb.base/sometest.exp: Loaded /path/to/build/testsuite/foo.exe PATH: gdb.base/sometest.exp: Loaded /path/to/build/testsuite/foo.exe This should make it easier to track down offending tests. Currently for a local run on my machine, I don't see any offending test names, but it is possible that different targets, or different configurations, might currently be breaking the no paths rule. In order to get this working I have needed to wrap two core Dejagnu functions, log_summary, and reset_vars. Relying on core functions that are not part of any API is always going to be risky, given the relatively slow rate of Dejagnu change this is probably OK for now, and we can possibly upstream some changes to Dejagnu that would allow this functionality to be supported in a more official way later on. Currently if the tests are run in parallel mode the new result type is not merged into the combined summary file so users will need to run in non-parallel mode to check this result. Similarly, the 'PATH: ' markers will not be merged into the combined summary file. A later commit will fix this. gdb/testsuite/ChangeLog: * lib/gdb.exp: Include check-test-names.exp library. * lib/check-test-names.exp: New file. --- gdb/testsuite/ChangeLog | 5 + gdb/testsuite/lib/check-test-names.exp | 143 +++++++++++++++++++++++++ gdb/testsuite/lib/gdb.exp | 1 + 3 files changed, 149 insertions(+) create mode 100644 gdb/testsuite/lib/check-test-names.exp diff --git a/gdb/testsuite/lib/check-test-names.exp b/gdb/testsuite/lib/check-test-names.exp new file mode 100644 index 00000000000..9a35ce7c5db --- /dev/null +++ b/gdb/testsuite/lib/check-test-names.exp @@ -0,0 +1,143 @@ +# Copyright 2020 Free Software Foundation, Inc. + +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# This library provides some protection against the introduction of +# tests that include either the source of build paths in the test +# name. When a test includes the path in its test name it is harder +# to compare results between two runs of GDB from different trees. + +namespace eval ::CheckTestNames { + # An associative array of counts of tests that include a path in their + # test name. There are two counts, 'count', which counts occurrences + # within a single variant run, and 'total', which counts across all + # variants. + variable counts + array set counts {} + foreach nm {paths} { + set counts($nm,count) 0 + set counts($nm,total) 0 + } + + # Increment the count, and total count for TYPE. + proc inc_count { type } { + variable counts + + incr counts($type,count) + incr counts($type,total) + } + + # Check if MESSAGE contains a build or source path, if it does increment + # the relevant counter and return true, otherwise, return false. + proc _check_paths { message } { + global srcdir objdir + + foreach path [list $srcdir $objdir] { + if { [ string first $path $message ] >= 0 } { + # Count each test just once. + inc_count paths + return true + } + } + + return false + } + + # Remove the leading Dejagnu status marker from MESSAGE, and + # return the remainder of MESSAGE. A status marker is something + # like 'PASS: '. It is assumed that MESSAGE does contain such a + # marker. If it doesn't then MESSAGE is returned unmodified. + proc _strip_status { message } { + # Find the position of the first ': ' string. + set pos [string first ": " $message] + if { $pos > -1 } { + # The '+ 2' is so we skip the ': ' we found above. + return [string range $message [expr $pos + 2] end] + } + + return $message + } + + # Check if MESSAGE contains either the source path or the build path. + # This will result in test names that can't easily be compared between + # different runs of GDB. + # + # Any offending test names cause the corresponding count to be + # incremented, and an extra message to be printed into the log + # file. + proc check { message } { + set message [ _strip_status $message ] + + if [ _check_paths $message ] { + clone_output "PATH: $message" + } + } + + # If COUNT is greater than zero, disply PREFIX followed by COUNT. + proc maybe_show_count { prefix count } { + if { $count > 0 } { + clone_output "$prefix$count" + } + } + + # Rename Dejagnu's log_summary procedure, and create do_log_summary to + # replace it. We arrange to have do_log_summary called later. + rename ::log_summary log_summary + proc do_log_summary { args } { + variable counts + + # If ARGS is the empty list then we don't want to pass a single + # empty string as a parameter here. + eval "CheckTestNames::log_summary $args" + + if { [llength $args] == 0 } { + set which "count" + } else { + set which [lindex $args 0] + } + + maybe_show_count "# of paths in test names\t" \ + $counts(paths,$which) + } + + # Rename Dejagnu's reset_vars procedure, and create do_reset_vars to + # replace it. We arrange to have do_reset_vars called later. + rename ::reset_vars reset_vars + proc do_reset_vars {} { + variable counts + + CheckTestNames::reset_vars + + foreach nm {paths} { + set counts($nm,count) 0 + } + } +} + +# Arrange for Dejagnu to call CheckTestNames::check for each test result. +foreach nm {pass fail xfail kfail xpass kpass unresolved untested \ + unsupported} { + set local_record_procs($nm) "CheckTestNames::check" +} + +# Create new global log_summary to replace Dejagnu's. +proc log_summary { args } { + eval "CheckTestNames::do_log_summary $args" +} + +# Create new global reset_vars to replace Dejagnu's. +proc reset_vars {} { + eval "CheckTestNames::do_reset_vars" +} + diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp index b72ce0cda7f..07e4ed83bc0 100644 --- a/gdb/testsuite/lib/gdb.exp +++ b/gdb/testsuite/lib/gdb.exp @@ -29,6 +29,7 @@ load_lib libgloss.exp load_lib cache.exp load_lib gdb-utils.exp load_lib memory.exp +load_lib check-test-names.exp global GDB -- 2.25.3