From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) by sourceware.org (Postfix) with ESMTP id 7A554386F430 for ; Tue, 26 May 2020 10:39:39 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 7A554386F430 Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-376-rmV_EoN7PTixiAWGsmahAA-1; Tue, 26 May 2020 06:39:37 -0400 X-MC-Unique: rmV_EoN7PTixiAWGsmahAA-1 Received: by mail-wr1-f70.google.com with SMTP id w16so1428969wru.18 for ; Tue, 26 May 2020 03:39:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=8yrE+edPmbpp6thaMKvD6zeUF5wfZOmfx2GWylIfh/s=; b=Y1CsUi1QLg+mmAgpr8N4ZXDXA5cArv6u5qnxqyJaJTk8sVrw9YqprRq7IL4aHw/hrx MHNIS4HFWTpN9NdtwoKy3txRs0EAtpT1KshEFcaj3mq5TPxMF6AtWVAGuYGh8b2UPyUu Pg5n7rQv/I13SiUINhV142fcrhgEvITkwmrdQon/FWK8YhuUsepLCelXJ2ICjeDKYlVu w20mX+Flxl/qxC821BNqwa1pA7/SjZ01WMHys/V52ULl433htytsJklIzNhdzbFKYZrX rhMtOgZNSMs9Dxxhza8GJ+RdDsqq5u2jXFofmUBROpq6WFOBVD1ZpY83B7PzcCylkq9T bZDw== X-Gm-Message-State: AOAM532wIIrVrwTccVaz3U+XFoacGAYD11M8ABLoDYSz0238Q16JNLeX OcXCuTtv2NWbS17qrUzSikAv7yyeG+5T1N6/7XWSVHT362ynur6JA9yvGTSXfpBfd8OnxiY27Us x3l2v6qBa72mgPtGt9JW8rw== X-Received: by 2002:adf:ea81:: with SMTP id s1mr18771769wrm.424.1590489576166; Tue, 26 May 2020 03:39:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyKaLBRBYiPb+W76f0MJUlRxgoO5gxGN6xpd5vIGnaJWKBn4+u58b1TGb9hhtM5+9t3Aqfidg== X-Received: by 2002:adf:ea81:: with SMTP id s1mr18771760wrm.424.1590489575954; Tue, 26 May 2020 03:39:35 -0700 (PDT) Received: from ?IPv6:2001:8a0:f909:7b00:2327:23ca:3e56:ef5f? ([2001:8a0:f909:7b00:2327:23ca:3e56:ef5f]) by smtp.gmail.com with ESMTPSA id n17sm20668090wrr.42.2020.05.26.03.39.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 26 May 2020 03:39:35 -0700 (PDT) Subject: Re: How to run multiple target boards To: Tom de Vries , Simon Marchi , "gdb-patches@sourceware.org" References: <20200512130002.GA3290@delia> <73a5e27b-a709-8861-1f88-a07a0a2a71f9@polymtl.ca> <256a0db7-98c2-7769-7991-981f59106ee4@suse.de> <82395a9a-1595-e16c-8977-5f929f9041cc@polymtl.ca> <85005cf2-783d-7048-1cfa-8b648854c981@suse.de> From: Pedro Alves Message-ID: <3319bd1f-a39f-1bf0-f291-e166edb2ab97@redhat.com> Date: Tue, 26 May 2020 11:39:34 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <85005cf2-783d-7048-1cfa-8b648854c981@suse.de> Content-Language: en-US X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-4.8 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 May 2020 10:39:41 -0000 On 5/26/20 9:49 AM, Tom de Vries wrote: > [ was: Re: [committed][gdb/testsuite] Fix incorrect string concat in > jit-elf.exp ] > > On 13-05-2020 15:45, Simon Marchi wrote: >> FWIW, I just learned it's possible to pass a list of target boards to --target_board: >> >> --target_board='unix native-gdbserver native-extended-gdbserver' >> >> And it gives a combined result at the end. So I'll make myself an alias to test on all >> the commonly used boards, probably: >> >> - cc-with-debug-names.exp >> - cc-with-dwz.exp >> - cc-with-dwz-m.exp >> - cc-with-gdb-index.exp >> - debug-types.exp >> - dwarf4-gdb-index.exp >> - fission-dwp.exp >> - fission.exp >> - native-extended-gdbserver.exp >> - native-gdbserver.exp >> - native-stdio-gdbserver.exp >> - readnow.exp What are the advantages of doing this compared to a script that runs the testsuite for each of the boards, separately? >> >> When working on a single test, it's usually not to long to run them all. Of course, it's >> not really practical to run the complete testsuite twice (before and after) for each board >> for every change we do... > > Hi Simon, > > I just tried this out. > > My test scripts use make check, but I didn't manage to make that work > yet, so I tried out using runtest directly (and I may be missing > something obvious, given that I haven't used this before): > ... > $ cd build/gdb/testsuite > $ runtest gdb.base/gold-gdb-index.exp --target_board='cc-with-gdb-index > unix' > ... One issue this with that I've run into in the past, is with global variables leaking from one board to the other. And I've just tried a quick test, and lucky me I seem to have run into something like that immediately: $ make check RUNTESTFLAGS="--target_board='native-gdbserver unix' break.exp" ... Schedule of variations: native-gdbserver unix ... FAIL: gdb.base/break.exp: run until function breakpoint FAIL: gdb.base/break.exp: run until breakpoint set at a line number (the program is no longer running) FAIL: gdb.base/break.exp: run until file:function(6) breakpoint (the program is no longer running) FAIL: gdb.base/break.exp: run until file:function(5) breakpoint (the program is no longer running) (snip a bunch more) (Running break.exp for each board in isolation passes cleanly, of course.) Another thing is that the test messages in gdb.sum don't indicate which board issued the PASS/FAIL (on each result), making it more difficult to analyze. > > It seems however that the global settings (CC_FOR_TARGET etc) from the > first board stay active in the unix board: Yeah, things like that. Also, the tests for all boards end up under the same testsuite/outputs/ directory, so whatever the last board was, wins. I'm not even sure how the testsuite caching works when you run more than one board in parallel mode. It would seem to me that if you're looking at running tests against different boards, that you would want the output directories to be separate, so you can analyze the results afterwards, avoid cache issues, etc. Maybe it would be better to come up with our own way to do this, without relying on dejagnu directly. Much like we have the TESTS variable, we could have a BOARDS variable: make check TESTS="gdb.base/break.exp" BOARDS="native-gdbserver unix" and then the outputs foreach board would be in "testsuite/outputs.native-gdbserver/" and "testsuite/outputs.unix/" respectively. Thanks, Pedro Alves