Re: [RFC] GDB performance testing infrastructure

[Doug Evans <dje@google.com>]

Yao Qi writes:
 > On 08/27/2013 09:49 PM, Agovic, Sanimir wrote:
 > >> * Remote debugging.  It is slower to read from the remote target, and
 > >> >     worse, GDB reads the same memory regions in multiple times, or reads
 > >> >     the consecutive memory by multiple packets.
 > >> >
 > > Once gdb and gdbserver share most of the target code, the overhead will be
 > > caused by the serial protocol roundtrips. But this will take a while...
 > 
 > Sanimir, thanks for your comments!
 > 
 > One of the motivations of the performance testing is to measure the
 > overhead of RSP in some scenarios, and look for the opportunities to
 > improve it, or add a completely new protocol, which is an extreme case.

For reference sake,
a big part of the "reading same memory region multiple times"
and "consecutive memory by multiple packets" is gdb's inability to use
its dcache (apropos dcache) for text segments.  Blech.

 > Once the infrastructure is ready, we can write some tests to see how 
 > efficient or inefficient RSP is.

"set debug remote 1" and you're there.  1/2 :-)

"But seriously ..."
Latency can be a huge problem with any remote protocol.
Running gdb+gdbserver on the same machine can hide issues
without tracking, e.g., packet counts in addition to cpu/wall time.
[*both* cpu and wall time are useful]
I hope the test harness will incorporate this.

 > >> >   * Tracepoint.  Tracepoint is designed to be efficient on collecting
 > >> >     data in the inferior, so we need performance tests to guarantee that
 > >> >     tracepoint is still efficient enough.  Note that we a test
 > >> >     `gdb.trace/tspeed.exp', but there are still some rooms to improve.
 > >> >
 > > Afaik the tracepoint functionality is quite separated from gdb may be tested
 > > in isolation. Having a generic benchmark framework covering the most parts of
 > > gdb is probably_the_  way to go but I see some room for specialized benchmarks
 > > e.g. for tracepoints with a custom driver. But my knowledge is too vague on
 > > the topic.
 > >
 > 
 > Well, it is sort of design trade-off.  We need a framework generic 
 > enough to handle most of the testing requirements for different GDB 
 > modules, (such as solib, symbols, backtrace, disassemble, etc), on the 
 > other hand, we want each test is specialized for the corresponding GDB 
 > module, so that we can find more details.
 > 
 > I am inclined to handle testing to _all_ modules under this generic
 > framework.

Agreed.

 > >> >   2. Detect performance regressions.  We collected the performance data
 > >> >      of each micro-benchmark, and we need to detect or identify the
 > >> >      performance regression by comparing with the previous run.  It is
 > >> >      more powerful to associate it with continuous testing.
 > >> >
 > > Something really simple, so simple that one could run it silently with every
 > > make invokation. For a newcomer, it took me some time to get used to make
 > > check e.g. setup, run, and interpret the tests with various settings. Something
 > > simpler would help to run it more often.
 > >
 > 
 > Yes, I agree, everything should be simple.  I assume that people
 > running performance testing should be familiar with GDB regular
 > regression test, like 'make check'.  We'll provide 'make check-perf' to 
 > run performance testing ,and it doesn't add extra difficulties on top of 
 > 'make check', from user's point of view, IMO.
 > 
 > > I like to add the Machine Interface (MI) to the list, but it is quite rudimentary:
 > >
 > > $ gdb -interpreter mi -q debugee
 > > [...]
 > > -enable-timings
 > > ^done
 > > (gdb)
 > > -break-insert -f main
 > > ^done,bkpt={...},time={wallclock="0.00656",user="0.00000",system="0.00000"}
 > > [...]
 > > (gdb)
 > > -exec-step
 > > ^running
 > > *running,thread-id="1"
 > > (gdb)
 > > *stopped,[...],time={wallclock="0.19425",user="0.09700",system="0.04200"}
 > > (gdb)
 > >
 > > With -enable-timings[1] enabled, every result record has a time triple
 > > appended, even for async[2] ones. If we come up with a full mi parser
 > > one could run tests w/o timings. A mi result is quite json-ish.
 > 
 > Thanks for the input.
 > 
 > >
 > > (To be honest I do not know how timings are composed of =D)
 > >
 > > In addition there are some tools for plotting benchmark results[3].
 > >
 > > [1]http://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI-Miscellaneous-Commands.html
 > > [2]https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI-Async-Records.html
 > > [3]http://speed.pypy.org/
 > 
 > I am using speed to track and show the performance data I got from the 
 > GDB performance tests.  It is able to associate the performance data to 
 > the commit, so easy to find which commit causes regressions.  However, 
 > my impression is that speed or its dependent packages are not 
 > well-maintained nowadays.
 > 
 > After some search online, I like the chromium performance test and its 
 > plot, personally.  It is integrated with buildbot (a customized version).
 > 
 >    http://build.chromium.org/f/chromium/perf/dashboard/overview.html
 > 
 > However, as I said in this proposal, let us focus on goal #1 first, get
 > the framework ready and collect performance data.

Agreed.
Let's get a good framework in place.