From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 58801 invoked by alias); 20 Aug 2015 18:22:42 -0000 Mailing-List: contact gdb-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-owner@sourceware.org Received: (qmail 58788 invoked by uid 89); 20 Aug 2015 18:22:42 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-0.2 required=5.0 tests=AWL,BAYES_50,KAM_ASCII_DIVIDERS,KAM_LAZY_DOMAIN_SECURITY,RP_MATCHES_RCVD autolearn=no version=3.3.2 X-HELO: e06smtp11.uk.ibm.com Received: from e06smtp11.uk.ibm.com (HELO e06smtp11.uk.ibm.com) (195.75.94.107) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (CAMELLIA256-SHA encrypted) ESMTPS; Thu, 20 Aug 2015 18:22:39 +0000 Received: from /spool/local by e06smtp11.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 20 Aug 2015 19:22:36 +0100 Received: from d06dlp02.portsmouth.uk.ibm.com (9.149.20.14) by e06smtp11.uk.ibm.com (192.168.101.141) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 20 Aug 2015 19:22:35 +0100 X-MailFrom: arnez@linux.vnet.ibm.com X-RcptTo: gdb@sourceware.org Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by d06dlp02.portsmouth.uk.ibm.com (Postfix) with ESMTP id 966FF2190046 for ; Thu, 20 Aug 2015 19:22:08 +0100 (BST) Received: from d06av04.portsmouth.uk.ibm.com (d06av04.portsmouth.uk.ibm.com [9.149.37.216]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id t7KIMYBS29163550 for ; Thu, 20 Aug 2015 18:22:34 GMT Received: from d06av04.portsmouth.uk.ibm.com (localhost [127.0.0.1]) by d06av04.portsmouth.uk.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id t7KIMYMF022582 for ; Thu, 20 Aug 2015 12:22:34 -0600 Received: from oc1027705133.ibm.com (dyn-9-152-212-195.boeblingen.de.ibm.com [9.152.212.195]) by d06av04.portsmouth.uk.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id t7KIMX1p022572 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 20 Aug 2015 12:22:33 -0600 From: Andreas Arnez To: Peter Griffin Cc: gdb@sourceware.org, lee.jones@linaro.org Subject: Re: RFC GDB Linux Awareness analysis References: <20150603142858.GA19370@griffinp-ThinkPad-X1-Carbon-2nd> Date: Thu, 20 Aug 2015 18:22:00 -0000 In-Reply-To: <20150603142858.GA19370@griffinp-ThinkPad-X1-Carbon-2nd> (Peter Griffin's message of "Wed, 3 Jun 2015 15:28:58 +0100") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15082018-0041-0000-0000-00000580D52F X-IsSubscribed: yes X-SW-Source: 2015-08/txt/msg00033.txt.bz2 Hi, Sorry for the late reply. Yao Qi pointed me to this RFC; I've just missed it before. For those who have missed it as well, the original posting was here: https://sourceware.org/ml/gdb-patches/2015-06/msg00040.html Independently of that proposal I have presented some thoughts about improving GDB's Linux kernel debugging support on the GNU Tools Cauldron. Surprisingly or not, some of the goals look very similar. The slides are here: https://gcc.gnu.org/wiki/cauldron2015?action=3DAttachFile&do=3Dview&targe= t=3DAndreas+Arnez_+Debugging+Linux+kernel+dumps+with+GDB.pdf In that talk I was focusing more on kernel dump debugging rather than live debugging, but I also tried to emphasize the fact that the feature sets required for both (should) have a large overlap. Only a few features are specific to one or the other, e.g. live debugging requires a notification method for refreshing the task list and the modules list. =46rom the resonance in the talk I got the impression that there is indeed wide-spread interest in having improved kernel debug support in GDB. So I believe posting the patches to get more feedback would be worth-while. On Wed, Jun 03 2015, Peter Griffin wrote: > Hi GDB community, > > Overview > =3D=3D=3D=3D=3D=3D=3D=3D > > The purpose of this email is to describe a useful feature which has been = developed > by STMicroelectronics inside of GDB for debugging Linux kernels. I will c= over at > a high level some of the implementation details and what I see as the adv= antages > / disadvantages of the implementation. I will also cover some other alter= native=20 > approaches that I'm aware of. > > The purpose is to facilitate discussion with the GDB experts on this > mailing list as to what the "correct" way to implement this functionality= would > be. I'm copying gdb@sourceware.org instead of gdb-patches, since I think it's better suited for now -- until we are actually discussing a patch. > The end goal is to have an upstream implementation of this functionality. > > Introduction > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > STMicroelectronics has a patchset on top of vanilla GDB which adds much b= etter > Linux kernel awareness into GDB. They have called this GDB extension LKD = (Linux > Kernel Debugger). This GDB extension is primarily used in conjunction wit= h ST's > JTAG debugger for debugging ARM / SH4 SoCs, and is implemented as an inte= rnal > GDB extension, written in C. > > The LKD extension is nicely abstracted from the underlying JTAG interface, > and I have used it to debug a ARMv7 kernel running in QEMU via gdbremote. > > ST would like to contribute these patches back to GDB, as we think it cou= ld > be useful not only for ARM Linux debugging, but also other CPU/OS combina= tions > in the future. > > LKD can be broadly split into the following parts: - > > 1) Linux task awareness > 2) Loadable module support > 3) Various Linux helper commands > > The next section looks at each of the three parts, and any other implemen= tations > I'm aware of that currently exist. > > > 1) Task Awareness > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 1.1) LKD Linux Task awareness > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > > When using mainline GDB for debugging a Linux kernel via JTAG GDB will ty= pically > only show the actual hardware threads. > > The LKD task awareness extension (lkd-process.c) adds the ability for GDB= to > parse some kernel data structures, so it can build a thread list of all k= ernel > threads in the running Linux kernel. > > When halting the processor (via a breakpoint, ctrl+c etc) the thread list= is > re-enumerated, so new tasks are visible to GDB and tasks which have exited > are removed from the thread list. > > To achieve this GDB has to know about various Linux kernel data structure= s, > and also various fields within these structures (mainly task_struct, > thread_info, mm_struct). Code has been implemented to parse these structs, > and also do virtual to physical address translations and cache handling. > Various frame unwinders are also implemented to stop backtracing on vario= us > exceptions, and entry points to the kernel (these would be a useful addit= ion > regardless of which task awareness approach is taken). > > Advantages > - Adds Linux kernel task awareness to GDB > - Supports symbolic debugging of user processes > - Contextual information (structs / field offsets) are readily available= inside of GDB > - Has been used and well tested inside ST for some time > > Disadvantages > - Being implemented in C within GDB creates a dependency between GDB and= the Linux kernel > - Mainly tested on 3.10 and 2.6.30 with ARM and SH4 kernels, being upstr= eamed would > expose it to many differing kernel versions and architectures. > > - I can't see any other "OS awareness" support currently in the GDB code= base Actually, it seems that there's a bit of special OS kernel debug support, e.g. in bsd-kvm.c. But I'd rather view this as providing support for the Linux kernel *runtime* and compare it to components like linux-thread-db.c for user-space threads. GDB has several of those already. They often leave the internals of accessing the thread library's data structures to a "thread debug library" outside GDB. We may or may not want to do something similar for the kernel, depending on the complexity of the data structures involved and on the likelihood for them to change. Maybe the "thread debug library" could actually be a Python script provided by the kernel developers, as outlined in 1.3 below. Or maybe it's just easier to put all the required logic into GDB and perform appropriate version checks where necessary. > 1.2) OpenOCD / GDB Linux task awareness > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Whilst looking for any prior art in this area I found that OpenOCD already > implements some basic Linux task awareness. See here: - > http://openocd.org/doc/doxygen/html/linux_8c_source.html > > I've this working to the extent where I can connect via JTAG to a ARMv7 U= 8500 > Snowball board and enumerate a thread list in GDB. It required some debug= ging > & hacking to get this far. > > This implementation passes the thread list to GDB via the gdbremote proto= col > and as such changes required in GDB are minimal. > > Advantages > - OpenOCD already supports many target types (ARM / MIPS), and has suppo= rt > for virt to phys translations / cache handling etc. > > - OpenOCD also implements task awareness for other RTOS=E2=80=99s (Threa= dX / FreeRTOS / eCos) > > - Using gdbremote means GDB changes are (so far) minimal. > > - OpenOCD / Linux Kernel dependency already exists > > > Disadvantages > - Creates a dependency between Linux kernel data structures and OpenOCD. > > - I believe finding field offsets within structs is currently not possib= le via > gdbremote protocol. > > Currently OpenOCD generates these offsets at compile time which is ugl= y and needs > fixing. See here http://openocd.org/doc-release/doxygen/linux__header_= 8h_source.html. > > Being able to find field offsets would in my opinion be a useful addit= ion to the > gdbremote protocol which would allow OpenOCD task awareness to work mu= ch better > at runtime. >=20=20=20=20 > - Doesn't support debugging user processes. I think this would still req= uire some > GDB changes, and also gdbremote protocol changes to get working correc= tly. > > - Needs to be made more generic > > > 1.3) Python Task awareness > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > > Jan Kiszka from Siemens has implemented some basic Linux kernel task awar= eness using > the GDB Python interface. > > See here https://lwn.net/Articles/631167/ and here > https://github.com/torvalds/linux/blob/master/Documentation/gdb-kernel-de= bugging.txt > > This support is currently limited to the following commands: - > > lx_current -- Return current task > lx_per_cpu -- Return per-cpu variable > lx_task_by_pid -- Find Linux task by PID and return the task_struct varia= ble > lx_thread_info -- Calculate Linux thread_info from task variable > > However this could be extended to build up a thread list. The GDB Python = interface > would I believe need to be extended to allow a thread list to be passed b= ack to GDB > via Python. > > Advantages > - Code parsing Linux kernel structures lives in the kernel source tree > - Contextual information is easily available > - Works with all OCD implementations not just OpenOCD=20 > > Disadvantages > - Doesn't exist yet > - Python / GDB interface would need to be extended to support threads > > Questions > =3D=3D=3D=3D=3D=3D=3D=3D > > 1) What do GDB community think about having Linux OS task awareness in th= e GDB > codebase? For my part, I don't see problem with that. We already have code in GDB for supporting various runtime environments. This would just be another one. Currently, even if an external thread debug library is involved, the code that uses it is still specially crafted for a particular runtime and does not work for another. > 2) Is there a preferred way to implement this (C extension / gdbremote / = Python > / something else)? Since many existing JTAG debuggers as well as Qemu will not offer a Linux task list, I think such logic belongs on the GDB client side. It's also on the client side where you have the DWARF info from vmlinux that enables dealing with varying field offsets in structures. Whether the logic should be written in C or as a Python (or Guile) extension, I don't have a strong opinion. > 3) Any other implementations / advantages / disadvantages that I'm not cu= rrently > aware of? > > 2) Loadable module support > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > > 2.1) LKD Loadable module support > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > lkd-modules.c adds better Linux kernel module symbolic symbol support to = GDB using the > GDB shared libraries infrastructure (solib). > > It has hooks to enable the debugging of module __init sections to reflect= that these pages > get discarded post-load. It also implements the same section layout algor= ithm as > the kernel to speed up symbol resolution, only inspecting the target=E2= =80=99s memory if there is a > mismatch (inspecting target memory can be slow). > > I think this part could be upstreamed separately to the task awareness su= pport, although > I=E2=80=99ve not tried separating it yet from the other LKD patches. Yes, I think that separate patch sets would be helpful. > Advantages > - Allows full symbolic debugging of Linux loadable modules including ini= t sections > > - Would be independent of underlying communication mechanism (JTAG / gdb= remote etc) > > Disadvantages > - Some dependency between GDB and Linux kernel is still present > > Questions: > > Are GDB community happy to have a Linux specific solib functionality in t= he GDB code base? For my part, same as with thread support: GDB already supports various shared library implementations and "knows" about their internals, so this would just be another one. > 2.2) Python Loadable module support > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > I've not managed to look to much at this, but some basic support exists, = see here > https://github.com/torvalds/linux/blob/master/Documentation/gdb-kernel-de= bugging.txt > > Having read the LKD modules implementation I don=E2=80=99t think this can= be anywhere near > as functionally complete. > > > 3) Linux Helper commands > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 3.1) LKD helper commands > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > LKD implements various Linux helper commands inside GDB such as: - > - dmesg - dump dmesg log buffer from kernel > - process_info - prints various info about current process > - pmap - prints memory map of current process > - vm_translate - translates virtual to physical address > - proc_interrupts - prints interrupt statistics > - proc_iomem - prints I/O mem map > - proc_cmdline - prints the contents of /proc/cmdline > - proc_version - prints the contents of /proc/version > - proc_mounts - print the contents of /proc/mounts > - proc_meminfo - print the contents of /proc/meminfo. > > Advantages > - Can be used by all GDB based debug solutions > - Some precedent of OS related commands in the GDB code base=20 > https://sourceware.org/gdb/current/onlinedocs/gdb/OS-Information.html#= OS-Information > > Disadvantages > - Creates a GDB dependency with the kernel > > Python helper commands > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Jan=E2=80=99s python scripts also implement some of the same LKD commands= such as 'dmesg'. > > See here https://lwn.net/Articles/631167/ and=20 > https://github.com/torvalds/linux/blob/master/Documentation/gdb-kernel-de= bugging.txt > > Advantages > - Code that is parsing Linux kernel structures lives in the kernel sourc= e tree > - Can be used by all GDB based debug solutions > > > Questions > - Do GDB community mind Linux specific custom commands being added to GD= B code base?=20 I'm not sure about that one. We also need to consider the capability for the user to add more such commands, e.g. specific to certain device drivers or certain debug scenarios. In my view both cases should ideally be covered with the same underlying mechanism. > My current opinion is that helper commands which can be, should be migrat= ed from C code > into Python, and merged into the kernel source tree (and then retired fro= m the LKD patchset). That's my current view as well. Or maybe we could consider using EPPIC for those. (See my Cauldron slides.) > If you got here, thanks for reading this far! Like I said at the beginnin= g, the purpose of > this email is to stimulate some discussion on what you folks consider the= 'correct' way to > implement this OS awareness functionality is. > > All feedback is welcome. > > Kind regards, > > Peter.