[RFC,0/6] Support for Linux kernel thread aware debug

Message ID 1548738199-9403-1-git-send-email-omair.javaid@linaro.org
Headers show
Series
  • Support for Linux kernel thread aware debug
Related show

Message

Omair Javaid Jan. 29, 2019, 5:03 a.m.
This patch series implements linux kernel target which allows linux kernel
tasks to be viewed as GDB threads. We have had multiple set of patches
submitted over last few years aiming to insert add-ons into GDB for debugging
Linux kernel. This patch series builds on top of Linux kernel infrastructure
submitted by Philipp Rudo in his various sets of patches aiming to debug
Linux kernel dumps.

Here is a list of patch series submitted by Philipp Rudo:
RFC v5 https://sourceware.org/ml/gdb-patches/2018-03/msg00243.html
RFC v4 https://sourceware.org/ml/gdb-patches/2017-06/msg00347.html
RFC v3 https://sourceware.org/ml/gdb-patches/2017-03/msg00268.html

Changes in this patch series:
- Rebase over current GDB
- Separate out core-dump support for future
- Rework implementation of linux_kernel_ops class for live task management
- Implements target class for linux kernel
- Add support for AArch64 and Arm targets to debug linux kernel with
thread awareness.

Limitation:
- No support for module debugging
- Only support direct mapped kernel addresses.

There has been a lot of discussion over mailing list about how to improve
debugability of Linux kernel using gdb. These discussion threads eventually
ended up with Phillip writing above patch series which unfortunately stalled
permanently due to various types of delays.

I have listed below some of the past mailing list threads for reference.

Andreas Arnez
https://gcc.gnu.org/wiki/cauldron2015?action=AttachFile&do=get&target=Andreas+Arnez_+Debugging+Linux+kernel+dumps+with+GDB.pdf

Peter Griffin: RFC GDB Linux Awareness analysis
https://sourceware.org/ml/gdb-patches/2015-06/msg00040.html

Ales Novak: gdb on Linux Kernel dumps (gdb-kdump)
https://sourceware.org/ml/gdb/2015-09/msg00014.html

Kieran Bingham: GDB Linux Awareness revisited
https://www.sourceware.org/ml/gdb/2016-01/msg00028.html

Omair Javaid (3):
  Linux kernel thread awareness Arm target support
  Linux kernel thread awareness AArch64 target support
  Linux kernel debug infrastructure and target Linux kernel

Philipp Rudo (4):
  Convert substitute_path_component to C++
  Add libiberty/concat styled concat_path function
  Add scoped_restore_regcache_ptid
  Linux kernel debug infrastructure and target Linux kernel

 gdb/Makefile.in                 |    7 +
 gdb/aarch64-lk-tdep.c           |  135 +++++
 gdb/arm-lk-tdep.c               |  139 +++++
 gdb/auto-load.c                 |   19 +-
 gdb/common/common-utils.h       |   11 +
 gdb/configure.tgt               |    9 +-
 gdb/defs.h                      |    1 +
 gdb/gdbarch.c                   |   34 +-
 gdb/gdbarch.h                   |   11 +-
 gdb/gdbarch.sh                  |    4 +
 gdb/lk-bitmap.h                 |  226 ++++++++
 gdb/lk-list.h                   |  200 +++++++
 gdb/lk-low.c                    | 1126 +++++++++++++++++++++++++++++++++++++++
 gdb/lk-low.h                    |  354 ++++++++++++
 gdb/osabi.c                     |    1 +
 gdb/regcache.h                  |   21 +
 gdb/unittests/utils-selftests.c |   10 +-
 gdb/utils.c                     |   85 +--
 gdb/utils.h                     |   26 +-
 19 files changed, 2364 insertions(+), 55 deletions(-)
 create mode 100644 gdb/aarch64-lk-tdep.c
 create mode 100644 gdb/arm-lk-tdep.c
 create mode 100644 gdb/lk-bitmap.h
 create mode 100644 gdb/lk-list.h
 create mode 100644 gdb/lk-low.c
 create mode 100644 gdb/lk-low.h

-- 
2.7.4

Comments

John Baldwin Jan. 29, 2019, 5:30 p.m. | #1
On 1/28/19 9:03 PM, Omair Javaid wrote:
> This patch series implements linux kernel target which allows linux kernel

> tasks to be viewed as GDB threads. We have had multiple set of patches

> submitted over last few years aiming to insert add-ons into GDB for debugging

> Linux kernel. This patch series builds on top of Linux kernel infrastructure

> submitted by Philipp Rudo in his various sets of patches aiming to debug

> Linux kernel dumps.


I just have one minor suggestion / comment about file names.  I maintain
FreeBSD kernel patches for gdb out-of-tree (for various reasons), and those
patches use some similar things (e.g. a different OSABI).  My comment has
to do with the filenames.  Other osabi-specific files tend to use more
verbose names such as 'linux-arm-nat.c'.  I wonder if it makes sense to
spell out linux here as well.  I have been using 'arm-fbsd-kern.c' as a
complement to 'arm-fbsd-tdep.c' for the kernel gdbarch.  The architecture
independent files follow the patter 'fbsd-k*.c' (e.g. fbsd-kld.c for modules
and fbsd-kthr.c for thread enumeration), but I would be happy to move those
to something like 'fbsd-kern-ld.c' and 'fbsd-kern-thr.c'.  For your current
patchset that might mean something like 'linux-kern-tdep.c' instead of
'lk-tdep.c'.  I would also be fine with 'arm-linux-kern-tdep.c' instead of
'arm-linux-kern.c' perhaps if other folks feel like that is more consistent.

-- 
John Baldwin
Andreas Arnez Jan. 30, 2019, 3:01 p.m. | #2
On Tue, Jan 29 2019, John Baldwin wrote:

> I just have one minor suggestion / comment about file names.  I maintain

> FreeBSD kernel patches for gdb out-of-tree (for various reasons), and those

> patches use some similar things (e.g. a different OSABI).  My comment has

> to do with the filenames.  Other osabi-specific files tend to use more

> verbose names such as 'linux-arm-nat.c'.  I wonder if it makes sense to

> spell out linux here as well.  I have been using 'arm-fbsd-kern.c' as a

> complement to 'arm-fbsd-tdep.c' for the kernel gdbarch.  The architecture

> independent files follow the patter 'fbsd-k*.c' (e.g. fbsd-kld.c for modules

> and fbsd-kthr.c for thread enumeration), but I would be happy to move those

> to something like 'fbsd-kern-ld.c' and 'fbsd-kern-thr.c'.  For your current

> patchset that might mean something like 'linux-kern-tdep.c' instead of

> 'lk-tdep.c'.  I would also be fine with 'arm-linux-kern-tdep.c' instead of

> 'arm-linux-kern.c' perhaps if other folks feel like that is more consistent.


I believe it was me who originated the "lk" notation, so maybe I should
provide my few cents on its rationale here:

1. The string "linux" in GDB source file names specifically means
"GNU/Linux user-space runtime".  For instance, "aarch64-linux-tdep.c"
implements target-dependent code for GNU/Linux user-space programs on
AArch64 hardware.  Thus "linux" is reserved for this particular purpose,
and collisions should better be avoided.

2. In principle, the Linux kernel runtime is just another runtime
environment, like the GNU/Linux user-space runtime or the QNX Neutrino
runtime (say).  And since all runtimes are named by a single short word
so far ("linux", "nbsd", "nto", etc.), it seems consistent to use the
same scheme for the Linux kernel runtime as well.  In particular this
means not to use any dashes or underscores.

3. The runtime name becomes part of many identifiers.  Any additional
characters make these identifiers more bulky.  Thus, in my opinion, the
name should be as short as possible.

If "lk" is not considered clear enough, alternatives could be something
like "lxk", "lkr" (for "Linux kernel runtime"), "linuxk", or "linuxkr".
For the FreeBSD kernel I suggest "fbsdk" or "fbsdkr".

--
Andreas
Omair Javaid Feb. 4, 2019, 9:12 a.m. | #3
On Tue, 29 Jan 2019 at 22:30, John Baldwin <jhb@freebsd.org> wrote:
>

> On 1/28/19 9:03 PM, Omair Javaid wrote:

> > This patch series implements linux kernel target which allows linux kernel

> > tasks to be viewed as GDB threads. We have had multiple set of patches

> > submitted over last few years aiming to insert add-ons into GDB for debugging

> > Linux kernel. This patch series builds on top of Linux kernel infrastructure

> > submitted by Philipp Rudo in his various sets of patches aiming to debug

> > Linux kernel dumps.

>

> I just have one minor suggestion / comment about file names.  I maintain

> FreeBSD kernel patches for gdb out-of-tree (for various reasons), and those

> patches use some similar things (e.g. a different OSABI).  My comment has

> to do with the filenames.  Other osabi-specific files tend to use more

> verbose names such as 'linux-arm-nat.c'.  I wonder if it makes sense to

> spell out linux here as well.  I have been using 'arm-fbsd-kern.c' as a

> complement to 'arm-fbsd-tdep.c' for the kernel gdbarch.  The architecture

> independent files follow the patter 'fbsd-k*.c' (e.g. fbsd-kld.c for modules

> and fbsd-kthr.c for thread enumeration), but I would be happy to move those

> to something like 'fbsd-kern-ld.c' and 'fbsd-kern-thr.c'.  For your current

> patchset that might mean something like 'linux-kern-tdep.c' instead of

> 'lk-tdep.c'.  I would also be fine with 'arm-linux-kern-tdep.c' instead of

> 'arm-linux-kern.c' perhaps if other folks feel like that is more consistent.


Hi John,

Andreas has aptly described the reason behind choosing the
nomenclature of new linux kernel specific files as it is right now and
i am open to any suggestion that my come up during reviews.

Also I was wondering if you can share details of kernel debugging
features which has implemented in your out of tree FreeBSD patches for
GDB. Also share some insight on ways to test this patchset.

Thanks!

>

> --

> John Baldwin

>

>
John Baldwin Feb. 4, 2019, 5:52 p.m. | #4
On 2/4/19 1:12 AM, Omair Javaid wrote:
> On Tue, 29 Jan 2019 at 22:30, John Baldwin <jhb@freebsd.org> wrote:

>>

>> On 1/28/19 9:03 PM, Omair Javaid wrote:

>>> This patch series implements linux kernel target which allows linux kernel

>>> tasks to be viewed as GDB threads. We have had multiple set of patches

>>> submitted over last few years aiming to insert add-ons into GDB for debugging

>>> Linux kernel. This patch series builds on top of Linux kernel infrastructure

>>> submitted by Philipp Rudo in his various sets of patches aiming to debug

>>> Linux kernel dumps.

>>

>> I just have one minor suggestion / comment about file names.  I maintain

>> FreeBSD kernel patches for gdb out-of-tree (for various reasons), and those

>> patches use some similar things (e.g. a different OSABI).  My comment has

>> to do with the filenames.  Other osabi-specific files tend to use more

>> verbose names such as 'linux-arm-nat.c'.  I wonder if it makes sense to

>> spell out linux here as well.  I have been using 'arm-fbsd-kern.c' as a

>> complement to 'arm-fbsd-tdep.c' for the kernel gdbarch.  The architecture

>> independent files follow the patter 'fbsd-k*.c' (e.g. fbsd-kld.c for modules

>> and fbsd-kthr.c for thread enumeration), but I would be happy to move those

>> to something like 'fbsd-kern-ld.c' and 'fbsd-kern-thr.c'.  For your current

>> patchset that might mean something like 'linux-kern-tdep.c' instead of

>> 'lk-tdep.c'.  I would also be fine with 'arm-linux-kern-tdep.c' instead of

>> 'arm-linux-kern.c' perhaps if other folks feel like that is more consistent.

> 

> Hi John,

> 

> Andreas has aptly described the reason behind choosing the

> nomenclature of new linux kernel specific files as it is right now and

> i am open to any suggestion that my come up during reviews.


Ok.  I'm not firmly tied to a specific naming scheme, and I'll defer to
whatever the global maintainers prefer.

> Also I was wondering if you can share details of kernel debugging

> features which has implemented in your out of tree FreeBSD patches for

> GDB. Also share some insight on ways to test this patchset.


So I don't have any formalized testing. :-/  For testing cross-debug of
crash dumps I modified the libkvm library in FreeBSD's base system to
support reading the kernel crash dump format of other architectures and
can then generate a crash dump in a VM / qemu instance and copy it to a
64-bit x86 host to examine against a sysroot.  However, most of my testing
is just by using the kernel target as most of my work is FreeBSD kernel
and userland stuff, so I exercise it that way.  I do have various gdb scripts
(still written in the ancient gdb script language from gdb 6.x rather than
python) at github.com/bsdjhb/kdbg/tree/master/gdb  They aren't generally
useful on other kernels though and are specific to FreeBSD.

In terms of features, the current FreeBSD patches support most FreeBSD
architectures (sparc64 is untested and I haven't gotten stack traces on
ppc to work right, though I think I can reuse what I just did for risc-v
to fix ppc at some point).  A few years ago I reworked the libkvm library
in FreeBSD to support cross-debugging of crashdumps, so you can generally
cross-debug crashdumps on FreeBSD.  Currently this only works on a FreeBSD
host because I haven't pulled out a portable version of libkvm that could
be compiled on a non-FreeBSD host yet.  It does support kernel modules by
reusing the shared library interface.  This is implemented by having a
custom set of solib_ops that get attached to the kernel-specific gdbarch
during that OSABI's init handler.  It has some custom commands that are
wrappers around 'thread N' that let you switch to a thread by process ID
or kernel thread ID.  The main things it requires of each architecture are
a method for populating a regcache from a kernel thread register state
('struct pcb' on FreeBSD), and custom frame unwinders to walk across
exception frames on the stack.  Currently these are marked as signal frames
so that GDB will unwind across them ok when you have a page fault in the
kernel due to a NULL function pointer (other frame unwinders stop unwinding
when they see a NULL pc, but signal frames can keep unwinding past those).

I gave a talk a couple of years ago at a BSD conference about GDB that
includes some kgdb slides.  You can see the slides at
https://people.freebsd.org/~jhb/papers/bsdcan/2016/slides.pdf if that is
helpful.

-- 
John Baldwin
Omair Javaid Feb. 8, 2019, 8:54 a.m. | #5
On Mon, 4 Feb 2019 at 22:52, John Baldwin <jhb@freebsd.org> wrote:
>

> On 2/4/19 1:12 AM, Omair Javaid wrote:

> > On Tue, 29 Jan 2019 at 22:30, John Baldwin <jhb@freebsd.org> wrote:

> >>

> >> On 1/28/19 9:03 PM, Omair Javaid wrote:

> >>> This patch series implements linux kernel target which allows linux kernel

> >>> tasks to be viewed as GDB threads. We have had multiple set of patches

> >>> submitted over last few years aiming to insert add-ons into GDB for debugging

> >>> Linux kernel. This patch series builds on top of Linux kernel infrastructure

> >>> submitted by Philipp Rudo in his various sets of patches aiming to debug

> >>> Linux kernel dumps.

> >>

> >> I just have one minor suggestion / comment about file names.  I maintain

> >> FreeBSD kernel patches for gdb out-of-tree (for various reasons), and those

> >> patches use some similar things (e.g. a different OSABI).  My comment has

> >> to do with the filenames.  Other osabi-specific files tend to use more

> >> verbose names such as 'linux-arm-nat.c'.  I wonder if it makes sense to

> >> spell out linux here as well.  I have been using 'arm-fbsd-kern.c' as a

> >> complement to 'arm-fbsd-tdep.c' for the kernel gdbarch.  The architecture

> >> independent files follow the patter 'fbsd-k*.c' (e.g. fbsd-kld.c for modules

> >> and fbsd-kthr.c for thread enumeration), but I would be happy to move those

> >> to something like 'fbsd-kern-ld.c' and 'fbsd-kern-thr.c'.  For your current

> >> patchset that might mean something like 'linux-kern-tdep.c' instead of

> >> 'lk-tdep.c'.  I would also be fine with 'arm-linux-kern-tdep.c' instead of

> >> 'arm-linux-kern.c' perhaps if other folks feel like that is more consistent.

> >

> > Hi John,

> >

> > Andreas has aptly described the reason behind choosing the

> > nomenclature of new linux kernel specific files as it is right now and

> > i am open to any suggestion that my come up during reviews.

>

> Ok.  I'm not firmly tied to a specific naming scheme, and I'll defer to

> whatever the global maintainers prefer.

>

> > Also I was wondering if you can share details of kernel debugging

> > features which has implemented in your out of tree FreeBSD patches for

> > GDB. Also share some insight on ways to test this patchset.

>

> So I don't have any formalized testing. :-/  For testing cross-debug of

> crash dumps I modified the libkvm library in FreeBSD's base system to

> support reading the kernel crash dump format of other architectures and

> can then generate a crash dump in a VM / qemu instance and copy it to a

> 64-bit x86 host to examine against a sysroot.  However, most of my testing

> is just by using the kernel target as most of my work is FreeBSD kernel

> and userland stuff, so I exercise it that way.  I do have various gdb scripts

> (still written in the ancient gdb script language from gdb 6.x rather than

> python) at github.com/bsdjhb/kdbg/tree/master/gdb  They aren't generally

> useful on other kernels though and are specific to FreeBSD.

>

> In terms of features, the current FreeBSD patches support most FreeBSD

> architectures (sparc64 is untested and I haven't gotten stack traces on

> ppc to work right, though I think I can reuse what I just did for risc-v

> to fix ppc at some point).  A few years ago I reworked the libkvm library

> in FreeBSD to support cross-debugging of crashdumps, so you can generally

> cross-debug crashdumps on FreeBSD.  Currently this only works on a FreeBSD

> host because I haven't pulled out a portable version of libkvm that could

> be compiled on a non-FreeBSD host yet.  It does support kernel modules by

> reusing the shared library interface.  This is implemented by having a

> custom set of solib_ops that get attached to the kernel-specific gdbarch

> during that OSABI's init handler.  It has some custom commands that are

> wrappers around 'thread N' that let you switch to a thread by process ID

> or kernel thread ID.  The main things it requires of each architecture are

> a method for populating a regcache from a kernel thread register state

> ('struct pcb' on FreeBSD), and custom frame unwinders to walk across

> exception frames on the stack.  Currently these are marked as signal frames

> so that GDB will unwind across them ok when you have a page fault in the

> kernel due to a NULL function pointer (other frame unwinders stop unwinding

> when they see a NULL pc, but signal frames can keep unwinding past those).

>

> I gave a talk a couple of years ago at a BSD conference about GDB that

> includes some kgdb slides.  You can see the slides at

> https://people.freebsd.org/~jhb/papers/bsdcan/2016/slides.pdf if that is

> helpful.


Thanks John. This is helpful.

>

> --

> John Baldwin

>

>
Omair Javaid March 7, 2019, 11:49 a.m. | #6
On Fri, 8 Feb 2019 at 13:54, Omair Javaid <omair.javaid@linaro.org> wrote:
>

> On Mon, 4 Feb 2019 at 22:52, John Baldwin <jhb@freebsd.org> wrote:

> >

> > On 2/4/19 1:12 AM, Omair Javaid wrote:

> > > On Tue, 29 Jan 2019 at 22:30, John Baldwin <jhb@freebsd.org> wrote:

> > >>

> > >> On 1/28/19 9:03 PM, Omair Javaid wrote:

> > >>> This patch series implements linux kernel target which allows linux kernel

> > >>> tasks to be viewed as GDB threads. We have had multiple set of patches

> > >>> submitted over last few years aiming to insert add-ons into GDB for debugging

> > >>> Linux kernel. This patch series builds on top of Linux kernel infrastructure

> > >>> submitted by Philipp Rudo in his various sets of patches aiming to debug

> > >>> Linux kernel dumps.

> > >>

> > >> I just have one minor suggestion / comment about file names.  I maintain

> > >> FreeBSD kernel patches for gdb out-of-tree (for various reasons), and those

> > >> patches use some similar things (e.g. a different OSABI).  My comment has

> > >> to do with the filenames.  Other osabi-specific files tend to use more

> > >> verbose names such as 'linux-arm-nat.c'.  I wonder if it makes sense to

> > >> spell out linux here as well.  I have been using 'arm-fbsd-kern.c' as a

> > >> complement to 'arm-fbsd-tdep.c' for the kernel gdbarch.  The architecture

> > >> independent files follow the patter 'fbsd-k*.c' (e.g. fbsd-kld.c for modules

> > >> and fbsd-kthr.c for thread enumeration), but I would be happy to move those

> > >> to something like 'fbsd-kern-ld.c' and 'fbsd-kern-thr.c'.  For your current

> > >> patchset that might mean something like 'linux-kern-tdep.c' instead of

> > >> 'lk-tdep.c'.  I would also be fine with 'arm-linux-kern-tdep.c' instead of

> > >> 'arm-linux-kern.c' perhaps if other folks feel like that is more consistent.

> > >

> > > Hi John,

> > >

> > > Andreas has aptly described the reason behind choosing the

> > > nomenclature of new linux kernel specific files as it is right now and

> > > i am open to any suggestion that my come up during reviews.

> >

> > Ok.  I'm not firmly tied to a specific naming scheme, and I'll defer to

> > whatever the global maintainers prefer.

> >

> > > Also I was wondering if you can share details of kernel debugging

> > > features which has implemented in your out of tree FreeBSD patches for

> > > GDB. Also share some insight on ways to test this patchset.

> >

> > So I don't have any formalized testing. :-/  For testing cross-debug of

> > crash dumps I modified the libkvm library in FreeBSD's base system to

> > support reading the kernel crash dump format of other architectures and

> > can then generate a crash dump in a VM / qemu instance and copy it to a

> > 64-bit x86 host to examine against a sysroot.  However, most of my testing

> > is just by using the kernel target as most of my work is FreeBSD kernel

> > and userland stuff, so I exercise it that way.  I do have various gdb scripts

> > (still written in the ancient gdb script language from gdb 6.x rather than

> > python) at github.com/bsdjhb/kdbg/tree/master/gdb  They aren't generally

> > useful on other kernels though and are specific to FreeBSD.

> >

> > In terms of features, the current FreeBSD patches support most FreeBSD

> > architectures (sparc64 is untested and I haven't gotten stack traces on

> > ppc to work right, though I think I can reuse what I just did for risc-v

> > to fix ppc at some point).  A few years ago I reworked the libkvm library

> > in FreeBSD to support cross-debugging of crashdumps, so you can generally

> > cross-debug crashdumps on FreeBSD.  Currently this only works on a FreeBSD

> > host because I haven't pulled out a portable version of libkvm that could

> > be compiled on a non-FreeBSD host yet.  It does support kernel modules by

> > reusing the shared library interface.  This is implemented by having a

> > custom set of solib_ops that get attached to the kernel-specific gdbarch

> > during that OSABI's init handler.  It has some custom commands that are

> > wrappers around 'thread N' that let you switch to a thread by process ID

> > or kernel thread ID.  The main things it requires of each architecture are

> > a method for populating a regcache from a kernel thread register state

> > ('struct pcb' on FreeBSD), and custom frame unwinders to walk across

> > exception frames on the stack.  Currently these are marked as signal frames

> > so that GDB will unwind across them ok when you have a page fault in the

> > kernel due to a NULL function pointer (other frame unwinders stop unwinding

> > when they see a NULL pc, but signal frames can keep unwinding past those).

> >

> > I gave a talk a couple of years ago at a BSD conference about GDB that

> > includes some kgdb slides.  You can see the slides at

> > https://people.freebsd.org/~jhb/papers/bsdcan/2016/slides.pdf if that is

> > helpful.

>

> Thanks John. This is helpful.

>

> >

> > --

> > John Baldwin

> >

> >



Ping!

I am going to be reworking the current patch-set accommodating the
suggestions and comments I have received so far.

Kindly post your comments in next week or so if there are any further
other ideas/suggestions.

Thanks!