dma-debug: fix soft lockup when a lot of debug are enabled

Message ID 20181211103616.22137-1-anders.roxell@linaro.org
State New
Headers show
Series
  • dma-debug: fix soft lockup when a lot of debug are enabled
Related show

Commit Message

Anders Roxell Dec. 11, 2018, 10:36 a.m.
When running a kernel in qemu with enough debugging options (slub-debug,
ftrace, kcov, kasan, ubsan, ...) enabled together, that results in a
slow initcall. So a 'watchdog: BUG: soft lockup' happens:

[   44.105619] Call trace:
[   44.106709]  __slab_alloc+0x70/0x88
[   44.107757]  kmem_cache_alloc_trace+0x13c/0x590
[   44.108764]  prealloc_memory+0xb8/0x240
[   44.109768]  dma_debug_init+0x174/0x1f0
[   44.110725]  do_one_initcall+0x430/0x8b0
[   44.111719]  do_initcall_level+0x548/0x5b4
[   44.112672]  do_initcalls+0x28/0x4c
[   44.113628]  do_basic_setup+0x34/0x3c
[   44.114559]  kernel_init_freeable+0x194/0x24c
[   44.115570]  kernel_init+0x24/0x18c
[   44.116498]  ret_from_fork+0x10/0x18

Rework to call cond_resched(), in the function prealloc_memory()'s
for-loop.

Signed-off-by: Anders Roxell <anders.roxell@linaro.org>

---
 kernel/dma/debug.c | 1 +
 1 file changed, 1 insertion(+)

-- 
2.19.2

Comments

Robin Murphy Dec. 11, 2018, 11:51 a.m. | #1
Hi Anders,

On 11/12/2018 10:36, Anders Roxell wrote:
> When running a kernel in qemu with enough debugging options (slub-debug,

> ftrace, kcov, kasan, ubsan, ...) enabled together, that results in a

> slow initcall. So a 'watchdog: BUG: soft lockup' happens:

> 

> [   44.105619] Call trace:

> [   44.106709]  __slab_alloc+0x70/0x88

> [   44.107757]  kmem_cache_alloc_trace+0x13c/0x590

> [   44.108764]  prealloc_memory+0xb8/0x240

> [   44.109768]  dma_debug_init+0x174/0x1f0

> [   44.110725]  do_one_initcall+0x430/0x8b0

> [   44.111719]  do_initcall_level+0x548/0x5b4

> [   44.112672]  do_initcalls+0x28/0x4c

> [   44.113628]  do_basic_setup+0x34/0x3c

> [   44.114559]  kernel_init_freeable+0x194/0x24c

> [   44.115570]  kernel_init+0x24/0x18c

> [   44.116498]  ret_from_fork+0x10/0x18

> 

> Rework to call cond_resched(), in the function prealloc_memory()'s

> for-loop.


Funnily enough I've just done a bit of work in this area which should 
make allocation a lot more efficient - would you mind giving this series 
a spin on the same setup to see how much it helps the performance?

https://lore.kernel.org/lkml/cover.1544447601.git.robin.murphy@arm.com/

Thanks,
Robin.

> 

> Signed-off-by: Anders Roxell <anders.roxell@linaro.org>

> ---

>   kernel/dma/debug.c | 1 +

>   1 file changed, 1 insertion(+)

> 

> diff --git a/kernel/dma/debug.c b/kernel/dma/debug.c

> index 231ca4628062..2abdc265aec4 100644

> --- a/kernel/dma/debug.c

> +++ b/kernel/dma/debug.c

> @@ -767,6 +767,7 @@ static int prealloc_memory(u32 num_entries)

>   	int i;

>   

>   	for (i = 0; i < num_entries; ++i) {

> +		cond_resched();

>   		entry = kzalloc(sizeof(*entry), GFP_KERNEL);

>   		if (!entry)

>   			goto out_err;

>
Anders Roxell Dec. 12, 2018, 6:15 p.m. | #2
On Tue, 11 Dec 2018 at 12:51, Robin Murphy <robin.murphy@arm.com> wrote:
>

> Hi Anders,


Hi Robin,

>

> On 11/12/2018 10:36, Anders Roxell wrote:

> > When running a kernel in qemu with enough debugging options (slub-debug,

> > ftrace, kcov, kasan, ubsan, ...) enabled together, that results in a

> > slow initcall. So a 'watchdog: BUG: soft lockup' happens:

> >

> > [   44.105619] Call trace:

> > [   44.106709]  __slab_alloc+0x70/0x88

> > [   44.107757]  kmem_cache_alloc_trace+0x13c/0x590

> > [   44.108764]  prealloc_memory+0xb8/0x240

> > [   44.109768]  dma_debug_init+0x174/0x1f0

> > [   44.110725]  do_one_initcall+0x430/0x8b0

> > [   44.111719]  do_initcall_level+0x548/0x5b4

> > [   44.112672]  do_initcalls+0x28/0x4c

> > [   44.113628]  do_basic_setup+0x34/0x3c

> > [   44.114559]  kernel_init_freeable+0x194/0x24c

> > [   44.115570]  kernel_init+0x24/0x18c

> > [   44.116498]  ret_from_fork+0x10/0x18

> >

> > Rework to call cond_resched(), in the function prealloc_memory()'s

> > for-loop.

>

> Funnily enough I've just done a bit of work in this area which should

> make allocation a lot more efficient - would you mind giving this series

> a spin on the same setup to see how much it helps the performance?

>

> https://lore.kernel.org/lkml/cover.1544447601.git.robin.murphy@arm.com/


I tried it on todays next tag and couldn't reproduce the soft lockup.

Cheers,
Anders

>

> Thanks,

> Robin.

>

> >

> > Signed-off-by: Anders Roxell <anders.roxell@linaro.org>

> > ---

> >   kernel/dma/debug.c | 1 +

> >   1 file changed, 1 insertion(+)

> >

> > diff --git a/kernel/dma/debug.c b/kernel/dma/debug.c

> > index 231ca4628062..2abdc265aec4 100644

> > --- a/kernel/dma/debug.c

> > +++ b/kernel/dma/debug.c

> > @@ -767,6 +767,7 @@ static int prealloc_memory(u32 num_entries)

> >       int i;

> >

> >       for (i = 0; i < num_entries; ++i) {

> > +             cond_resched();

> >               entry = kzalloc(sizeof(*entry), GFP_KERNEL);

> >               if (!entry)

> >                       goto out_err;

> >

Patch

diff --git a/kernel/dma/debug.c b/kernel/dma/debug.c
index 231ca4628062..2abdc265aec4 100644
--- a/kernel/dma/debug.c
+++ b/kernel/dma/debug.c
@@ -767,6 +767,7 @@  static int prealloc_memory(u32 num_entries)
 	int i;
 
 	for (i = 0; i < num_entries; ++i) {
+		cond_resched();
 		entry = kzalloc(sizeof(*entry), GFP_KERNEL);
 		if (!entry)
 			goto out_err;