From patchwork Thu May 25 15:42:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 100516 Delivered-To: patch@linaro.org Received: by 10.140.96.100 with SMTP id j91csp824233qge; Thu, 25 May 2017 08:42:49 -0700 (PDT) X-Received: by 10.99.100.129 with SMTP id y123mr45327913pgb.217.1495726969686; Thu, 25 May 2017 08:42:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1495726969; cv=none; d=google.com; s=arc-20160816; b=otzyRRKSknjUIcfKV2sRzbKRw4O1lb8GD10zDjduz9bW+oKit3rrQkjgHpxUntL/+H q7V8Ws5IPHcIX6mNcljH24JQn0L41bpx6sGzrA5MyqBpuMCmX75SWZ9e0RZ8hA9gnYns 5OYcET1DTHk5/oK/IlRxXIx5V9KWDtAcwn7naV/UjAuw4VOHDV1K2JqOj1zYNhH5SMte lTLjnWoQ1Kh9qPdB8wie+hERGsvKZwYAFyqRL/3YRYjOV+y0SQIMdaqfxcnwCaofJIBl NQECVmJ2e+QbpjB0nATdq52p9VJ4M/DP19bpHHlyFyUe5kRmEfD6CJM7cHe3VSwAsOUE 2+QQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=bcbtj5TMY8QV3VjG/1A98vWrjFfVPMKV3+U3WVGZB1w=; b=XyMMRuSGOGD3Ne6WggOjOxaYkH550PtQfniIkidmYufJqKETQu3fJP7qHH8aX6ueA0 EVl9NbOWls6Xdf0I8US4bxi3372b5pUdHr60zzfp2b2O9MkryQ53L9dlI62gfPFkBXEL PVGKs595EAtRUAEWOX9+GL24sjiIrkh8hwuYfqVw1jRw5ausTzMx05wgSioRCJuQYkCx 33yyLU/QZxmOrP7QfhL1LglQRs0MjX0vCcDfcHd48ECbfBSFCp8mIxSeGtqoQbK271oY gLm2zQVrLYy5S6oTFily59+p19/pisXDQAGOf/sw8CuWtJx758obONLKHvUNh2M6fUPq 8oOw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o123si28886685pfb.32.2017.05.25.08.42.49; Thu, 25 May 2017 08:42:49 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S942391AbdEYPm2 (ORCPT + 25 others); Thu, 25 May 2017 11:42:28 -0400 Received: from foss.arm.com ([217.140.101.70]:51354 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S938727AbdEYPm1 (ORCPT ); Thu, 25 May 2017 11:42:27 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B00E71596; Thu, 25 May 2017 08:42:26 -0700 (PDT) Received: from e104818-lin.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 783793F41F; Thu, 25 May 2017 08:42:25 -0700 (PDT) From: Catalin Marinas To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Michal Hocko , Andy Lutomirski , "Luis R. Rodriguez" , Andrew Morton Subject: [PATCH v2 1/3] mm: kmemleak: Slightly reduce the size of some structures on 64-bit architectures Date: Thu, 25 May 2017 16:42:15 +0100 Message-Id: <1495726937-23557-2-git-send-email-catalin.marinas@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1495726937-23557-1-git-send-email-catalin.marinas@arm.com> References: <1495726937-23557-1-git-send-email-catalin.marinas@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch changes the kmemleak_object.flags type to unsigned int and moves the early_log.min_count (int) near early_log.op_type (int) to slightly reduce the size of these structures on 64-bit architectures. Cc: Michal Hocko Cc: Andy Lutomirski Cc: "Luis R. Rodriguez" Signed-off-by: Catalin Marinas --- mm/kmemleak.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 20036d4f9f13..964b12eba2c1 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -150,7 +150,7 @@ struct kmemleak_scan_area { */ struct kmemleak_object { spinlock_t lock; - unsigned long flags; /* object status flags */ + unsigned int flags; /* object status flags */ struct list_head object_list; struct list_head gray_list; struct rb_node rb_node; @@ -262,9 +262,9 @@ enum { */ struct early_log { int op_type; /* kmemleak operation type */ + int min_count; /* minimum reference count */ const void *ptr; /* allocated/freed memory block */ size_t size; /* memory block size */ - int min_count; /* minimum reference count */ unsigned long trace[MAX_TRACE]; /* stack trace */ unsigned int trace_len; /* stack trace length */ }; @@ -393,7 +393,7 @@ static void dump_object_info(struct kmemleak_object *object) object->comm, object->pid, object->jiffies); pr_notice(" min_count = %d\n", object->min_count); pr_notice(" count = %d\n", object->count); - pr_notice(" flags = 0x%lx\n", object->flags); + pr_notice(" flags = 0x%x\n", object->flags); pr_notice(" checksum = %u\n", object->checksum); pr_notice(" backtrace:\n"); print_stack_trace(&trace, 4); From patchwork Thu May 25 15:42:16 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 100518 Delivered-To: patch@linaro.org Received: by 10.140.96.100 with SMTP id j91csp824307qge; Thu, 25 May 2017 08:42:59 -0700 (PDT) X-Received: by 10.98.93.217 with SMTP id n86mr45954111pfj.113.1495726978794; Thu, 25 May 2017 08:42:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1495726978; cv=none; d=google.com; s=arc-20160816; b=vmGMYshz+aeDEWHSYq4h/s7GTtB42o0SKb2Ya2omZvV17MoX5PKotjU9BSua9Bt9h6 ReAR3E9DobqLAebkiTir1bTztJb764xSdTb+xjCsCp3w/ns3oN9XaeGRZrdLGPLSA1aZ Ae4Xjv3Q41YCFUrcc+P0a17g0053iXwOHmWcqT3NlCmuPN6ElQzQXMrL9WvKIGdu/t/7 I4F+6tPOne5rymqiQvOvJ+ZbF22dOcK3N5OjBTxTyxCygu/ylVmdJ6kKd2pfL1me80Gt mcjvd7FFOCPbaRCy5EFyluGW8EbnnOyf/h4/3rA2Iek9jv87NROPyiy9UzyOspNTmSS7 nHAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=AJL3dchf55zUdzNawMtX5SfQ0r5h5QNx+Y9paII/36M=; b=EQmwoqOTkKyrteoFYyZGMGGc3Jtz1ytWl3xXDLRzM4Fedn8lZMN7IjMBKbZHflNxLT g4w8sUVsPaIZcbHb7+fRtdBi6gZ+ilBbIUp3HQEnd7NIjcoPjtIxR+1GiFIaXWXTWmk9 HcAIRSxJe0RK6B7I7dfOMIcE94SEUI8kqo63f2p/M9qC2wL0YtuLN0gaR9gt+nwGMbsy bPBuTBfEpb6+H9rfSDeQpIbUrzrbdRm9/ZuTcZjIpwRSoqJNhSIXEx2O9C9SCPNfZ0Ny OS3LNJ+PhJPcPftTAKOl5o0G3x34xd8xj/zqtd4YzaQly29VG1An1CkYeyT5dE/ZEL9x aGGw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m23si6536710pli.111.2017.05.25.08.42.58; Thu, 25 May 2017 08:42:58 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S942452AbdEYPms (ORCPT + 25 others); Thu, 25 May 2017 11:42:48 -0400 Received: from foss.arm.com ([217.140.101.70]:51358 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S942385AbdEYPm2 (ORCPT ); Thu, 25 May 2017 11:42:28 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 33C14344; Thu, 25 May 2017 08:42:28 -0700 (PDT) Received: from e104818-lin.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id EF9943F41F; Thu, 25 May 2017 08:42:26 -0700 (PDT) From: Catalin Marinas To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Michal Hocko , Andy Lutomirski , "Luis R. Rodriguez" , Andrew Morton Subject: [PATCH v2 2/3] mm: kmemleak: Factor object reference updating out of scan_block() Date: Thu, 25 May 2017 16:42:16 +0100 Message-Id: <1495726937-23557-3-git-send-email-catalin.marinas@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1495726937-23557-1-git-send-email-catalin.marinas@arm.com> References: <1495726937-23557-1-git-send-email-catalin.marinas@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The scan_block() function updates the number of references (pointers) to objects, adding them to the gray_list when object->min_count is reached. The patch factors out this functionality into a separate update_refs() function. Cc: Michal Hocko Cc: Andy Lutomirski Cc: "Luis R. Rodriguez" Signed-off-by: Catalin Marinas --- mm/kmemleak.c | 43 +++++++++++++++++++++++++------------------ 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 964b12eba2c1..266482f460c2 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1188,6 +1188,30 @@ static bool update_checksum(struct kmemleak_object *object) } /* + * Update an object's references. object->lock must be held by the caller. + */ +static void update_refs(struct kmemleak_object *object) +{ + if (!color_white(object)) { + /* non-orphan, ignored or new */ + return; + } + + /* + * Increase the object's reference count (number of pointers to the + * memory block). If this count reaches the required minimum, the + * object's color will become gray and it will be added to the + * gray_list. + */ + object->count++; + if (color_gray(object)) { + /* put_object() called when removing from gray_list */ + WARN_ON(!get_object(object)); + list_add_tail(&object->gray_list, &gray_list); + } +} + +/* * Memory scanning is a long process and it needs to be interruptable. This * function checks whether such interrupt condition occurred. */ @@ -1259,24 +1283,7 @@ static void scan_block(void *_start, void *_end, * enclosed by scan_mutex. */ spin_lock_nested(&object->lock, SINGLE_DEPTH_NESTING); - if (!color_white(object)) { - /* non-orphan, ignored or new */ - spin_unlock(&object->lock); - continue; - } - - /* - * Increase the object's reference count (number of pointers - * to the memory block). If this count reaches the required - * minimum, the object's color will become gray and it will be - * added to the gray_list. - */ - object->count++; - if (color_gray(object)) { - /* put_object() called when removing from gray_list */ - WARN_ON(!get_object(object)); - list_add_tail(&object->gray_list, &gray_list); - } + update_refs(object); spin_unlock(&object->lock); } read_unlock_irqrestore(&kmemleak_lock, flags); From patchwork Thu May 25 15:42:17 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 100517 Delivered-To: patch@linaro.org Received: by 10.140.96.100 with SMTP id j91csp824246qge; Thu, 25 May 2017 08:42:51 -0700 (PDT) X-Received: by 10.99.61.134 with SMTP id k128mr45920632pga.201.1495726971392; Thu, 25 May 2017 08:42:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1495726971; cv=none; d=google.com; s=arc-20160816; b=chOkuYH1Ca9r+/mWj9w4f3WMybV/4AwcGHqlP0C1GWt+s2aS2jMnCDlvmHrEl6M4z8 hDQQ6s/+J/tGcQQq06Fp32tTL5qa922b9j2NDi7HGml9t3els89FQFKz8jDCzJ53aF4c 1X7+I98a575S0P0044GxPQbvX1CuclgfgPfkA0N4wKEHyzneS6pMuMYhQJJNoXTH8WEo Guc5Im5N2OrspuOiqzWKLsDs3cYoue8mGFl9klKWeyUnT0QHfvoE4TLkT8r6fsMXgIDV eaPnabrfnJVC2wVAchnRsjGcIJAxnYJdmOyCa8D+RXuVBkJyBWxjY0LjKZK0KhlG74zB t5/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=suRKdtodrgxj7qwZ7xOZwCV+zfmApF1uSoOfV43Tpc0=; b=ag/vTKwCKOBdjZKinaa8/DvB/pSie4G+RmCwr/hMjDv5X8mAyBF+tvVP6zg6UMBP5N ptYJjMRARb8dLJ4jTS7iZyRN3LSLtoe/8XpSR7q9O+EnscYqjZQ02uk11E9K0by0gJjv GJr4b63RtxWDVmKRHcIwdAa+ZAc9sn3Yh4w6lsXLfI1kV7gyhnK1uoX+5uJT0ROT147F ZcCgpnGZY7XrS0OvkdcfodCUkeOlNJ6MGPpYftiOSCi+NiETV20eNlPfbe1nH5YHvCc9 vahL7JypaOppmw5s8TxQelAv++dC5z0igmII65J3xhoT4szHy81J/Ydh20Vsq1R79/xm Dwhw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o123si28886685pfb.32.2017.05.25.08.42.50; Thu, 25 May 2017 08:42:51 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S942437AbdEYPmq (ORCPT + 25 others); Thu, 25 May 2017 11:42:46 -0400 Received: from foss.arm.com ([217.140.101.70]:51370 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S942436AbdEYPmk (ORCPT ); Thu, 25 May 2017 11:42:40 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A946B1596; Thu, 25 May 2017 08:42:29 -0700 (PDT) Received: from e104818-lin.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 721B53F41F; Thu, 25 May 2017 08:42:28 -0700 (PDT) From: Catalin Marinas To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Michal Hocko , Andy Lutomirski , "Luis R. Rodriguez" , Andrew Morton Subject: [PATCH v2 3/3] mm: kmemleak: Treat vm_struct as alternative reference to vmalloc'ed objects Date: Thu, 25 May 2017 16:42:17 +0100 Message-Id: <1495726937-23557-4-git-send-email-catalin.marinas@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1495726937-23557-1-git-send-email-catalin.marinas@arm.com> References: <1495726937-23557-1-git-send-email-catalin.marinas@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Kmemleak requires that vmalloc'ed objects have a minimum reference count of 2: one in the corresponding vm_struct object and the other owned by the vmalloc() caller. There are cases, however, where the original vmalloc() returned pointer is lost and, instead, a pointer to vm_struct is stored (see free_thread_stack()). Kmemleak currently reports such objects as leaks. This patch adds support for treating any surplus references to an object as additional references to a specified object. It introduces the kmemleak_vmalloc() API function which takes a vm_struct pointer and sets its surplus reference passing to the actual vmalloc() returned pointer. The __vmalloc_node_range() calling site has been modified accordingly. Cc: Michal Hocko Cc: Andy Lutomirski Cc: "Luis R. Rodriguez" Reported-by: "Luis R. Rodriguez" Signed-off-by: Catalin Marinas --- Documentation/dev-tools/kmemleak.rst | 1 + include/linux/kmemleak.h | 7 +++ mm/kmemleak.c | 93 ++++++++++++++++++++++++++++++++++-- mm/vmalloc.c | 7 +-- 4 files changed, 98 insertions(+), 10 deletions(-) diff --git a/Documentation/dev-tools/kmemleak.rst b/Documentation/dev-tools/kmemleak.rst index b2391b829169..cb8862659178 100644 --- a/Documentation/dev-tools/kmemleak.rst +++ b/Documentation/dev-tools/kmemleak.rst @@ -150,6 +150,7 @@ See the include/linux/kmemleak.h header for the functions prototype. - ``kmemleak_init`` - initialize kmemleak - ``kmemleak_alloc`` - notify of a memory block allocation - ``kmemleak_alloc_percpu`` - notify of a percpu memory block allocation +- ``kmemleak_vmalloc`` - notify of a vmalloc() memory allocation - ``kmemleak_free`` - notify of a memory block freeing - ``kmemleak_free_part`` - notify of a partial memory block freeing - ``kmemleak_free_percpu`` - notify of a percpu memory block freeing diff --git a/include/linux/kmemleak.h b/include/linux/kmemleak.h index 1c2a32829620..590343f6c1b1 100644 --- a/include/linux/kmemleak.h +++ b/include/linux/kmemleak.h @@ -22,6 +22,7 @@ #define __KMEMLEAK_H #include +#include #ifdef CONFIG_DEBUG_KMEMLEAK @@ -30,6 +31,8 @@ extern void kmemleak_alloc(const void *ptr, size_t size, int min_count, gfp_t gfp) __ref; extern void kmemleak_alloc_percpu(const void __percpu *ptr, size_t size, gfp_t gfp) __ref; +extern void kmemleak_vmalloc(const struct vm_struct *area, size_t size, + gfp_t gfp) __ref; extern void kmemleak_free(const void *ptr) __ref; extern void kmemleak_free_part(const void *ptr, size_t size) __ref; extern void kmemleak_free_percpu(const void __percpu *ptr) __ref; @@ -81,6 +84,10 @@ static inline void kmemleak_alloc_percpu(const void __percpu *ptr, size_t size, gfp_t gfp) { } +static inline void kmemleak_vmalloc(const struct vm_struct *area, size_t size, + gfp_t gfp) +{ +} static inline void kmemleak_free(const void *ptr) { } diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 266482f460c2..7780cd83a495 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -159,6 +159,8 @@ struct kmemleak_object { atomic_t use_count; unsigned long pointer; size_t size; + /* pass surplus references to this pointer */ + unsigned long excess_ref; /* minimum number of a pointers found before it is considered leak */ int min_count; /* the total number of pointers found pointing to this object */ @@ -253,7 +255,8 @@ enum { KMEMLEAK_NOT_LEAK, KMEMLEAK_IGNORE, KMEMLEAK_SCAN_AREA, - KMEMLEAK_NO_SCAN + KMEMLEAK_NO_SCAN, + KMEMLEAK_SET_EXCESS_REF }; /* @@ -264,7 +267,10 @@ struct early_log { int op_type; /* kmemleak operation type */ int min_count; /* minimum reference count */ const void *ptr; /* allocated/freed memory block */ - size_t size; /* memory block size */ + union { + size_t size; /* memory block size */ + unsigned long excess_ref; /* surplus reference passing */ + }; unsigned long trace[MAX_TRACE]; /* stack trace */ unsigned int trace_len; /* stack trace length */ }; @@ -562,6 +568,7 @@ static struct kmemleak_object *create_object(unsigned long ptr, size_t size, object->flags = OBJECT_ALLOCATED; object->pointer = ptr; object->size = size; + object->excess_ref = 0; object->min_count = min_count; object->count = 0; /* white color initially */ object->jiffies = jiffies; @@ -795,6 +802,30 @@ static void add_scan_area(unsigned long ptr, size_t size, gfp_t gfp) } /* + * Any surplus references (object already gray) to 'ptr' are passed to + * 'excess_ref'. This is used in the vmalloc() case where a pointer to + * vm_struct may be used as an alternative reference to the vmalloc'ed object + * (see free_thread_stack()). + */ +static void object_set_excess_ref(unsigned long ptr, unsigned long excess_ref) +{ + unsigned long flags; + struct kmemleak_object *object; + + object = find_and_get_object(ptr, 0); + if (!object) { + kmemleak_warn("Setting excess_ref on unknown object at 0x%08lx\n", + ptr); + return; + } + + spin_lock_irqsave(&object->lock, flags); + object->excess_ref = excess_ref; + spin_unlock_irqrestore(&object->lock, flags); + put_object(object); +} + +/* * Set the OBJECT_NO_SCAN flag for the object corresponding to the give * pointer. Such object will not be scanned by kmemleak but references to it * are searched. @@ -908,7 +939,7 @@ static void early_alloc_percpu(struct early_log *log) * @gfp: kmalloc() flags used for kmemleak internal memory allocations * * This function is called from the kernel allocators when a new object - * (memory block) is allocated (kmem_cache_alloc, kmalloc, vmalloc etc.). + * (memory block) is allocated (kmem_cache_alloc, kmalloc etc.). */ void __ref kmemleak_alloc(const void *ptr, size_t size, int min_count, gfp_t gfp) @@ -952,6 +983,36 @@ void __ref kmemleak_alloc_percpu(const void __percpu *ptr, size_t size, EXPORT_SYMBOL_GPL(kmemleak_alloc_percpu); /** + * kmemleak_vmalloc - register a newly vmalloc'ed object + * @area: pointer to vm_struct + * @size: size of the object + * @gfp: __vmalloc() flags used for kmemleak internal memory allocations + * + * This function is called from the vmalloc() kernel allocator when a new + * object (memory block) is allocated. + */ +void __ref kmemleak_vmalloc(const struct vm_struct *area, size_t size, gfp_t gfp) +{ + pr_debug("%s(0x%p, %zu)\n", __func__, area, size); + + /* + * A min_count = 2 is needed because vm_struct contains a reference to + * the virtual address of the vmalloc'ed block. + */ + if (kmemleak_enabled) { + create_object((unsigned long)area->addr, size, 2, gfp); + object_set_excess_ref((unsigned long)area, + (unsigned long)area->addr); + } else if (kmemleak_early_log) { + log_early(KMEMLEAK_ALLOC, area->addr, size, 2); + /* reusing early_log.size for storing area->addr */ + log_early(KMEMLEAK_SET_EXCESS_REF, + area, (unsigned long)area->addr, 0); + } +} +EXPORT_SYMBOL_GPL(kmemleak_vmalloc); + +/** * kmemleak_free - unregister a previously registered object * @ptr: pointer to beginning of the object * @@ -1248,6 +1309,7 @@ static void scan_block(void *_start, void *_end, for (ptr = start; ptr < end; ptr++) { struct kmemleak_object *object; unsigned long pointer; + unsigned long excess_ref; if (scan_should_stop()) break; @@ -1283,8 +1345,27 @@ static void scan_block(void *_start, void *_end, * enclosed by scan_mutex. */ spin_lock_nested(&object->lock, SINGLE_DEPTH_NESTING); - update_refs(object); + /* only pass surplus references (object already gray) */ + if (color_gray(object)) { + excess_ref = object->excess_ref; + /* no need for update_refs() if object already gray */ + } else { + excess_ref = 0; + update_refs(object); + } spin_unlock(&object->lock); + + if (excess_ref) { + object = lookup_object(excess_ref, 0); + if (!object) + continue; + if (object == scanned) + /* circular reference, ignore */ + continue; + spin_lock_nested(&object->lock, SINGLE_DEPTH_NESTING); + update_refs(object); + spin_unlock(&object->lock); + } } read_unlock_irqrestore(&kmemleak_lock, flags); } @@ -1987,6 +2068,10 @@ void __init kmemleak_init(void) case KMEMLEAK_NO_SCAN: kmemleak_no_scan(log->ptr); break; + case KMEMLEAK_SET_EXCESS_REF: + object_set_excess_ref((unsigned long)log->ptr, + log->excess_ref); + break; default: kmemleak_warn("Unknown early log operation: %d\n", log->op_type); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 34a1c3e46ed7..b805cc5ecca0 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1759,12 +1759,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, */ clear_vm_uninitialized_flag(area); - /* - * A ref_count = 2 is needed because vm_struct allocated in - * __get_vm_area_node() contains a reference to the virtual address of - * the vmalloc'ed block. - */ - kmemleak_alloc(addr, real_size, 2, gfp_mask); + kmemleak_vmalloc(area, size, gfp_mask); return addr;