From patchwork Mon Mar 21 14:21:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 553410 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D144BC433F5 for ; Mon, 21 Mar 2022 14:28:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349350AbiCUOaF (ORCPT ); Mon, 21 Mar 2022 10:30:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349264AbiCUO1x (ORCPT ); Mon, 21 Mar 2022 10:27:53 -0400 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D9C81EAE4; Mon, 21 Mar 2022 07:21:38 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id A92A15C01C7; Mon, 21 Mar 2022 10:21:37 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 21 Mar 2022 10:21:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to :reply-to:sender:subject:subject:to:to; s=fm3; bh=+/Wp7/w8DInb8D 6QqZ6ciS73TAw3rM4RvE2nNIS9rTw=; b=nJwqj/l0JgzTMam8BnG9lxPaM1+x+A 7+BLwRD6YpZFX/4SGSM+zGhlBll/868j4rmMy4wHqx6S/Mhbm7N4MK9SIMk07/Sv lyV+mK7PvJPSi1ceMe7ksQxi1gAqjbX2F6kwzIqpktkQ097Urt1FGrX6qyRmQFrb Z0WWiVBSIEKHUEFI+zxb2+jVSYMR63BVvIgP+hRfHbUbih+D4S49xQfOrtvmg0lG jrFkgswGEkYcymBxevlpmm5BnAbwNvr4/FGF3as2cIL1LiusHSWPeYd9wYbYMiVh QSe0r8SltxpC4q3F4ZwVLFEBH2mUZrEyMVH1EGfftoTNrIr/UP+qIccQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=+/Wp7/w8DInb8D6QqZ6ciS73TAw3rM4RvE2nNIS9rTw=; b=o1C99GeY L7HpALqGAWJawa5uOAHc04OvriLZIsh8ZFIgj1UAYmVMs+7Hb062PRIlyk1gSdH2 De5V7tZVHYQ9VhlQzoBzbMk1A0qyRk8pZG1Vc91b6Rp5vrdGwVAiEnX/U5siTzco iQwRe0WmSn1VmRcjQ6AdmrktO7DtR/V01OWycm0Usw22pj8SbSh/OEkEZHn4UlXJ 0W4zDMwokwtIEgvEaUwU9PzDQ5AAoaSCqi4PSG7MvOuuMWhvj3ni+DXFNfrbId3x 0t9+SjJ6a3yBxWwAzeuHBijO12paNSSff/whVfOHeuRtl80fluF8lC7CWV10ba0A bSP98/ubPnwGlA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgieegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepieejue dvueduuefhgefhheeiuedvtedvuefgieegveetueeiueehtdegudehfeelnecuvehluhhs thgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnsehsvg hnthdrtghomh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 10:21:37 -0400 (EDT) From: Zi Yan To: Matthew Wilcox , linux-mm@kvack.org Cc: Roman Gushchin , Shuah Khan , Yang Shi , Miaohe Lin , Hugh Dickins , "Kirill A . Shutemov" , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, Zi Yan Subject: [RFC PATCH 2/5] mm: page_owner: add support for splitting to any order in split page_owner. Date: Mon, 21 Mar 2022 10:21:25 -0400 Message-Id: <20220321142128.2471199-3-zi.yan@sent.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220321142128.2471199-1-zi.yan@sent.com> References: <20220321142128.2471199-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org From: Zi Yan It adds a new_order parameter to set new page order in page owner and uses old_order instead of nr to make the parameters look consistent. It prepares for upcoming changes to support split huge page to any lower order. Signed-off-by: Zi Yan --- include/linux/page_owner.h | 12 +++++++----- mm/huge_memory.c | 3 ++- mm/page_alloc.c | 2 +- mm/page_owner.c | 13 +++++++------ 4 files changed, 17 insertions(+), 13 deletions(-) diff --git a/include/linux/page_owner.h b/include/linux/page_owner.h index 119a0c9d2a8b..16050cc89274 100644 --- a/include/linux/page_owner.h +++ b/include/linux/page_owner.h @@ -11,7 +11,8 @@ extern struct page_ext_operations page_owner_ops; extern void __reset_page_owner(struct page *page, unsigned short order); extern void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask); -extern void __split_page_owner(struct page *page, unsigned int nr); +extern void __split_page_owner(struct page *page, unsigned short old_order, + unsigned short new_order); extern void __folio_copy_owner(struct folio *newfolio, struct folio *old); extern void __set_page_owner_migrate_reason(struct page *page, int reason); extern void __dump_page_owner(const struct page *page); @@ -31,10 +32,11 @@ static inline void set_page_owner(struct page *page, __set_page_owner(page, order, gfp_mask); } -static inline void split_page_owner(struct page *page, unsigned int nr) +static inline void split_page_owner(struct page *page, unsigned int old_order, + unsigned int new_order) { if (static_branch_unlikely(&page_owner_inited)) - __split_page_owner(page, nr); + __split_page_owner(page, old_order, new_order); } static inline void folio_copy_owner(struct folio *newfolio, struct folio *old) { @@ -56,11 +58,11 @@ static inline void reset_page_owner(struct page *page, unsigned short order) { } static inline void set_page_owner(struct page *page, - unsigned int order, gfp_t gfp_mask) + unsigned short order, gfp_t gfp_mask) { } static inline void split_page_owner(struct page *page, - unsigned short order) + unsigned short old_order, unsigned short new_order) { } static inline void folio_copy_owner(struct folio *newfolio, struct folio *folio) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 640040c386f0..fcfa46af6c4c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2367,6 +2367,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct lruvec *lruvec; struct address_space *swap_cache = NULL; unsigned long offset = 0; + unsigned int order = thp_order(head); unsigned int nr = thp_nr_pages(head); int i; @@ -2408,7 +2409,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, unlock_page_lruvec(lruvec); /* Caller disabled irqs, so they are still disabled here */ - split_page_owner(head, nr); + split_page_owner(head, order, 0); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d982919b9e51..9cac40c26c58 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3514,7 +3514,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); - split_page_owner(page, 1 << order); + split_page_owner(page, order, 0); split_page_memcg(page, 1 << order, 0); } EXPORT_SYMBOL_GPL(split_page); diff --git a/mm/page_owner.c b/mm/page_owner.c index 0a9588506571..52013c846d19 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -202,19 +202,20 @@ void __set_page_owner_migrate_reason(struct page *page, int reason) page_owner->last_migrate_reason = reason; } -void __split_page_owner(struct page *page, unsigned int nr) +void __split_page_owner(struct page *page, unsigned short old_order, + unsigned short new_order) { - int i; - struct page_ext *page_ext = lookup_page_ext(page); + int i, old_nr = 1 << old_order, new_nr = 1 << new_order; + struct page_ext *page_ext; struct page_owner *page_owner; if (unlikely(!page_ext)) return; - for (i = 0; i < nr; i++) { + for (i = 0; i < old_nr; i += new_nr) { + page_ext = lookup_page_ext(page + i); page_owner = get_page_owner(page_ext); - page_owner->order = 0; - page_ext = page_ext_next(page_ext); + page_owner->order = new_order; } } From patchwork Mon Mar 21 14:21:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 553411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 547B3C433EF for ; Mon, 21 Mar 2022 14:28:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349471AbiCUOaA (ORCPT ); Mon, 21 Mar 2022 10:30:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349269AbiCUO1x (ORCPT ); Mon, 21 Mar 2022 10:27:53 -0400 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C6621FCDE; Mon, 21 Mar 2022 07:21:39 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id EA8175C01CE; Mon, 21 Mar 2022 10:21:38 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 21 Mar 2022 10:21:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to :reply-to:sender:subject:subject:to:to; s=fm3; bh=kIpuEveA7/YbA0 pSxKdFASwEtWdRLdBg+ont8QAUBuo=; b=CibfHPfPc9bJSaTTlmtIw04wVD3Dd8 3VqLqKQ5Qj071muc/kNIs2STVuogk3JFo13FuAM03CbTtbgDmccuFLrp16kcWIFO d8hl8FQdMKdjmKXPCk8wtZrl7DxcvoIkEU6WPV5hxZfDz6lMpV6fIMjLkx/76nna EiQGdMJQ4AHaL7eEaHpuGV9NSE8Jz4Uyw4k7ck2Xv9E4nyia9/ltn8+f8P4NCO4S pWq2hAlxRldvFm0hWFywi+rVo9wSbo+fCZjdR9azvHjTXQYxlq5iIoHuX4RkIEVm FQSLn+aFimBxOseDQcxdLoDoRcSy19BnL6Vr8ZjP8+p6fihROf/0XYog== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=kIpuEveA7/YbA0pSxKdFASwEtWdRLdBg+ont8QAUBuo=; b=E/noCP1M yVMAjJAqn0BOQsC1e5+7xwiXuijpihKmellqXIdOMkNlCUVSAxhx4Kd+SpYbrHZC 859nDk2ffs3FaMVJ2H8nYHlyBwQfulEJt82/xmqBk35GfQl17pHlh+LsRT5amo9F 9fHbMugbzGkL1bUPv04Em/0LCmEuKSST9ks7BXA+EtCtAejx77G71LwjzTDABnRk QC+4FeO+2n45rW13BcrSz+Rb+rhR+ykehSmSEa2+UEGc/q3br86XTzp0kPBFcpjg NGv2FBJh9ZEVa9hqzEd15pOAjSLI0fyr4n8FHDlffeWtOwjp5cEWta7I5OOKO5aj 5TsOJLkr5wx+Vw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgieegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfrhgggfestdhqredtredttdenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepieejue dvueduuefhgefhheeiuedvtedvuefgieegveetueeiueehtdegudehfeelnecuvehluhhs thgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepiihirdihrghnsehsvg hnthdrtghomh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 10:21:38 -0400 (EDT) From: Zi Yan To: Matthew Wilcox , linux-mm@kvack.org Cc: Roman Gushchin , Shuah Khan , Yang Shi , Miaohe Lin , Hugh Dickins , "Kirill A . Shutemov" , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org, Zi Yan Subject: [RFC PATCH 4/5] mm: truncate: split huge page cache page to a non-zero order if possible. Date: Mon, 21 Mar 2022 10:21:27 -0400 Message-Id: <20220321142128.2471199-5-zi.yan@sent.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220321142128.2471199-1-zi.yan@sent.com> References: <20220321142128.2471199-1-zi.yan@sent.com> Reply-To: Zi Yan MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org From: Zi Yan To minimize the number of pages after a huge page truncation, we do not need to split it all the way down to order-0. The huge page has at most three parts, the part before offset, the part to be truncated, the part remaining at the end. Find the greatest common power of two multiplier of the non-zero values of them as the new order, so we can split the huge page to this order and keep the remaining pages as large and as few as possible. Signed-off-by: Zi Yan --- mm/huge_memory.c | 1 + mm/truncate.c | 33 +++++++++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3617aa3ad0b1..76db0092a1e2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2349,6 +2349,7 @@ static void __split_huge_page_tail(struct page *head, int tail, prep_compound_page(page_tail, new_order); prep_transhuge_page(page_tail); } + VM_BUG_ON_PAGE(PageTail(page_tail), page_tail); /* Finally unfreeze refcount. Additional reference from page cache. */ page_ref_unfreeze(page_tail, 1 + ((!PageAnon(head) || diff --git a/mm/truncate.c b/mm/truncate.c index ab50d0d59a2a..4f71e67dec09 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -197,6 +197,14 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio) return 0; } +static unsigned int greatest_pow_of_two_multiplier(unsigned int num) +{ + if (num & 1) + return 0; + return min_t(unsigned int, ilog2(num), + ilog2(num - rounddown_pow_of_two(num))); +} + /* * Handle partial folios. The folio may be entirely within the * range if a split has raced with us. If not, we zero the part of the @@ -211,7 +219,8 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio) bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) { loff_t pos = folio_pos(folio); - unsigned int offset, length; + unsigned int offset, length, remaining; + unsigned int new_order = folio_order(folio); if (pos < start) offset = start - pos; @@ -222,6 +231,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) length = length - offset; else length = end + 1 - pos - offset; + remaining = folio_size(folio) - offset - length; folio_wait_writeback(folio); if (length == folio_size(folio)) { @@ -236,11 +246,30 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) */ folio_zero_range(folio, offset, length); + /* + * Find the greatest common power of two multiplier of the non-zero + * offset, length, and remaining as the new order. So we can truncate + * a subpage as large as possible. + */ + if (offset) + new_order = greatest_pow_of_two_multiplier(offset / PAGE_SIZE); + if (length) + new_order = min_t(unsigned int, new_order, + greatest_pow_of_two_multiplier(length / PAGE_SIZE)); + if (remaining) + new_order = min_t(unsigned int, new_order, + greatest_pow_of_two_multiplier(remaining / PAGE_SIZE)); + + /* order-1 THP not supported, downgrade to order-0 */ + if (new_order == 1) + new_order = 0; + + if (folio_has_private(folio)) folio_invalidate(folio, offset, length); if (!folio_test_large(folio)) return true; - if (split_huge_page(&folio->page) == 0) + if (split_huge_page_to_list_to_order(&folio->page, NULL, new_order) == 0) return true; if (folio_test_dirty(folio)) return false;