From patchwork Tue Aug 2 09:58:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Olivier Masse X-Patchwork-Id: 599964 Delivered-To: patches@linaro.org Received: by 2002:a05:7000:4388:0:0:0:0 with SMTP id w8csp290098mae; Thu, 25 Aug 2022 06:29:53 -0700 (PDT) X-Google-Smtp-Source: AA6agR6LiJfl9Cwik7l9EaWJTmiqjZyy7QaEQto6pplmzlur+lXE855KJcoNMYNXTKI0MBdZePOm X-Received: by 2002:a05:622a:110:b0:344:c2fd:dc33 with SMTP id u16-20020a05622a011000b00344c2fddc33mr3556095qtw.317.1661434193748; Thu, 25 Aug 2022 06:29:53 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [3.208.193.21]) by mx.google.com with ESMTPS id f9-20020a05620a408900b006bba7415065si11482236qko.679.2022.08.25.06.29.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 06:29:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linaro-mm-sig-bounces+patches=linaro.org@lists.linaro.org designates 3.208.193.21 as permitted sender) client-ip=3.208.193.21; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@nxp.com header.s=selector2 header.b=RMEpMqfh; arc=fail (body hash mismatch); spf=pass (google.com: domain of linaro-mm-sig-bounces+patches=linaro.org@lists.linaro.org designates 3.208.193.21 as permitted sender) smtp.mailfrom="linaro-mm-sig-bounces+patches=linaro.org@lists.linaro.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from lists.linaro.org (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 9A1203F60E for ; Thu, 25 Aug 2022 13:29:53 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2062.outbound.protection.outlook.com [40.107.20.62]) by lists.linaro.org (Postfix) with ESMTPS id BABBF4796B for ; Tue, 2 Aug 2022 09:58:52 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OrAAmiL71WJEZ2sYcwu162xdjsCfJz+mi7d/EviKKbG6KkKoS1cEZp57kwt5M2caT/5fpTNOpTgb/bGZfk1MP9rb8rwBHC8SPthFRrFj7TURgGMrd9QoBcQ7u8Q5JsFC3vO6bdtsyXSx/S0LUY/vvSGuNiDjLNMXH7dWTy34egH02ufX1MJs7m17IdVfV2lAkg077YbuVekb5YocaRpjjt3GrAS7L/4GwAzlPJBaeplyzwHouAFWbDsXMdlyqmPhg/B6N0e8iqmm9WYRa0biiTCbIcrItwyCIu5z+XgqmqhyQlqR1jEYLln5RWwGFow+u4wQ1T2TCMgYxV4rTf1Ywg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wXOdF3cE6IyAZrsT4wTECkt/yCciaBYokLHgvJeWwJ4=; b=ICi+ZmSbWO9+hfxeVNHXBoztbMHjnE80WQtf34gxy9x71ekVVLh30XZsqSTbzhl7usF+X49k5IFDa77pJ+lT4/JrtHcYvNkAXWsC2ncaUm1tOfAoXML21GdMMtHslNCzMQlysRMJ71JnWcuaHHU46SbuyIRIj8FMszSH1tfsAfZzrHZBX6pabLij5VzR9TALy+oUiAhAE5YrmQy6QdCjnbk89dKwM/aGgIpEqgdHst9Sg6fVCJzMBH6Twe48KwinSc7hgxxIqnRpy+qWwsy6iKn38F920sEucxDUufw3SDivAi2sQSLkAkZj1P/v1l4W/b6oFmBEgqqqHujBt5HfPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wXOdF3cE6IyAZrsT4wTECkt/yCciaBYokLHgvJeWwJ4=; b=RMEpMqfhlhwZTCCuJdBytFnkHQwT2TjkzRvZq6PvpJ9fHE7fMk1Z+YJ+wPLin72I8paK2y/07aO7+3PCARnLBFSovApPgxqoq943CA89gn2ThQAHvy5HxRRtv34iP6cq99TYsNJ9qK8vOCHsvn7adqElJK1h2xSnE2rbwmuruoA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from AM0PR04MB6737.eurprd04.prod.outlook.com (2603:10a6:208:176::18) by AS8PR04MB8341.eurprd04.prod.outlook.com (2603:10a6:20b:3b0::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5482.16; Tue, 2 Aug 2022 09:58:51 +0000 Received: from AM0PR04MB6737.eurprd04.prod.outlook.com ([fe80::6ca7:299e:d92f:7d1b]) by AM0PR04MB6737.eurprd04.prod.outlook.com ([fe80::6ca7:299e:d92f:7d1b%5]) with mapi id 15.20.5482.011; Tue, 2 Aug 2022 09:58:51 +0000 From: Olivier Masse To: sumit.semwal@linaro.org, benjamin.gaignard@collabora.com, Brian.Starkey@arm.com, christian.koenig@amd.com, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-kernel@vger.kernel.org Date: Tue, 2 Aug 2022 11:58:39 +0200 Message-Id: <20220802095843.14614-2-olivier.masse@nxp.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20220802095843.14614-1-olivier.masse@nxp.com> References: <20220802095843.14614-1-olivier.masse@nxp.com> X-ClientProxiedBy: AM3PR07CA0145.eurprd07.prod.outlook.com (2603:10a6:207:8::31) To AM0PR04MB6737.eurprd04.prod.outlook.com (2603:10a6:208:176::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a0696432-f15e-42e3-edcf-08da746d9994 X-MS-TrafficTypeDiagnostic: AS8PR04MB8341:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4j7UWFRmKfOkhZ9E1PD5Am1xX/1ykjbLzUYkFeGXDQPUQJZWkWm+30KxykNIVXKC2UCJGS9UXrL1PEkNpSwAjkjQSV3rqfohOudMgEoH9ewkbxdtDLdMoNDBuBAtnDazjUkdl3rrXVSeWwIPhOGVPKdoHoJt2etVAMQ7nBZZuarljoF/DZe2oeE+IO4neWq+V2vMCtEKikksoqSakAunl7qkTQd9uQaZgv8K5qoDoHOP1x7iAbukQMIikTgYfGT+QDq1S53EHZ74K9VHoOyc0rz4ibudKKG+YvArmyUu3+rn1qoIU0BR4PmrhNT04tkkIrbgUAgIL8u9Py/gznfjgqIMyzOWdLyCA2hSJWaKfS5y/jo6yiZEQOvCdxQ3NKsuboawC2YBrBo731d880r2TNnwxiV80fJXO6RrJZld6rYJ/9gWRfCmG+vxsECHK7Ku93Ik1ipTTOv2TQnuDC2kJoeBCwK5ad0g7/t9A/Z8VjFiJZXkAC8HuM5GmJuI0CPDH/m6MkgiWKMaWZcvoVe8AwtrL/gOkmg8urq9evZGoe51bnDV0p6Ga50zlXSEZy/79urTHD639M8S9pg2IznlyrlWrkmjqdMkzKRfGyThmgVb6gDN10rGsoYWPHQAze8jgs4BfoW/78MuJYLcMRCoLL6UgQRQIV/VfPbh+BxMSs0eJRZ1gO9/+5dlFezBc3Shf7djOd/jY4XYOxc8o8tjiwEMEGoR+H3T0vNs6YpwsDHAHtTON7dKcEWdSxLXW265/zI+hPdjQhJ2ld+jzYOP9T3bG1lIKQJLk4koeUditEo= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR04MB6737.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(396003)(346002)(366004)(136003)(39860400002)(376002)(41300700001)(2906002)(83380400001)(2616005)(86362001)(6666004)(52116002)(6506007)(44832011)(1076003)(36756003)(26005)(6512007)(38100700002)(186003)(38350700002)(6486002)(5660300002)(8676002)(66556008)(66476007)(66946007)(8936002)(478600001)(4326008)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?+hyDQfI/PnFnlP/s5tYkc50HD5bC?= =?utf-8?q?f1hR8LOQoZhOICQm+RpFZBf8D0ylkWekF/bnv88WPOnJYcJaYpo5ZX2u54ovsmkna?= =?utf-8?q?0nCqi2dPGcDcQd7iL77GlWtNYc1xcJ1YNg+RIIKPPn1TNmyGRm49Y6v33VGDsxTze?= =?utf-8?q?giA7+Q5zPxJZbHs0jW+fECWdD99g/+YGOuePjNQthhIb9PnCw8zp80toBH947jgHD?= =?utf-8?q?a9YsRfhyvRHNSN6t23yD8WCBSr+GKh52F1ZXSm5XRP7bdghtZeE2t+3+twJdRyiR2?= =?utf-8?q?iIV8ASYjw2HXorq/e6/FeIo8iP1ldObgcKFZMS/FfjIBj+zyQRFK6Wg7dQMsn+lKk?= =?utf-8?q?IJqLbnrdhg3CGXWlwqpGbqMR5J/UGItQ6lY85+cVjXbyuf4ae1eP0TyBskpQvXpTe?= =?utf-8?q?GsE2AwDqr3Gz5GYNrFXcB7QxUzbtZw0jqUFb7xf0Ffej8p5NfV+8hSf4z24k4atmn?= =?utf-8?q?jZNDD3b33Wr0V//N1W7UtGV2a8M5o1XRN02s+q5Ze5qyZcmXjuUIZUuFebTMwCJm3?= =?utf-8?q?l2gcDjAyqF0OtFW2E3QpSsj/2A5O6yxT52/uYVmAbusdID8FeV/ijaCHDUC6MTwwX?= =?utf-8?q?qUp7SkNdyxirqnimDeY9AdkPjxYHfHr4q5Al/qpRXi5mFOhLL9cB2PfxyqI7SD2B7?= =?utf-8?q?Py2Zc/Oxk1IUaIcVpFqV+bJHWRxUZt2yjODD1c3H+MgY0teGu3Oth0ThA2VpJ76DM?= =?utf-8?q?gzBGMUkHzwa56dVABfROk23Ew8xkw+kN7wa1lD5RYDk0U2DvqP7Ij4z8vCLZPrkwL?= =?utf-8?q?cbSelBIXB2xkfxcDAxFrARka7QVgPbWzAEanolswfUZ9CcoachdtdbIkmWevxAYp3?= =?utf-8?q?IlOrUMi3ZxSuGsCmN+RNb9ssosRt2lZsDZ/ZKFlf02OiNdykcUZTROnVQx+oRRSAl?= =?utf-8?q?ueYDJHaKGSNQpSNIe9HzXFDD3U+TIcG54fkdtY8LWP91qq0pFoEu9KojvzL5nwMJL?= =?utf-8?q?SDqEaaALcN/rfq1hkAC6dc5nIJe2zanTfsL+XazGNaO5dFxJUr1WsbRghULd6oc4Z?= =?utf-8?q?/cBpYublDARj8eAt480N8n0Fgh8HURl2C7ppsZ+TG4QO+7qJ2uC7+aqMuwmJDe2MC?= =?utf-8?q?I1d6L7im9aut7QnBIv0hD91ZpHlkpzyji0XkvaLgDtT5wtZ080P7EK9+kPprdPJZ6?= =?utf-8?q?oxjgvrHZqgSQaXwhGMIMsPE91VpC1P3bkcJ+UkfyBhes9F6i541fX0YK5X62ipeX0?= =?utf-8?q?IIazcWF/bWJq5YF4OKz4Y4K+4b4OZqOgyG9ILRsnAenQK+pzyDaGsq51Za8vSijig?= =?utf-8?q?E+upWDEiVEbkTXldsM8DCwOj1XBEWWeqmB09jwI2Z9Y+OK0CqyKcAsUDd4uH6PaxO?= =?utf-8?q?6mEHPsMDmeR323NChJOlnVYZJWjtdL2coqaV8NXk2lxid6YvZs7qSsPdrTP/bw9nd?= =?utf-8?q?sB6HgWw+hhoiSfnn2gzp6++bZyp+O6iiHEiyDOZcDLSyFv1DrjYhTsf3xpP2ELEjA?= =?utf-8?q?sClMmdij9o7zQeh+peQ9KwguNP3p4L5rsbKNVrTwSTH322atVZ/ii+uhXwqf8DPyR?= =?utf-8?q?k5szjkSEsnvH?= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: a0696432-f15e-42e3-edcf-08da746d9994 X-MS-Exchange-CrossTenant-AuthSource: AM0PR04MB6737.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2022 09:58:51.2496 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kwtc640SNxd0zYeuD/PzoeUH9BLGiec7c8muO2Mcr5RCmYh/PLkhOu7uaB2fb5QGo3f3ov9WL/C4wIg1rvm6NQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB8341 X-MailFrom: olivier.masse@nxp.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: DQZTOKT5XDQHKZK2VKQZLLKX7OUHS7Y2 X-Message-ID-Hash: DQZTOKT5XDQHKZK2VKQZLLKX7OUHS7Y2 X-Mailman-Approved-At: Thu, 25 Aug 2022 13:29:14 +0000 CC: clement.faure@nxp.com, olivier.masse@nxp.com X-Mailman-Version: 3.3.5 Precedence: list Subject: [Linaro-mm-sig] [PATCH 1/5] ANDROID: dma-buf: heaps: Add deferred-free-helper library code List-Id: "Unified memory management interest group." Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: John Stultz This patch provides infrastructure for deferring buffer frees. This is a feature ION provided which when used with some form of a page pool, provides a nice performance boost in an allocation microbenchmark. The reason it helps is it allows the page-zeroing to be done out of the normal allocation/free path, and pushed off to a kthread. As not all heaps will find this useful, its implemented as a optional helper library that heaps can utilize. Cc: Daniel Vetter Cc: Sumit Semwal Cc: Liam Mark Cc: Chris Goldsworthy Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Daniel Mentz Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz Bug: 168742043 --- drivers/dma-buf/heaps/Kconfig | 3 + drivers/dma-buf/heaps/Makefile | 1 + drivers/dma-buf/heaps/deferred-free-helper.c | 136 +++++++++++++++++++ drivers/dma-buf/heaps/deferred-free-helper.h | 63 +++++++++ 4 files changed, 203 insertions(+) create mode 100644 drivers/dma-buf/heaps/deferred-free-helper.c create mode 100644 drivers/dma-buf/heaps/deferred-free-helper.h diff --git a/drivers/dma-buf/heaps/Kconfig b/drivers/dma-buf/heaps/Kconfig index 3782eeeb91c0..8ee64277a5d2 100644 --- a/drivers/dma-buf/heaps/Kconfig +++ b/drivers/dma-buf/heaps/Kconfig @@ -1,3 +1,6 @@ +config DMABUF_HEAPS_DEFERRED_FREE + tristate + config DMABUF_HEAPS_SYSTEM bool "DMA-BUF System Heap" depends on DMABUF_HEAPS diff --git a/drivers/dma-buf/heaps/Makefile b/drivers/dma-buf/heaps/Makefile index 29733f84c354..5de95b77e169 100644 --- a/drivers/dma-buf/heaps/Makefile +++ b/drivers/dma-buf/heaps/Makefile @@ -1,4 +1,5 @@ # SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_DMABUF_HEAPS_DEFERRED_FREE) += deferred-free-helper.o obj-$(CONFIG_DMABUF_HEAPS_SYSTEM) += system_heap.o obj-$(CONFIG_DMABUF_HEAPS_CMA) += cma_heap.o obj-$(CONFIG_DMABUF_HEAPS_DSP) += dsp_heap.o diff --git a/drivers/dma-buf/heaps/deferred-free-helper.c b/drivers/dma-buf/heaps/deferred-free-helper.c new file mode 100644 index 000000000000..478faa319908 --- /dev/null +++ b/drivers/dma-buf/heaps/deferred-free-helper.c @@ -0,0 +1,136 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Deferred dmabuf freeing helper + * + * Copyright (C) 2020 Linaro, Ltd. + * + * Based on the ION page pool code + * Copyright (C) 2011 Google, Inc. + */ + +#include +#include +#include +#include +#include + +#include "deferred-free-helper.h" + +static LIST_HEAD(free_list); +static size_t list_nr_pages; +static wait_queue_head_t freelist_waitqueue; +static struct task_struct *freelist_task; +static DEFINE_SPINLOCK(free_list_lock); + +void deferred_free(struct deferred_freelist_item *item, + void (*free)(struct deferred_freelist_item*, + enum df_reason), + size_t nr_pages) +{ + unsigned long flags; + + if (!nr_pages) + return; + + INIT_LIST_HEAD(&item->list); + item->nr_pages = nr_pages; + item->free = free; + + spin_lock_irqsave(&free_list_lock, flags); + list_add(&item->list, &free_list); + list_nr_pages += nr_pages; + spin_unlock_irqrestore(&free_list_lock, flags); + wake_up(&freelist_waitqueue); +} +EXPORT_SYMBOL_GPL(deferred_free); + +static size_t free_one_item(enum df_reason reason) +{ + unsigned long flags; + size_t nr_pages; + struct deferred_freelist_item *item; + + spin_lock_irqsave(&free_list_lock, flags); + if (list_empty(&free_list)) { + spin_unlock_irqrestore(&free_list_lock, flags); + return 0; + } + item = list_first_entry(&free_list, struct deferred_freelist_item, list); + list_del(&item->list); + nr_pages = item->nr_pages; + list_nr_pages -= nr_pages; + spin_unlock_irqrestore(&free_list_lock, flags); + + item->free(item, reason); + return nr_pages; +} + +static unsigned long get_freelist_nr_pages(void) +{ + unsigned long nr_pages; + unsigned long flags; + + spin_lock_irqsave(&free_list_lock, flags); + nr_pages = list_nr_pages; + spin_unlock_irqrestore(&free_list_lock, flags); + return nr_pages; +} + +static unsigned long freelist_shrink_count(struct shrinker *shrinker, + struct shrink_control *sc) +{ + return get_freelist_nr_pages(); +} + +static unsigned long freelist_shrink_scan(struct shrinker *shrinker, + struct shrink_control *sc) +{ + unsigned long total_freed = 0; + + while (total_freed < sc->nr_to_scan) { + size_t pages_freed = free_one_item(DF_UNDER_PRESSURE); + + if (!pages_freed) + break; + + total_freed += pages_freed; + } + + return total_freed; +} + +static struct shrinker freelist_shrinker = { + .count_objects = freelist_shrink_count, + .scan_objects = freelist_shrink_scan, + .seeks = DEFAULT_SEEKS, + .batch = 0, +}; + +static int deferred_free_thread(void *data) +{ + while (true) { + wait_event_freezable(freelist_waitqueue, + get_freelist_nr_pages() > 0); + + free_one_item(DF_NORMAL); + } + + return 0; +} + +static int deferred_freelist_init(void) +{ + init_waitqueue_head(&freelist_waitqueue); + freelist_task = kthread_run(deferred_free_thread, NULL, + "%s", "dmabuf-deferred-free-worker"); + if (IS_ERR(freelist_task)) { + pr_err("Creating thread for deferred free failed\n"); + return PTR_ERR(freelist_task); + } + sched_set_normal(freelist_task, 19); + + return register_shrinker(&freelist_shrinker); +} +module_init(deferred_freelist_init); +MODULE_LICENSE("GPL v2"); + diff --git a/drivers/dma-buf/heaps/deferred-free-helper.h b/drivers/dma-buf/heaps/deferred-free-helper.h new file mode 100644 index 000000000000..4ed5893fdf3a --- /dev/null +++ b/drivers/dma-buf/heaps/deferred-free-helper.h @@ -0,0 +1,63 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Deferred dmabuf freeing helper + * + * Copyright (C) 2020 Linaro, Ltd. + * + * Based on the ION page pool code + * Copyright (C) 2011 Google, Inc. + */ + +#ifndef DEFERRED_FREE_HELPER_H +#define DEFERRED_FREE_HELPER_H + +/** + * df_reason - enum for reason why item was freed + * + * This provides a reason for why the free function was called + * on the item. This is useful when deferred_free is used in + * combination with a pagepool, so under pressure the page can + * be immediately freed. + * + * DF_NORMAL: Normal deferred free + * + * DF_UNDER_PRESSURE: Free was called because the system + * is under memory pressure. Usually + * from a shrinker. Avoid allocating + * memory in the free call, as it may + * fail. + */ +enum df_reason { + DF_NORMAL, + DF_UNDER_PRESSURE, +}; + +/** + * deferred_freelist_item - item structure for deferred freelist + * + * This is to be added to the structure for whatever you want to + * defer freeing on. + * + * @nr_pages: number of pages used by item to be freed + * @free: function pointer to be called when freeing the item + * @list: list entry for the deferred list + */ +struct deferred_freelist_item { + size_t nr_pages; + void (*free)(struct deferred_freelist_item *i, + enum df_reason reason); + struct list_head list; +}; + +/** + * deferred_free - add item to the deferred free list + * + * @item: Pointer to deferred_freelist_item field of a structure + * @free: Function pointer to the free call + * @nr_pages: number of pages to be freed + */ +void deferred_free(struct deferred_freelist_item *item, + void (*free)(struct deferred_freelist_item *i, + enum df_reason reason), + size_t nr_pages); +#endif From patchwork Tue Aug 2 09:58:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Olivier Masse X-Patchwork-Id: 599965 Delivered-To: patches@linaro.org Received: by 2002:a05:7000:4388:0:0:0:0 with SMTP id w8csp290212mae; Thu, 25 Aug 2022 06:30:02 -0700 (PDT) X-Google-Smtp-Source: AA6agR4zBWzvHAePwQrECZnPz/mq26iBaAYfSuFHaNUW8qXEzo2fD4dbbsQgalqqQi+4a8UEGLEk X-Received: by 2002:a05:622a:1986:b0:343:225d:f9e1 with SMTP id u6-20020a05622a198600b00343225df9e1mr3525875qtc.651.1661434202028; Thu, 25 Aug 2022 06:30:02 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [3.208.193.21]) by mx.google.com with ESMTPS id jf10-20020a0562142a4a00b00492cce62509si10194834qvb.116.2022.08.25.06.30.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 06:30:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linaro-mm-sig-bounces+patches=linaro.org@lists.linaro.org designates 3.208.193.21 as permitted sender) client-ip=3.208.193.21; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@nxp.com header.s=selector2 header.b=hHeWLaNk; arc=fail (body hash mismatch); spf=pass (google.com: domain of linaro-mm-sig-bounces+patches=linaro.org@lists.linaro.org designates 3.208.193.21 as permitted sender) smtp.mailfrom="linaro-mm-sig-bounces+patches=linaro.org@lists.linaro.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from lists.linaro.org (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id D5AA23F613 for ; Thu, 25 Aug 2022 13:30:01 +0000 (UTC) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2062.outbound.protection.outlook.com [40.107.20.62]) by lists.linaro.org (Postfix) with ESMTPS id 00F704798B for ; Tue, 2 Aug 2022 09:58:53 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lhYAwWmzEbXWMWfplvvTPs2jdv7G5K/U4X06eCW2LDiUDLYFCrTWZNdvmwAbAKnFyV5r+T6SIqU1z+jZIXMnhiWPLKIUun4caev1LGsSmmYrybXBxybv95TxkHO74BZeensHNJyuH2TViLOP+Ra1LlMbVNQA9i4WJrJrrlFbXQuakvv7xkdwep04hl8RUjyGBK7khA17mdkiHS1wkj1nUR+NTMmhLJhD4E+8Dn5bEdsQnz9t8snIzVulVNvt2mHZilnpAn/Np6njvnLS0A5XaA+76Xq3gbhNfJXoQvtkZuF0p6Lid2ESWWJTLx+4WIsEC3+iHS18kZZvDnBvtMniKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=B6F4CMRdmn0GSb1NELVUv8fyXfsdW1uJtKJqqy0RZHs=; b=c04TM49Wfkm+Bng/rkCbN1wkDmskI1ZKdrsB8axBnq/NWm4ujX4jwrne9WSYqtJpJDuh05NDR6I79Cve8QnmigtG5rPjtkO9oxF+cvmE9jgILx7CpxpRVdpCyeYn5yD9uugBrPNVT3IxHCy2u7wXuJn0o5u5sg0Vo7axo/1awLun3/DG3Lfaonf7A0wA27AEIEn6zd1OsEe9pBWQlG0qs8LhnkP27aoShq+5c4b3aew0XN3a/nF5eUqseiAMP8gVq5lXqpEIf3jdFEdLNJrGMrinNyRbOdJ2e3R1tCnYtujEyusEKEpcU67h0ZDeuWt1ahId5v4HGI3jBI5akRfT0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B6F4CMRdmn0GSb1NELVUv8fyXfsdW1uJtKJqqy0RZHs=; b=hHeWLaNkxVsyN7YIJ58EQ18YBZNAoOfutrJWRBbRDlwwq6dvopuX9ARso3HH86ArB9vC1zZpQRPXnwvZYQ5GOHRRnFC+MVnMdTy6izvE4lS6aEk8/5GOQXiZpIYOX1w7AuWleeuTccqAZlgr/a5KXjshZdKoDnNCUfEO/DwqZrk= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from AM0PR04MB6737.eurprd04.prod.outlook.com (2603:10a6:208:176::18) by AS8PR04MB8341.eurprd04.prod.outlook.com (2603:10a6:20b:3b0::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5482.16; Tue, 2 Aug 2022 09:58:52 +0000 Received: from AM0PR04MB6737.eurprd04.prod.outlook.com ([fe80::6ca7:299e:d92f:7d1b]) by AM0PR04MB6737.eurprd04.prod.outlook.com ([fe80::6ca7:299e:d92f:7d1b%5]) with mapi id 15.20.5482.011; Tue, 2 Aug 2022 09:58:52 +0000 From: Olivier Masse To: sumit.semwal@linaro.org, benjamin.gaignard@collabora.com, Brian.Starkey@arm.com, christian.koenig@amd.com, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-kernel@vger.kernel.org Date: Tue, 2 Aug 2022 11:58:40 +0200 Message-Id: <20220802095843.14614-3-olivier.masse@nxp.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20220802095843.14614-1-olivier.masse@nxp.com> References: <20220802095843.14614-1-olivier.masse@nxp.com> X-ClientProxiedBy: AM3PR07CA0145.eurprd07.prod.outlook.com (2603:10a6:207:8::31) To AM0PR04MB6737.eurprd04.prod.outlook.com (2603:10a6:208:176::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d1aa91d2-bdc0-4ee8-19a8-08da746d9a88 X-MS-TrafficTypeDiagnostic: AS8PR04MB8341:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OrZ+Mf2Im1BWeUgPbN/aeFSeGEekynp4KPB7T98fHlEIdvs8OvGsFdFjRALozCFA/gkdbPPx8avo2CD8IiVLwq+FeSb8bhmN6/XlBai9zjnk3AxtOwM3OLZlimRwL8CqtaFhuNRNKwlIDrMs5/4KhN5msNAX3uN5jV0zNOy6QCN1PzcY8iXmqTAeysEiahk7f5aUCnErBUBN5RLp/d0ErRHmNVel8fVNnN0Fvbiwvy+mSpV/px610mP07GPKuBuuD5mt9DmAW3e5bLCvWBVXhjYMvBhf+eLO4bjMq7RBV0T7wyvKk/p5jBheFx6tyC0FcFtx8GDT8MMnGpzvS1D5mzy50PZWhJmMpbkior99F0gtjL18MSveLPzWY2hKg6IVpXNCpmJb5xy6OogG1nHO4S0jD137ouRxhNPXsgdJEQQr6mJFOfEMOyNVhYh3E4YBWxajhnaDygzA/+sSWe91gaiB6b8Jp3hUrYaNTrlJ367MK7c3+OizXmgzbkSZjJAYX68XVnh8pG2cLmfaqW9W1zzXaI1lvhOYfZJ1PFxo99yupkGA7FV0SYfHOggsBOl/ep6QVvRzdsl7KftFJjK7HSsO6WoofMu5qv/GyPD9jv16t2B8eqrv4bnPg7tQ8wHkV35oDV3IoVqrkDBYHFFRkQYjCb0dczyEwr0rXbYgNOkiFdiSzIGHAC0L9t/RS0agtyy0VDFvRPCizmcZDeJng2mEoG22k4Mqv+2WKUuoKuf15iZnnxaKifyFewTFJNeJGUgUipQyIoS3Fbd0xOXrhKV9n4OiQUltOV11TZx6Crk= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR04MB6737.eurprd04.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(396003)(346002)(366004)(136003)(39860400002)(376002)(41300700001)(2906002)(83380400001)(2616005)(86362001)(6666004)(52116002)(6506007)(44832011)(1076003)(30864003)(36756003)(26005)(6512007)(38100700002)(186003)(38350700002)(6486002)(5660300002)(8676002)(66556008)(66476007)(66946007)(8936002)(478600001)(4326008)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?9GrWezgtjpF/neKQaeu2sOFhzACQ?= =?utf-8?q?CpNpyXw5AT75hwFad626OyvF5/bjPp5X6wgI9da3SW4pplKq2oO3EC2uZqoJJvo6+?= =?utf-8?q?eDr/4ErDUWP1cyrhN6fan2+2Vq8AaDefj6+okul6nkGFiRG1xW+mO4xf4XAKg+FcZ?= =?utf-8?q?7v8KZN2ij1t4yO5Jrg5QIVtH4exOSpo07Mn+vw65Gq4eZ6hDYczaTx16UXhTj7Yap?= =?utf-8?q?MD0K4vWEi4hUgM54VAkdPDxaBZzsqWqN4FfM6KWKs6WH7R3pvwcwcph3EkBUhfzOi?= =?utf-8?q?EcXvj0aTMC9XOLRNZSEFfHu3rx9CvkxFSELrV7jiYisK47M8EoJkgqQ9jxsdoZ2bn?= =?utf-8?q?gtGVLr7vpo45C/K1GPPzcM7mSkyck+lhVgKCQ6yp0zzZEsF4SPAAJxwY74cpruwP8?= =?utf-8?q?n/U7jSzx6NYT5HRoeNgkw/HikNGFDMqQR7iMURu24ID4z9RFMg87Iweur/DCG1xrE?= =?utf-8?q?o+Vs/BQoK/pKbPuPt7902ruINKN3gDWMz6LAwOdyH1F9wBmsbeAXadQ1FPg7Zfrvg?= =?utf-8?q?Db+sWG3m+dZg/+ywZOcs0PCjAZTt8Zigqsvze0/gwAJfClJtlaJk2NZa7bphXv1Qo?= =?utf-8?q?8WaZHNs2trpMprc3yOG5KwdEGmVjZ7JOr/J1eIY+KVicHnL4wRAjVrsF6L30n9/Jh?= =?utf-8?q?kBHV0z8BcHPFoSLeiTznINlnJiQnhqXGnAqLSoeD8dbmuKJXFb3/6HP8DMs0z8j9j?= =?utf-8?q?KTPRoFjw5vgF9a+lu9x7K4JQpuufu2HAiY7+xql35tr8ZSWkOGg0zw/wJ5Cw7MBKk?= =?utf-8?q?vb4dpaEuuadUFU8bcX0fTdH0hKutAIQULh409eltO6wWYFlgw08V+9OqMv7LYci6I?= =?utf-8?q?az5DcFLbibpquFxrxJcjP+ELI6x9CV58hJIxs0FY5MglfeLHzVinwNSD/UipjqYp1?= =?utf-8?q?EhOv/eVJw7RTLcCUGKnFFmuHCwDg9qzyb+7FVWPTSDXm469l6H6zlrdxUwTjgx8yE?= =?utf-8?q?jRfCEDDwIQQtdD9UTntn5l+SuU/Jb9NsD635oMqo0UzDy7JXRybfBloTZt81W408K?= =?utf-8?q?Xa94DgczyElh+wcs3hdk6IXV22GV1MMLQIDm6GpVFfJvu5aCAO5u82UYf3zagUMz2?= =?utf-8?q?Q11qdiqvTLnhbXCfkQoLM9dDkrEZQWyCsznAG2q5G0iI3lpnsUMt85ED3wocYtSZz?= =?utf-8?q?7kbg1IKy141QlFXp9yDOqCtcmo8fvkcySzg+pQcG1XyiaAZ1VAhlMTUZs6+iY36v0?= =?utf-8?q?BFh8inzs0T3oS7bUbkdA1JFWHbNajQJdHkJE003G+WX3asytLtSALTt79Ei/KY6Ne?= =?utf-8?q?A94UXrcabdh1gw5sKURhECXZ0/FnXhUC8Sw3vVHfiqqMoxjCuO0HCTiO5kRbB2dlE?= =?utf-8?q?U19GHk1N68Cy4iccCAcryEZFBl1HA9VkZsg5pYk5snv3C9w2bnP5OXIG+SEPDoP/g?= =?utf-8?q?wD//f6RUATYymq1frrTtdNGeu8GtfM36X7sceuH62BoiHU9cmHwi+SzlbaV2XhQyH?= =?utf-8?q?y03V0uP1xpWj0o7rigEyUf0OtWjNgYeIJ99deCwwGqn1Qp+rdTfbqzUe9BkSK6iQ3?= =?utf-8?q?2qnk95Y7Wilb?= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: d1aa91d2-bdc0-4ee8-19a8-08da746d9a88 X-MS-Exchange-CrossTenant-AuthSource: AM0PR04MB6737.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2022 09:58:52.0620 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JJVQHhZVNHPI+vhQST2F/FbSJjoy0Oqz6BAmLzShGIaCRPcNiniHit7RCDbtHPsZK50coI1r7PjNQ7KlBC0mfw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR04MB8341 X-MailFrom: olivier.masse@nxp.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: RJVE5ZOFA5X5PWENRU5V2SEFJFCR35TF X-Message-ID-Hash: RJVE5ZOFA5X5PWENRU5V2SEFJFCR35TF X-Mailman-Approved-At: Thu, 25 Aug 2022 13:29:17 +0000 CC: clement.faure@nxp.com, olivier.masse@nxp.com X-Mailman-Version: 3.3.5 Precedence: list Subject: [Linaro-mm-sig] [PATCH 2/5] ANDROID: dma-buf: heaps: Add a shrinker controlled page pool List-Id: "Unified memory management interest group." Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: John Stultz This patch adds a simple shrinker controlled page pool to the dmabuf heaps subsystem. This replaces the use of the networking page_pool, over concerns that the lack of a shrinker for that implementation may cause additional low-memory kills TODO: Take another pass at trying to unify this w/ the ttm pool Thoughts and feedback would be greatly appreciated! Cc: Sumit Semwal Cc: Liam Mark Cc: Laura Abbott Cc: Brian Starkey Cc: Hridya Valsaraju Cc: Suren Baghdasaryan Cc: Sandeep Patil Cc: Daniel Mentz Cc: Chris Goldsworthy Cc: Ørjan Eide Cc: Robin Murphy Cc: Ezequiel Garcia Cc: Simon Ser Cc: James Jones Cc: Andrew Morton Cc: Dave Hansen Cc: linux-mm@kvack.org Cc: linux-media@vger.kernel.org Cc: dri-devel@lists.freedesktop.org Signed-off-by: John Stultz Signed-off-by: Olivier Masse Bug: 168742043 --- drivers/dma-buf/heaps/Kconfig | 3 + drivers/dma-buf/heaps/Makefile | 1 + drivers/dma-buf/heaps/page_pool.c | 246 ++++++++++++++++++++++++++++++ drivers/dma-buf/heaps/page_pool.h | 55 +++++++ 4 files changed, 305 insertions(+) create mode 100644 drivers/dma-buf/heaps/page_pool.c create mode 100644 drivers/dma-buf/heaps/page_pool.h diff --git a/drivers/dma-buf/heaps/Kconfig b/drivers/dma-buf/heaps/Kconfig index 8ee64277a5d2..6a33193a7b3e 100644 --- a/drivers/dma-buf/heaps/Kconfig +++ b/drivers/dma-buf/heaps/Kconfig @@ -1,6 +1,9 @@ config DMABUF_HEAPS_DEFERRED_FREE tristate +config DMABUF_HEAPS_PAGE_POOL + tristate + config DMABUF_HEAPS_SYSTEM bool "DMA-BUF System Heap" depends on DMABUF_HEAPS diff --git a/drivers/dma-buf/heaps/Makefile b/drivers/dma-buf/heaps/Makefile index 5de95b77e169..e70722ea615e 100644 --- a/drivers/dma-buf/heaps/Makefile +++ b/drivers/dma-buf/heaps/Makefile @@ -1,5 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_DMABUF_HEAPS_DEFERRED_FREE) += deferred-free-helper.o +obj-$(CONFIG_DMABUF_HEAPS_PAGE_POOL) += page_pool.o obj-$(CONFIG_DMABUF_HEAPS_SYSTEM) += system_heap.o obj-$(CONFIG_DMABUF_HEAPS_CMA) += cma_heap.o obj-$(CONFIG_DMABUF_HEAPS_DSP) += dsp_heap.o diff --git a/drivers/dma-buf/heaps/page_pool.c b/drivers/dma-buf/heaps/page_pool.c new file mode 100644 index 000000000000..3dd4c3862dca --- /dev/null +++ b/drivers/dma-buf/heaps/page_pool.c @@ -0,0 +1,246 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * DMA BUF page pool system + * + * Copyright (C) 2020 Linaro Ltd. + * + * Based on the ION page pool code + * Copyright (C) 2011 Google, Inc. + */ + +#include +#include +#include +#include +#include +#include "page_pool.h" + +static LIST_HEAD(pool_list); +static DEFINE_MUTEX(pool_list_lock); + +static struct page *dmabuf_page_pool_alloc_pages(struct dmabuf_page_pool *pool) +{ + if (fatal_signal_pending(current)) + return NULL; + return alloc_pages(pool->gfp_mask, pool->order); +} + +static void dmabuf_page_pool_free_pages(struct dmabuf_page_pool *pool, + struct page *page) +{ + __free_pages(page, pool->order); +} + +static void dmabuf_page_pool_add(struct dmabuf_page_pool *pool, struct page *page) +{ + int index; + + if (PageHighMem(page)) + index = POOL_HIGHPAGE; + else + index = POOL_LOWPAGE; + + mutex_lock(&pool->mutex); + list_add_tail(&page->lru, &pool->items[index]); + pool->count[index]++; + mod_node_page_state(page_pgdat(page), NR_KERNEL_MISC_RECLAIMABLE, + 1 << pool->order); + mutex_unlock(&pool->mutex); +} + +static struct page *dmabuf_page_pool_remove(struct dmabuf_page_pool *pool, int index) +{ + struct page *page; + + mutex_lock(&pool->mutex); + page = list_first_entry_or_null(&pool->items[index], struct page, lru); + if (page) { + pool->count[index]--; + list_del(&page->lru); + mod_node_page_state(page_pgdat(page), NR_KERNEL_MISC_RECLAIMABLE, + -(1 << pool->order)); + } + mutex_unlock(&pool->mutex); + + return page; +} + +static struct page *dmabuf_page_pool_fetch(struct dmabuf_page_pool *pool) +{ + struct page *page = NULL; + + page = dmabuf_page_pool_remove(pool, POOL_HIGHPAGE); + if (!page) + page = dmabuf_page_pool_remove(pool, POOL_LOWPAGE); + + return page; +} + +struct page *dmabuf_page_pool_alloc(struct dmabuf_page_pool *pool) +{ + struct page *page = NULL; + + if (WARN_ON(!pool)) + return NULL; + + page = dmabuf_page_pool_fetch(pool); + if (!page) + page = dmabuf_page_pool_alloc_pages(pool); + + return page; +} +EXPORT_SYMBOL_GPL(dmabuf_page_pool_alloc); + +void dmabuf_page_pool_free(struct dmabuf_page_pool *pool, struct page *page) +{ + if (WARN_ON(pool->order != compound_order(page))) + return; + + dmabuf_page_pool_add(pool, page); +} +EXPORT_SYMBOL_GPL(dmabuf_page_pool_free); + +static int dmabuf_page_pool_total(struct dmabuf_page_pool *pool, bool high) +{ + int count = pool->count[POOL_LOWPAGE]; + + if (high) + count += pool->count[POOL_HIGHPAGE]; + + return count << pool->order; +} + +struct dmabuf_page_pool *dmabuf_page_pool_create(gfp_t gfp_mask, unsigned int order) +{ + struct dmabuf_page_pool *pool = kmalloc(sizeof(*pool), GFP_KERNEL); + int i; + + if (!pool) + return NULL; + + for (i = 0; i < POOL_TYPE_SIZE; i++) { + pool->count[i] = 0; + INIT_LIST_HEAD(&pool->items[i]); + } + pool->gfp_mask = gfp_mask | __GFP_COMP; + pool->order = order; + mutex_init(&pool->mutex); + + mutex_lock(&pool_list_lock); + list_add(&pool->list, &pool_list); + mutex_unlock(&pool_list_lock); + + return pool; +} +EXPORT_SYMBOL_GPL(dmabuf_page_pool_create); + +void dmabuf_page_pool_destroy(struct dmabuf_page_pool *pool) +{ + struct page *page; + int i; + + /* Remove us from the pool list */ + mutex_lock(&pool_list_lock); + list_del(&pool->list); + mutex_unlock(&pool_list_lock); + + /* Free any remaining pages in the pool */ + for (i = 0; i < POOL_TYPE_SIZE; i++) { + while ((page = dmabuf_page_pool_remove(pool, i))) + dmabuf_page_pool_free_pages(pool, page); + } + + kfree(pool); +} +EXPORT_SYMBOL_GPL(dmabuf_page_pool_destroy); + +static int dmabuf_page_pool_do_shrink(struct dmabuf_page_pool *pool, gfp_t gfp_mask, + int nr_to_scan) +{ + int freed = 0; + bool high; + + if (current_is_kswapd()) + high = true; + else + high = !!(gfp_mask & __GFP_HIGHMEM); + + if (nr_to_scan == 0) + return dmabuf_page_pool_total(pool, high); + + while (freed < nr_to_scan) { + struct page *page; + + /* Try to free low pages first */ + page = dmabuf_page_pool_remove(pool, POOL_LOWPAGE); + if (!page) + page = dmabuf_page_pool_remove(pool, POOL_HIGHPAGE); + + if (!page) + break; + + dmabuf_page_pool_free_pages(pool, page); + freed += (1 << pool->order); + } + + return freed; +} + +static int dmabuf_page_pool_shrink(gfp_t gfp_mask, int nr_to_scan) +{ + struct dmabuf_page_pool *pool; + int nr_total = 0; + int nr_freed; + bool only_scan = false; + + if (!nr_to_scan) + only_scan = true; + + mutex_lock(&pool_list_lock); + list_for_each_entry(pool, &pool_list, list) { + if (only_scan) { + nr_total += dmabuf_page_pool_do_shrink(pool, + gfp_mask, + nr_to_scan); + } else { + nr_freed = dmabuf_page_pool_do_shrink(pool, + gfp_mask, + nr_to_scan); + nr_to_scan -= nr_freed; + nr_total += nr_freed; + if (nr_to_scan <= 0) + break; + } + } + mutex_unlock(&pool_list_lock); + + return nr_total; +} + +static unsigned long dmabuf_page_pool_shrink_count(struct shrinker *shrinker, + struct shrink_control *sc) +{ + return dmabuf_page_pool_shrink(sc->gfp_mask, 0); +} + +static unsigned long dmabuf_page_pool_shrink_scan(struct shrinker *shrinker, + struct shrink_control *sc) +{ + if (sc->nr_to_scan == 0) + return 0; + return dmabuf_page_pool_shrink(sc->gfp_mask, sc->nr_to_scan); +} + +struct shrinker pool_shrinker = { + .count_objects = dmabuf_page_pool_shrink_count, + .scan_objects = dmabuf_page_pool_shrink_scan, + .seeks = DEFAULT_SEEKS, + .batch = 0, +}; + +static int dmabuf_page_pool_init_shrinker(void) +{ + return register_shrinker(&pool_shrinker); +} +module_init(dmabuf_page_pool_init_shrinker); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/dma-buf/heaps/page_pool.h b/drivers/dma-buf/heaps/page_pool.h new file mode 100644 index 000000000000..e3ec9eaacbc2 --- /dev/null +++ b/drivers/dma-buf/heaps/page_pool.h @@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * DMA BUF PagePool implementation + * Based on earlier ION code by Google + * + * Copyright (C) 2011 Google, Inc. + * Copyright (C) 2020 Linaro Ltd. + */ + +#ifndef _DMABUF_PAGE_POOL_H +#define _DMABUF_PAGE_POOL_H + +#include +#include +#include +#include +#include +#include + +/* page types we track in the pool */ +enum { + POOL_LOWPAGE, /* Clean lowmem pages */ + POOL_HIGHPAGE, /* Clean highmem pages */ + + POOL_TYPE_SIZE +}; + +/** + * struct dmabuf_page_pool - pagepool struct + * @count[]: array of number of pages of that type in the pool + * @items[]: array of list of pages of the specific type + * @mutex: lock protecting this struct and especially the count + * item list + * @gfp_mask: gfp_mask to use from alloc + * @order: order of pages in the pool + * @list: list node for list of pools + * + * Allows you to keep a pool of pre allocated pages to use + */ +struct dmabuf_page_pool { + int count[POOL_TYPE_SIZE]; + struct list_head items[POOL_TYPE_SIZE]; + struct mutex mutex; + gfp_t gfp_mask; + unsigned int order; + struct list_head list; +}; + +struct dmabuf_page_pool *dmabuf_page_pool_create(gfp_t gfp_mask, + unsigned int order); +void dmabuf_page_pool_destroy(struct dmabuf_page_pool *pool); +struct page *dmabuf_page_pool_alloc(struct dmabuf_page_pool *pool); +void dmabuf_page_pool_free(struct dmabuf_page_pool *pool, struct page *page); + +#endif /* _DMABUF_PAGE_POOL_H */