From patchwork Fri Mar 31 03:24:23 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Leizhen \(ThunderTown\)" X-Patchwork-Id: 96341 Delivered-To: patch@linaro.org Received: by 10.140.89.233 with SMTP id v96csp537292qgd; Thu, 30 Mar 2017 20:27:56 -0700 (PDT) X-Received: by 10.98.68.82 with SMTP id r79mr736380pfa.41.1490930876621; Thu, 30 Mar 2017 20:27:56 -0700 (PDT) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n19si3691697pgc.376.2017.03.30.20.27.56; Thu, 30 Mar 2017 20:27:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932954AbdCaD14 (ORCPT + 20 others); Thu, 30 Mar 2017 23:27:56 -0400 Received: from szxga01-in.huawei.com ([45.249.212.187]:5275 "EHLO dggrg01-dlp.huawei.com" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S932350AbdCaD0Z (ORCPT ); Thu, 30 Mar 2017 23:26:25 -0400 Received: from 172.30.72.54 (EHLO DGGEML401-HUB.china.huawei.com) ([172.30.72.54]) by dggrg01-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ALU41361; Fri, 31 Mar 2017 11:26:21 +0800 (CST) Received: from localhost (10.177.23.164) by DGGEML401-HUB.china.huawei.com (10.3.17.32) with Microsoft SMTP Server id 14.3.301.0; Fri, 31 Mar 2017 11:26:11 +0800 From: Zhen Lei To: Joerg Roedel , iommu , Robin Murphy , David Woodhouse , Sudeep Dutt , Ashutosh Dixit , linux-kernel CC: Zefan Li , Xinwei Hu , "Tianhong Ding" , Hanjun Guo , Zhen Lei Subject: [PATCH v2 5/7] iommu/iova: to optimize the allocation performance of dma64 Date: Fri, 31 Mar 2017 11:24:23 +0800 Message-ID: <1490930665-9696-6-git-send-email-thunder.leizhen@huawei.com> X-Mailer: git-send-email 1.9.5.msysgit.0 In-Reply-To: <1490930665-9696-1-git-send-email-thunder.leizhen@huawei.com> References: <1490930665-9696-1-git-send-email-thunder.leizhen@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.23.164] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020205.58DDCC5E.0080, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 27e4a5b24e439e2d90073c5476dc07f5 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently we always search free iova space for dma64 begin at the last node of iovad rb-tree. In the worst case, there maybe too many nodes exist at the tail, so that we should traverse many times for the first loop in __alloc_and_insert_iova_range. As we traced, more than 10K times for the case of iperf. __alloc_and_insert_iova_range: ...... curr = __get_cached_rbnode(iovad, &limit_pfn); //--> return rb_last(&iovad->rbroot); while (curr) { ...... curr = rb_prev(curr); } So add cached64_node to take the same effect as cached32_node, and add the start_pfn boundary of dma64, to prevent a iova cross both dma32 and dma64 area. |-------------------|------------------------------| |<--cached32_node-->|<--------cached64_node------->| | | start_pfn dma_32bit_pfn + 1 Signed-off-by: Zhen Lei --- drivers/iommu/iova.c | 46 +++++++++++++++++++++++++++------------------- include/linux/iova.h | 5 +++-- 2 files changed, 30 insertions(+), 21 deletions(-) -- 2.5.0 diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index 87a9332..23abe84 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -37,10 +37,15 @@ insert_iova_boundary(struct iova_domain *iovad) { struct iova *iova; unsigned long start_pfn_32bit = iovad->start_pfn; + unsigned long start_pfn_64bit = iovad->dma_32bit_pfn + 1; iova = reserve_iova(iovad, start_pfn_32bit, start_pfn_32bit); BUG_ON(!iova); iovad->cached32_node = &iova->node; + + iova = reserve_iova(iovad, start_pfn_64bit, start_pfn_64bit); + BUG_ON(!iova); + iovad->cached64_node = &iova->node; } void @@ -62,8 +67,8 @@ init_iova_domain(struct iova_domain *iovad, unsigned long granule, init_iova_rcaches(iovad); /* - * Insert boundary nodes for dma32. So cached32_node can not be NULL in - * future. + * Insert boundary nodes for dma32 and dma64. So cached32_node and + * cached64_node can not be NULL in future. */ insert_iova_boundary(iovad); } @@ -75,10 +80,10 @@ __get_cached_rbnode(struct iova_domain *iovad, unsigned long *limit_pfn) struct rb_node *cached_node; struct rb_node *next_node; - if (*limit_pfn > iovad->dma_32bit_pfn) - return rb_last(&iovad->rbroot); - else + if (*limit_pfn <= iovad->dma_32bit_pfn) cached_node = iovad->cached32_node; + else + cached_node = iovad->cached64_node; next_node = rb_next(cached_node); if (next_node) { @@ -94,29 +99,32 @@ static void __cached_rbnode_insert_update(struct iova_domain *iovad, struct iova *new) { struct iova *cached_iova; + struct rb_node **cached_node; - if (new->pfn_hi > iovad->dma_32bit_pfn) - return; + if (new->pfn_hi <= iovad->dma_32bit_pfn) + cached_node = &iovad->cached32_node; + else + cached_node = &iovad->cached64_node; - cached_iova = rb_entry(iovad->cached32_node, struct iova, node); + cached_iova = rb_entry(*cached_node, struct iova, node); if (new->pfn_lo <= cached_iova->pfn_lo) - iovad->cached32_node = rb_prev(&new->node); + *cached_node = rb_prev(&new->node); } static void __cached_rbnode_delete_update(struct iova_domain *iovad, struct iova *free) { struct iova *cached_iova; - struct rb_node *curr; + struct rb_node **cached_node; - curr = iovad->cached32_node; - cached_iova = rb_entry(curr, struct iova, node); + if (free->pfn_hi <= iovad->dma_32bit_pfn) + cached_node = &iovad->cached32_node; + else + cached_node = &iovad->cached64_node; - if (free->pfn_lo >= cached_iova->pfn_lo) { - /* only cache if it's below 32bit pfn */ - if (free->pfn_hi <= iovad->dma_32bit_pfn) - iovad->cached32_node = rb_prev(&free->node); - } + cached_iova = rb_entry(*cached_node, struct iova, node); + if (free->pfn_lo >= cached_iova->pfn_lo) + *cached_node = rb_prev(&free->node); } /* @@ -283,7 +291,7 @@ EXPORT_SYMBOL_GPL(iova_cache_put); * alloc_iova - allocates an iova * @iovad: - iova domain in question * @size: - size of page frames to allocate - * @limit_pfn: - max limit address + * @limit_pfn: - max limit address(included) * @size_aligned: - set if size_aligned address range is required * This function allocates an iova in the range iovad->start_pfn to limit_pfn, * searching top-down from limit_pfn to iovad->start_pfn. If the size_aligned @@ -402,7 +410,7 @@ EXPORT_SYMBOL_GPL(free_iova); * alloc_iova_fast - allocates an iova from rcache * @iovad: - iova domain in question * @size: - size of page frames to allocate - * @limit_pfn: - max limit address + * @limit_pfn: - max limit address(included) * This function tries to satisfy an iova allocation from the rcache, * and falls back to regular allocation on failure. */ diff --git a/include/linux/iova.h b/include/linux/iova.h index f27bb2c..844d723 100644 --- a/include/linux/iova.h +++ b/include/linux/iova.h @@ -40,10 +40,11 @@ struct iova_rcache { struct iova_domain { spinlock_t iova_rbtree_lock; /* Lock to protect update of rbtree */ struct rb_root rbroot; /* iova domain rbtree root */ - struct rb_node *cached32_node; /* Save last alloced node */ + struct rb_node *cached32_node; /* Save last alloced node, 32bits */ + struct rb_node *cached64_node; /* Save last alloced node, 64bits */ unsigned long granule; /* pfn granularity for this domain */ unsigned long start_pfn; /* Lower limit for this domain */ - unsigned long dma_32bit_pfn; + unsigned long dma_32bit_pfn; /* max dma32 limit address(included) */ struct iova_rcache rcaches[IOVA_RANGE_CACHE_MAX_SIZE]; /* IOVA range caches */ };