From patchwork Thu Sep 21 15:52:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 113288 Delivered-To: patch@linaro.org Received: by 10.140.106.117 with SMTP id d108csp2185286qgf; Thu, 21 Sep 2017 08:53:03 -0700 (PDT) X-Google-Smtp-Source: AOwi7QDsXs/ZEOe217/ZWin5nv4+O75nsNQRZRCThVcWKy8wPgLhoMNzgqlqVdeFeUH84m9EiswT X-Received: by 10.84.212.2 with SMTP id d2mr5956935pli.412.1506009183178; Thu, 21 Sep 2017 08:53:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506009183; cv=none; d=google.com; s=arc-20160816; b=zpxYLenVivthmm/fNH3JoC6siraaxZjnTF5NYeq6cV9qeLQKV2/EyMriMcJyNUqLZL 1B854JgCd9l3oUxGHIRdQdJ5S3noFo8k26xpJ//yOKDumTukb/qkMpWN8FK8W/0z1DE9 B93nHHPPSO/LZl59H+tfOBxK/VSSsultVu8J+O7w22oVVBI24r/j1+W0hXYyUiIIavax lHjwBjYfyhp6FCwPevVEAt69FuzBoRkQ/meKxnjSYv65zpFbMJwG23BntbYyTj74c0CV UMqNCv1dtTGuPCjYIwTCp87okVcnU8HAo6fBnSHEvPYc3CWJpyE6rP9HAVQNf2Xu2ugd Nt/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=EH8k+nbhoh8bCxj+375jkpUrYBPDDYonX627lRw3AKA=; b=tK3jo/dPcFiBVOCAFt3hZppgaqg1Nv1LMDl5nQpB7S/RAf/jIhxVOcVCd+hHjHYvW9 ri19ibDXTm5zqODsuzDcGn5DrdyGi0DuDWyzVTw0OtAX+gNik06I/00sgGnyVZfLgNjK fBLocUleRusxw8WB+65iIEh4NNYHz2/8+JXKsE+PyVtNmw09s8H3VyssYfYK8w0TKT1S Y39pEE2NiJ63NfMXNnaXwk4BFJTWPAypJUpeQ0ReaKXhrkvhpFUihnhk6Ru+IqamB+4v YxNg99TE5LAy7UkOnpkkRGIbc6a/SFK22Yft64WB/VmUSWygVoXHzYButOnQmizYDInX lt+g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d1si1200104pgu.506.2017.09.21.08.53.02; Thu, 21 Sep 2017 08:53:03 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751750AbdIUPxB (ORCPT + 26 others); Thu, 21 Sep 2017 11:53:01 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:48696 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751709AbdIUPw6 (ORCPT ); Thu, 21 Sep 2017 11:52:58 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ED4861435; Thu, 21 Sep 2017 08:52:57 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.cambridge.arm.com [10.1.210.88]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 97C6B3F578; Thu, 21 Sep 2017 08:52:56 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org Cc: iommu@lists.linux-foundation.org, thunder.leizhen@huawei.com, nwatters@codeaurora.org, tomasz.nowicki@caviumnetworks.com, dwoods@mellanox.com, linux-kernel@vger.kernel.org Subject: [PATCH v5 1/6] iommu/iova: Optimise rbtree searching Date: Thu, 21 Sep 2017 16:52:42 +0100 Message-Id: X-Mailer: git-send-email 2.13.4.dirty In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei Checking the IOVA bounds separately before deciding which direction to continue the search (if necessary) results in redundantly comparing both pfns twice each. GCC can already determine that the final comparison op is redundant and optimise it down to 3 in total, but we can go one further with a little tweak of the ordering (which makes the intent of the code that much cleaner as a bonus). Signed-off-by: Zhen Lei Tested-by: Ard Biesheuvel Tested-by: Zhen Lei Tested-by: Nate Watterson [rm: rewrote commit message to clarify] Signed-off-by: Robin Murphy --- v5: No change drivers/iommu/iova.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) -- 2.13.4.dirty diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index 33edfa794ae9..f129ff4f5c89 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -342,15 +342,12 @@ private_find_iova(struct iova_domain *iovad, unsigned long pfn) while (node) { struct iova *iova = rb_entry(node, struct iova, node); - /* If pfn falls within iova's range, return iova */ - if ((pfn >= iova->pfn_lo) && (pfn <= iova->pfn_hi)) { - return iova; - } - if (pfn < iova->pfn_lo) node = node->rb_left; - else if (pfn > iova->pfn_lo) + else if (pfn > iova->pfn_hi) node = node->rb_right; + else + return iova; /* pfn falls within iova's range */ } return NULL; From patchwork Thu Sep 21 15:52:43 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 113289 Delivered-To: patch@linaro.org Received: by 10.140.106.117 with SMTP id d108csp2185337qgf; Thu, 21 Sep 2017 08:53:07 -0700 (PDT) X-Google-Smtp-Source: AOwi7QB6fng8Fhl2JF5BShbIbSe1/7i5P1w95nrStP6/2Igj24vai9zeWChEbusw8NILU1TJcAjs X-Received: by 10.99.112.24 with SMTP id l24mr6014027pgc.302.1506009187342; Thu, 21 Sep 2017 08:53:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506009187; cv=none; d=google.com; s=arc-20160816; b=Yf0ndsim4wKrf3SbIBG4ECRrFL1Dsk8nvBcuaoZWpal+npiu+idz7WMv2dzzIIRE4Y C0V1BBOqC5XJqu4+QukIMvGII89QHP52tmERudH+JbOQ6nQspdtURMRHSKRF+3KwrHJ/ 6QS/UVr+aZrsQm9zoLhRVdpDwc9nSg4cM2RCLHPrdumiqTcXeBdnSZTB6vwWE+MKHJiU bUM8gqEMq1JHgA8kL/t1q50tPlQ1To2KH0A+jZ3t8Z+CwuEI1y1IpoCOvqWQxAI7owBo vuMq5TI9x7fyTsIHgEuWV34nLJ2FwZQ2+1L0FWDt5k3rt8Oj/jdPQ/GyAMQ6qrtDLxQU sFGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=n+wtCAE/S2Bb6EoyCLOK5/XzdDmPQ9L7zOyKCWwOmOk=; b=a4j7LzbzdGLJcCr+gyjw0TLVC/bT/qga4iNfbID9umLnDzI3yuPtfPuXaVF9kbixoE NKsmkBWSPinPBKB/wrUA5DH0D4OtgJz9YVw2V1OyWtD9tv0D2eHllMTnEdJlS6DYYQQJ /k0GzTZfWkflmg3/qChER5kK+eik++KoMU6ZKB2Gts3J1VsD5eY3g/3T3w8hvwdm+37I 4/4jg9tfgsW+sVBLgGtLZ5JnYizlSfWh6u5NTWmbQvFDwFaJffDKBcAMWEv6V2sQcvVI 1c8NCowpKSnNd8eR6GJnFuaLWRceNkbkz8OmjN6Ax1UyeRCiwvduUjdKsSj9AZwR6cgh hvKw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i17si1199212pgn.373.2017.09.21.08.53.06; Thu, 21 Sep 2017 08:53:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751792AbdIUPxE (ORCPT + 26 others); Thu, 21 Sep 2017 11:53:04 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:48710 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751722AbdIUPxC (ORCPT ); Thu, 21 Sep 2017 11:53:02 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B744A1435; Thu, 21 Sep 2017 08:53:01 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.cambridge.arm.com [10.1.210.88]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 614263F578; Thu, 21 Sep 2017 08:53:00 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org Cc: iommu@lists.linux-foundation.org, thunder.leizhen@huawei.com, nwatters@codeaurora.org, tomasz.nowicki@caviumnetworks.com, dwoods@mellanox.com, linux-kernel@vger.kernel.org Subject: [PATCH v5 2/6] iommu/iova: Optimise the padding calculation Date: Thu, 21 Sep 2017 16:52:43 +0100 Message-Id: <69401ae633a0a869423eaf6b4ea61bc38a3e0e80.1506007392.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.13.4.dirty In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei The mask for calculating the padding size doesn't change, so there's no need to recalculate it every loop iteration. Furthermore, Once we've done that, it becomes clear that we don't actually need to calculate a padding size at all - by flipping the arithmetic around, we can just combine the upper limit, size, and mask directly to check against the lower limit. For an arm64 build, this alone knocks 20% off the object code size of the entire alloc_iova() function! Signed-off-by: Zhen Lei Tested-by: Ard Biesheuvel Tested-by: Zhen Lei Tested-by: Nate Watterson [rm: simplified more of the arithmetic, rewrote commit message] Signed-off-by: Robin Murphy --- v5: No change drivers/iommu/iova.c | 42 +++++++++++++++--------------------------- 1 file changed, 15 insertions(+), 27 deletions(-) -- 2.13.4.dirty diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index f129ff4f5c89..20be9a8b3188 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -182,24 +182,17 @@ iova_insert_rbtree(struct rb_root *root, struct iova *iova, rb_insert_color(&iova->node, root); } -/* - * Computes the padding size required, to make the start address - * naturally aligned on the power-of-two order of its size - */ -static unsigned int -iova_get_pad_size(unsigned int size, unsigned int limit_pfn) -{ - return (limit_pfn - size) & (__roundup_pow_of_two(size) - 1); -} - static int __alloc_and_insert_iova_range(struct iova_domain *iovad, unsigned long size, unsigned long limit_pfn, struct iova *new, bool size_aligned) { struct rb_node *prev, *curr = NULL; unsigned long flags; - unsigned long saved_pfn; - unsigned int pad_size = 0; + unsigned long saved_pfn, new_pfn; + unsigned long align_mask = ~0UL; + + if (size_aligned) + align_mask <<= fls_long(size - 1); /* Walk the tree backwards */ spin_lock_irqsave(&iovad->iova_rbtree_lock, flags); @@ -209,31 +202,26 @@ static int __alloc_and_insert_iova_range(struct iova_domain *iovad, while (curr) { struct iova *curr_iova = rb_entry(curr, struct iova, node); - if (limit_pfn <= curr_iova->pfn_lo) { + if (limit_pfn <= curr_iova->pfn_lo) goto move_left; - } else if (limit_pfn > curr_iova->pfn_hi) { - if (size_aligned) - pad_size = iova_get_pad_size(size, limit_pfn); - if ((curr_iova->pfn_hi + size + pad_size) < limit_pfn) - break; /* found a free slot */ - } + + if (((limit_pfn - size) & align_mask) > curr_iova->pfn_hi) + break; /* found a free slot */ + limit_pfn = curr_iova->pfn_lo; move_left: prev = curr; curr = rb_prev(curr); } - if (!curr) { - if (size_aligned) - pad_size = iova_get_pad_size(size, limit_pfn); - if ((iovad->start_pfn + size + pad_size) > limit_pfn) { - spin_unlock_irqrestore(&iovad->iova_rbtree_lock, flags); - return -ENOMEM; - } + new_pfn = (limit_pfn - size) & align_mask; + if (limit_pfn < size || new_pfn < iovad->start_pfn) { + spin_unlock_irqrestore(&iovad->iova_rbtree_lock, flags); + return -ENOMEM; } /* pfn_lo will point to size aligned address if size_aligned is set */ - new->pfn_lo = limit_pfn - (size + pad_size); + new->pfn_lo = new_pfn; new->pfn_hi = new->pfn_lo + size - 1; /* If we have 'prev', it's a valid place to start the insertion. */ From patchwork Thu Sep 21 15:52:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 113290 Delivered-To: patch@linaro.org Received: by 10.140.106.117 with SMTP id d108csp2185425qgf; Thu, 21 Sep 2017 08:53:12 -0700 (PDT) X-Google-Smtp-Source: AOwi7QAZLqAAc1ltBoNHnJU1/bRnRH6YYChNEW9luv4zHmcVyJ99uZeoPPazYD/PZzj/OLpkPB00 X-Received: by 10.98.67.209 with SMTP id l78mr6282574pfi.3.1506009192637; Thu, 21 Sep 2017 08:53:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1506009192; cv=none; d=google.com; s=arc-20160816; b=I4ZTmT3YIIsImAyFVVg4VS6dTEbqBVbAuqia1wRJ1YqDMcm8zzE+n/yrtboYHWsewj qi2YjM6lO/hbsOTQ/Zpef8D71wwu3dP6agcqZwV3z99+RoFIuNLTOXd3c8LsGBHzjKQr 5aOfnhXLA6s/c/V2OK9/5umRA2flx9eB4Reqoz+/q+5Spor60nQWl9HxDbbYvhKvnrvj De+OHebwSF1dDOMwnwR15djkl1Y3l/9Lyt3ZkxMbPhkyZkcz9F9KBLroZfIhipQFcl12 5rSpRwUqQespR8pQ/maMoYCj4BjkGJFMCsLTqGdRsZds9A1/ALSaChxwgVvKfK+yhkm5 LXqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=6oo/AFeyoocrUrURDmP2lVxlomoXRUkMT6xWe6XowTg=; b=09M2xecvYoxCMKUdBIoe99IrCyDAX1Rh6KRderuVHdTuLoS4+AljHQIJgimqbv8g0p NcIyb5yB4FKFklr6Kj+1oLrYXW61zk4YOk/k862PVU4/+PyNE0jSdcKgjPFc8YnBCaf7 GxNZeKZly0B/kvFMZz0kp+4Mke+gp/xsr2jK98Img/gV4wOk3k53yWqQV5eFx4O2bW3O 1lccepxurmNQW2RDH/2Vk9oaZWEYXjEaEWk1KhS5ADmYOl6CoE3s7aX6D9+3cdvCn21P Wajuj8+E7PC6YC9Se+150iEbcxEZf0HyFIqBrqSjkAya8Dn1U1DZOBEHN7/xncZ87OQt iuHw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i17si1199212pgn.373.2017.09.21.08.53.12; Thu, 21 Sep 2017 08:53:12 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751818AbdIUPxJ (ORCPT + 26 others); Thu, 21 Sep 2017 11:53:09 -0400 Received: from foss.arm.com ([217.140.101.70]:48744 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751754AbdIUPxH (ORCPT ); Thu, 21 Sep 2017 11:53:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D9DD51435; Thu, 21 Sep 2017 08:53:06 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.cambridge.arm.com [10.1.210.88]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C6B1E3F578; Thu, 21 Sep 2017 08:53:04 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org Cc: iommu@lists.linux-foundation.org, thunder.leizhen@huawei.com, nwatters@codeaurora.org, tomasz.nowicki@caviumnetworks.com, dwoods@mellanox.com, linux-kernel@vger.kernel.org, Thierry Reding , Jonathan Hunter , David Airlie , Sudeep Dutt , Ashutosh Dixit Subject: [PATCH v5 4/6] iommu/iova: Make dma_32bit_pfn implicit Date: Thu, 21 Sep 2017 16:52:45 +0100 Message-Id: <303e5bcd57ca4d829ef6cfdec704834f1f8de5af.1506007392.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.13.4.dirty In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei Now that the cached node optimisation can apply to all allocations, the couple of users which were playing tricks with dma_32bit_pfn in order to benefit from it can stop doing so. Conversely, there is also no need for all the other users to explicitly calculate a 'real' 32-bit PFN, when init_iova_domain() can happily do that itself from the page granularity. CC: Thierry Reding CC: Jonathan Hunter CC: David Airlie CC: Sudeep Dutt CC: Ashutosh Dixit Signed-off-by: Zhen Lei Tested-by: Ard Biesheuvel Tested-by: Zhen Lei Tested-by: Nate Watterson [rm: use iova_shift(), rewrote commit message] Signed-off-by: Robin Murphy --- v5: No change drivers/gpu/drm/tegra/drm.c | 3 +-- drivers/gpu/host1x/dev.c | 3 +-- drivers/iommu/amd_iommu.c | 7 ++----- drivers/iommu/dma-iommu.c | 18 +----------------- drivers/iommu/intel-iommu.c | 11 +++-------- drivers/iommu/iova.c | 4 ++-- drivers/misc/mic/scif/scif_rma.c | 3 +-- include/linux/iova.h | 5 ++--- 8 files changed, 13 insertions(+), 41 deletions(-) -- 2.13.4.dirty diff --git a/drivers/gpu/drm/tegra/drm.c b/drivers/gpu/drm/tegra/drm.c index 597d563d636a..b822e484b7e5 100644 --- a/drivers/gpu/drm/tegra/drm.c +++ b/drivers/gpu/drm/tegra/drm.c @@ -155,8 +155,7 @@ static int tegra_drm_load(struct drm_device *drm, unsigned long flags) order = __ffs(tegra->domain->pgsize_bitmap); init_iova_domain(&tegra->carveout.domain, 1UL << order, - carveout_start >> order, - carveout_end >> order); + carveout_start >> order); tegra->carveout.shift = iova_shift(&tegra->carveout.domain); tegra->carveout.limit = carveout_end >> tegra->carveout.shift; diff --git a/drivers/gpu/host1x/dev.c b/drivers/gpu/host1x/dev.c index 7f22c5c37660..5267c62e8896 100644 --- a/drivers/gpu/host1x/dev.c +++ b/drivers/gpu/host1x/dev.c @@ -198,8 +198,7 @@ static int host1x_probe(struct platform_device *pdev) order = __ffs(host->domain->pgsize_bitmap); init_iova_domain(&host->iova, 1UL << order, - geometry->aperture_start >> order, - geometry->aperture_end >> order); + geometry->aperture_start >> order); host->iova_end = geometry->aperture_end; } diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 51f8215877f5..647ab7691aee 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -63,7 +63,6 @@ /* IO virtual address start page frame number */ #define IOVA_START_PFN (1) #define IOVA_PFN(addr) ((addr) >> PAGE_SHIFT) -#define DMA_32BIT_PFN IOVA_PFN(DMA_BIT_MASK(32)) /* Reserved IOVA ranges */ #define MSI_RANGE_START (0xfee00000) @@ -1788,8 +1787,7 @@ static struct dma_ops_domain *dma_ops_domain_alloc(void) if (!dma_dom->domain.pt_root) goto free_dma_dom; - init_iova_domain(&dma_dom->iovad, PAGE_SIZE, - IOVA_START_PFN, DMA_32BIT_PFN); + init_iova_domain(&dma_dom->iovad, PAGE_SIZE, IOVA_START_PFN); if (init_iova_flush_queue(&dma_dom->iovad, iova_domain_flush_tlb, NULL)) goto free_dma_dom; @@ -2696,8 +2694,7 @@ static int init_reserved_iova_ranges(void) struct pci_dev *pdev = NULL; struct iova *val; - init_iova_domain(&reserved_iova_ranges, PAGE_SIZE, - IOVA_START_PFN, DMA_32BIT_PFN); + init_iova_domain(&reserved_iova_ranges, PAGE_SIZE, IOVA_START_PFN); lockdep_set_class(&reserved_iova_ranges.iova_rbtree_lock, &reserved_rbtree_key); diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 9d1cebe7f6cb..191be9c80a8a 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -292,18 +292,7 @@ int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, /* ...then finally give it a kicking to make sure it fits */ base_pfn = max_t(unsigned long, base_pfn, domain->geometry.aperture_start >> order); - end_pfn = min_t(unsigned long, end_pfn, - domain->geometry.aperture_end >> order); } - /* - * PCI devices may have larger DMA masks, but still prefer allocating - * within a 32-bit mask to avoid DAC addressing. Such limitations don't - * apply to the typical platform device, so for those we may as well - * leave the cache limit at the top of their range to save an rb_last() - * traversal on every allocation. - */ - if (dev && dev_is_pci(dev)) - end_pfn &= DMA_BIT_MASK(32) >> order; /* start_pfn is always nonzero for an already-initialised domain */ if (iovad->start_pfn) { @@ -312,16 +301,11 @@ int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, pr_warn("Incompatible range for DMA domain\n"); return -EFAULT; } - /* - * If we have devices with different DMA masks, move the free - * area cache limit down for the benefit of the smaller one. - */ - iovad->dma_32bit_pfn = min(end_pfn + 1, iovad->dma_32bit_pfn); return 0; } - init_iova_domain(iovad, 1UL << order, base_pfn, end_pfn); + init_iova_domain(iovad, 1UL << order, base_pfn); if (!dev) return 0; diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 6784a05dd6b2..ebb48353dd39 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -82,8 +82,6 @@ #define IOVA_START_PFN (1) #define IOVA_PFN(addr) ((addr) >> PAGE_SHIFT) -#define DMA_32BIT_PFN IOVA_PFN(DMA_BIT_MASK(32)) -#define DMA_64BIT_PFN IOVA_PFN(DMA_BIT_MASK(64)) /* page table handling */ #define LEVEL_STRIDE (9) @@ -1878,8 +1876,7 @@ static int dmar_init_reserved_ranges(void) struct iova *iova; int i; - init_iova_domain(&reserved_iova_list, VTD_PAGE_SIZE, IOVA_START_PFN, - DMA_32BIT_PFN); + init_iova_domain(&reserved_iova_list, VTD_PAGE_SIZE, IOVA_START_PFN); lockdep_set_class(&reserved_iova_list.iova_rbtree_lock, &reserved_rbtree_key); @@ -1938,8 +1935,7 @@ static int domain_init(struct dmar_domain *domain, struct intel_iommu *iommu, unsigned long sagaw; int err; - init_iova_domain(&domain->iovad, VTD_PAGE_SIZE, IOVA_START_PFN, - DMA_32BIT_PFN); + init_iova_domain(&domain->iovad, VTD_PAGE_SIZE, IOVA_START_PFN); err = init_iova_flush_queue(&domain->iovad, iommu_flush_iova, iova_entry_free); @@ -4897,8 +4893,7 @@ static int md_domain_init(struct dmar_domain *domain, int guest_width) { int adjust_width; - init_iova_domain(&domain->iovad, VTD_PAGE_SIZE, IOVA_START_PFN, - DMA_32BIT_PFN); + init_iova_domain(&domain->iovad, VTD_PAGE_SIZE, IOVA_START_PFN); domain_reserve_special_ranges(domain); /* calculate AGAW */ diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index c6f5a22f8d20..65032e60a5d1 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -37,7 +37,7 @@ static void fq_flush_timeout(unsigned long data); void init_iova_domain(struct iova_domain *iovad, unsigned long granule, - unsigned long start_pfn, unsigned long pfn_32bit) + unsigned long start_pfn) { /* * IOVA granularity will normally be equal to the smallest @@ -52,7 +52,7 @@ init_iova_domain(struct iova_domain *iovad, unsigned long granule, iovad->cached32_node = NULL; iovad->granule = granule; iovad->start_pfn = start_pfn; - iovad->dma_32bit_pfn = pfn_32bit + 1; + iovad->dma_32bit_pfn = 1UL << (32 - iova_shift(iovad)); iovad->flush_cb = NULL; iovad->fq = NULL; init_iova_rcaches(iovad); diff --git a/drivers/misc/mic/scif/scif_rma.c b/drivers/misc/mic/scif/scif_rma.c index 329727e00e97..c824329f7012 100644 --- a/drivers/misc/mic/scif/scif_rma.c +++ b/drivers/misc/mic/scif/scif_rma.c @@ -39,8 +39,7 @@ void scif_rma_ep_init(struct scif_endpt *ep) struct scif_endpt_rma_info *rma = &ep->rma_info; mutex_init(&rma->rma_lock); - init_iova_domain(&rma->iovad, PAGE_SIZE, SCIF_IOVA_START_PFN, - SCIF_DMA_64BIT_PFN); + init_iova_domain(&rma->iovad, PAGE_SIZE, SCIF_IOVA_START_PFN); spin_lock_init(&rma->tc_lock); mutex_init(&rma->mmn_lock); INIT_LIST_HEAD(&rma->reg_list); diff --git a/include/linux/iova.h b/include/linux/iova.h index 69ea3e258ff2..953cfd20f152 100644 --- a/include/linux/iova.h +++ b/include/linux/iova.h @@ -154,7 +154,7 @@ struct iova *reserve_iova(struct iova_domain *iovad, unsigned long pfn_lo, unsigned long pfn_hi); void copy_reserved_iova(struct iova_domain *from, struct iova_domain *to); void init_iova_domain(struct iova_domain *iovad, unsigned long granule, - unsigned long start_pfn, unsigned long pfn_32bit); + unsigned long start_pfn); int init_iova_flush_queue(struct iova_domain *iovad, iova_flush_cb flush_cb, iova_entry_dtor entry_dtor); struct iova *find_iova(struct iova_domain *iovad, unsigned long pfn); @@ -230,8 +230,7 @@ static inline void copy_reserved_iova(struct iova_domain *from, static inline void init_iova_domain(struct iova_domain *iovad, unsigned long granule, - unsigned long start_pfn, - unsigned long pfn_32bit) + unsigned long start_pfn) { }