From patchwork Thu Sep 13 16:42:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146631 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp812410ljw; Thu, 13 Sep 2018 09:42:39 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZWUo7EGrQteDiIZoE69VcwRVAjSA32P5wp7zhQIykISGZZgg5Eh2jK1G0AMU43Z0Zv3z8j X-Received: by 2002:a17:902:b70f:: with SMTP id d15-v6mr8262486pls.53.1536856959667; Thu, 13 Sep 2018 09:42:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536856959; cv=none; d=google.com; s=arc-20160816; b=vTb8+6AXgFQa95bLWQrxjPHOBGGAGmlJUyi73MnDCk2nZdON/gjGcw+FJ0rfRDbyGS MSp7wPkHbEmlodRs+fuvkMHAZqRqVupqKUoh6SloVEcpiQ2hsm+ONZbjs4AfWT2aSUN2 iOQWCoSd5mLoNyxI1Pxz2YDl8z9Ljt/VqUSgTuNgnP163Fg/I9SLg1eG2ifVUQ4hKzqq 2AnIy6cbQsZgTNsXw9t/0Sn10aGfI4jJts2FKnJ5HDj5ReoYHEpNMy8OPUy4BL9pNY5e YKhSInDf4VNrRpTMB6zRk1oqmo2CHWtbFM+Eadbyi87zzMhrXcIWIemBxDRFIjn/NSPw ogaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=TxGcoem7bN74tOOynqUeMCYbvFifD3l7WEsIfvxmpXs=; b=nf4DzqEKP4DmJ/Sfz7ssPduhkKF3YaEqKZOE7QhE2QpJGvAQwWYDxJ/v0foG9ioi0Z wq+u5zXWS8NWLUCoySToBXwhOYcD4snBKGsuRNGksYUXtWnWrE201cM6NHcN+VOXv3FB uw6fUT/ayIgNluPqKkL91bQIxvdP5PM6Oe0RLYcfLXfIxWZYSHnRmysDkIsdvNwhOEC1 RmpQE2tW31ZYyIb5LZnVrizOefYOWA/Rf9qbQAzNIB6BkYYYwcayYfIcsqQShX6YuaWR YWKCInShLQ9XTM7s5jzVorGQLj9jdZEi+K0g3M62PACOxk05QlZxgvdKDfijfVTKvjop I5zQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 14-v6si4726138plb.230.2018.09.13.09.42.39; Thu, 13 Sep 2018 09:42:39 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728138AbeIMVwx (ORCPT + 32 others); Thu, 13 Sep 2018 17:52:53 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51100 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726452AbeIMVwx (ORCPT ); Thu, 13 Sep 2018 17:52:53 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CCD511596; Thu, 13 Sep 2018 09:42:36 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0E2483F703; Thu, 13 Sep 2018 09:42:34 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v6 1/7] iommu/arm-smmu-v3: Implement flush_iotlb_all hook Date: Thu, 13 Sep 2018 17:42:18 +0100 Message-Id: X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei .flush_iotlb_all is currently stubbed to arm_smmu_iotlb_sync() since the only time it would ever need to actually do anything is for callers doing their own explicit batching, e.g.: iommu_unmap_fast(domain, ...); iommu_unmap_fast(domain, ...); iommu_iotlb_flush_all(domain, ...); where since io-pgtable still issues the TLBI commands implicitly in the unmap instead of implementing .iotlb_range_add, the "flush" only needs to ensure completion of those already-in-flight invalidations. However, we're about to start using it in anger with flush queues, so let's get a proper implementation wired up. Signed-off-by: Zhen Lei Reviewed-by: Robin Murphy [rm: expand commit message] Signed-off-by: Robin Murphy --- drivers/iommu/arm-smmu-v3.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) -- 2.19.0.dirty diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index e395f1ff3f81..f10c852479fc 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -1781,6 +1781,14 @@ arm_smmu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) return ops->unmap(ops, iova, size); } +static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain) +{ + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + if (smmu_domain->smmu) + arm_smmu_tlb_inv_context(smmu_domain); +} + static void arm_smmu_iotlb_sync(struct iommu_domain *domain) { struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; @@ -2008,7 +2016,7 @@ static struct iommu_ops arm_smmu_ops = { .attach_dev = arm_smmu_attach_dev, .map = arm_smmu_map, .unmap = arm_smmu_unmap, - .flush_iotlb_all = arm_smmu_iotlb_sync, + .flush_iotlb_all = arm_smmu_flush_iotlb_all, .iotlb_sync = arm_smmu_iotlb_sync, .iova_to_phys = arm_smmu_iova_to_phys, .add_device = arm_smmu_add_device, From patchwork Thu Sep 13 16:42:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146632 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp812475ljw; Thu, 13 Sep 2018 09:42:43 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaqwKXvcRsSEAPpXbC0N+Iz5QeRXBTD7CO3WkWb+I6Gu8acleBRhF+fcPuhPFM/9CKq4zTO X-Received: by 2002:a17:902:4403:: with SMTP id k3-v6mr7958230pld.243.1536856963073; Thu, 13 Sep 2018 09:42:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536856963; cv=none; d=google.com; s=arc-20160816; b=JF2LaCiY0XhAcpQRxfQe6pXgmCmBxuvzer5gqUYs7S4r363vgrbX0eHlTIOSmdw1lk 7b4EO85zPTXFPtJWuK7e3b+vcxaSU1KCI4frm4n8wI9ILwVWxQnud0G+qj2FmhovZxhx EDbPVb2Bhxm/QtHbJHzgu4yvlcX8K6wMsyO16/mRJFH4OV3xELH62DjqRxETkHggDpXb usSbI4KTDLWAsv4AMihzFT5kAxTP3KoIl+afCJlgKMag1OnWBKhHDQoCiXTaGqcikRfo b2tdf0zxRyYIsnW7e4jbX3FAB1SOqaeq0ly/RIpGSm3C7nkXBpq5vqI+3R+N7QFZneNi kzfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=VO+SOhQYcjg8d4MISSqnbdk7eAwIW87RwAoIVXyZh3M=; b=qDOENlQp6msOO+jjiO+LjnEL4TIoWQkajNpaY8mbM2DLv6lPEsHxp8K696ZWX+e/co fUwymNrHxMQCKtTBOuFa0VqD3ppLjluA09XP++yv3Rw2IS+BHdONSf1/OSB7gOFTqA+4 diP4SXz1MwvGyvxLy26sOiPHZfHnfr2zXBHVdfCVt2hsMHMDR26OahguXFZQkNXlc0XQ NMCcun6jYh3fI2RGAq9ZAekkvavr1xRBtOa8o2xzkCo2lwULxmxVaACircN0T612CEwt Mfcb0LGyJaAcKo+ltBeVZXc8te/8ua+L4BqgiiOm2MBbB+wNqEzs0B+5ekFpcwDN01jy VZdg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 14-v6si4726138plb.230.2018.09.13.09.42.42; Thu, 13 Sep 2018 09:42:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728200AbeIMVw5 (ORCPT + 32 others); Thu, 13 Sep 2018 17:52:57 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51114 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726452AbeIMVw4 (ORCPT ); Thu, 13 Sep 2018 17:52:56 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A504915BF; Thu, 13 Sep 2018 09:42:39 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D93B83F703; Thu, 13 Sep 2018 09:42:37 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v6 2/7] iommu/dma: Add support for non-strict mode Date: Thu, 13 Sep 2018 17:42:19 +0100 Message-Id: X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei 1. Save the related domain pointer in struct iommu_dma_cookie, make iovad capable call domain->ops->flush_iotlb_all to flush TLB. 2. During the iommu domain initialization phase, base on domain->non_strict field to check whether non-strict mode is supported or not. If so, call init_iova_flush_queue to register iovad->flush_cb callback. 3. All unmap(contains iova-free) APIs will finally invoke __iommu_dma_unmap -->iommu_dma_free_iova. If the domain is non-strict, call queue_iova to put off iova freeing, and omit iommu_tlb_sync operation. Signed-off-by: Zhen Lei [rm: convert raw boolean to domain attribute] Signed-off-by: Robin Murphy --- drivers/iommu/dma-iommu.c | 31 ++++++++++++++++++++++++++++++- include/linux/iommu.h | 1 + 2 files changed, 31 insertions(+), 1 deletion(-) -- 2.19.0.dirty diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 511ff9a1d6d9..d91849fe4ebe 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -55,8 +55,13 @@ struct iommu_dma_cookie { }; struct list_head msi_page_list; spinlock_t msi_lock; + + /* Only be assigned in non-strict mode, otherwise it's NULL */ + struct iommu_domain *domain; }; +static bool iommu_dma_non_strict __read_mostly; + static inline size_t cookie_msi_granule(struct iommu_dma_cookie *cookie) { if (cookie->type == IOMMU_DMA_IOVA_COOKIE) @@ -257,6 +262,17 @@ static int iova_reserve_iommu_regions(struct device *dev, return ret; } +static void iommu_dma_flush_iotlb_all(struct iova_domain *iovad) +{ + struct iommu_dma_cookie *cookie; + struct iommu_domain *domain; + + cookie = container_of(iovad, struct iommu_dma_cookie, iovad); + domain = cookie->domain; + + domain->ops->flush_iotlb_all(domain); +} + /** * iommu_dma_init_domain - Initialise a DMA mapping domain * @domain: IOMMU domain previously prepared by iommu_get_dma_cookie() @@ -275,6 +291,7 @@ int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; unsigned long order, base_pfn, end_pfn; + int attr = 1; if (!cookie || cookie->type != IOMMU_DMA_IOVA_COOKIE) return -EINVAL; @@ -308,6 +325,13 @@ int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, } init_iova_domain(iovad, 1UL << order, base_pfn); + + if (iommu_dma_non_strict && !iommu_domain_set_attr(domain, + DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE, &attr)) { + cookie->domain = domain; + init_iova_flush_queue(iovad, iommu_dma_flush_iotlb_all, NULL); + } + if (!dev) return 0; @@ -393,6 +417,9 @@ static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, /* The MSI case is only ever cleaning up its most recent allocation */ if (cookie->type == IOMMU_DMA_MSI_COOKIE) cookie->msi_iova -= size; + else if (cookie->domain) /* non-strict mode */ + queue_iova(iovad, iova_pfn(iovad, iova), + size >> iova_shift(iovad), 0); else free_iova_fast(iovad, iova_pfn(iovad, iova), size >> iova_shift(iovad)); @@ -408,7 +435,9 @@ static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t dma_addr, dma_addr -= iova_off; size = iova_align(iovad, size + iova_off); - WARN_ON(iommu_unmap(domain, dma_addr, size) != size); + WARN_ON(iommu_unmap_fast(domain, dma_addr, size) != size); + if (!cookie->domain) + iommu_tlb_sync(domain); iommu_dma_free_iova(cookie, dma_addr, size); } diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 87994c265bf5..decabe8e8dbe 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -124,6 +124,7 @@ enum iommu_attr { DOMAIN_ATTR_FSL_PAMU_ENABLE, DOMAIN_ATTR_FSL_PAMUV1, DOMAIN_ATTR_NESTING, /* two stages of translation */ + DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE, DOMAIN_ATTR_MAX, }; From patchwork Thu Sep 13 16:42:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146633 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp812523ljw; Thu, 13 Sep 2018 09:42:45 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb2uGTA+Iao17jyeeZHkN4yPB1Tq3hccmQ+ScOG1hxv7f942FZgXpsEhaoqdMRaBIFaIH85 X-Received: by 2002:a63:4860:: with SMTP id x32-v6mr7886217pgk.375.1536856965737; Thu, 13 Sep 2018 09:42:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536856965; cv=none; d=google.com; s=arc-20160816; b=nXvJ4M37WG/4CqD7ViFxAjQwsP20nCK9Gb25GeDETcoV0RX5rzaAEa8/YwM9x2NY0d j6ncY3AeVjC82e4fVNFhKIVwyCw9QSri3WLxdkOheeDIp6IOwx9v4vIQbHG/j6nssrSr yA2almvTwDuZEHyA8PBgiJC8xycLgGT1YN1EEcgjuulvZQ8yx53nw3xzsSkr2b4IWoBP z1XxUS289p6kfao8BxER2d++0Eg33UMS5KSMOKjZIiKdnJsaiQhGgoON9Jj3EOClBMiB Qm7wTSEabL4omlN+VH678ZxnPVNtsjdAaH7wH7AEIrrYjZGKXPPXXftzSYbO3O69uUhm SedQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=dLiTFHPE9aU4Hn6YgQH8nqBIpwAxz2ElmjiuA4HNLw4=; b=LYqMM+6hMZg09TTWCqRGm1g7auy8WhxuScBROzmk1AZUKdNSoTTvdC1MhahzQeDmV9 M+5LXne6OO+6acdrnPPUFsHdjy3EpWLGclT9ACOxnDgvnmlT19GRA966G25ROcwokvA6 vEe1ALq0IKSPfK4Iin+Y5phAZds+nzSa+0FYGbbTTUxE5CFZ+SE307CE04jn7V1pm1Sz EG7E3IhbZsn3+4Z2+743aEjNO9PAXtgVQBiPEf7X/W88gULk38ELXWmZLhxh5ivIqaOV kqY6sh4b5mjRLPPeFTP0PxlD5CUvUkI9MXzkWwiPGco7dMoqlxVQ6TSOcSc5UL3pY/jw PisQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 14-v6si4726138plb.230.2018.09.13.09.42.45; Thu, 13 Sep 2018 09:42:45 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728287AbeIMVxA (ORCPT + 32 others); Thu, 13 Sep 2018 17:53:00 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51124 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726452AbeIMVw7 (ORCPT ); Thu, 13 Sep 2018 17:52:59 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7AA4C1682; Thu, 13 Sep 2018 09:42:42 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B03C83F703; Thu, 13 Sep 2018 09:42:40 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v6 3/7] iommu/io-pgtable-arm: Add support for non-strict mode Date: Thu, 13 Sep 2018 17:42:20 +0100 Message-Id: <842302266d7a079e78ac6db3739be8dbf8a5cdab.1536856828.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei To support non-strict mode, now we only TLBI and sync for strict mode, except for non-leaf invalidations since page table updates themselves must always be synchronous. To save having to reason about it too much, make sure the invalidation in arm_lpae_split_blk_unmap() just performs its own unconditional sync to minimise the window in which we're technically violating the break- before-make requirement on a live mapping. This might work out redundant with an outer-level sync for strict unmaps, but we'll never be splitting blocks on a DMA fastpath anyway. Signed-off-by: Zhen Lei [rm: tweak comment, commit message, and split_blk_unmap logic] Signed-off-by: Robin Murphy --- drivers/iommu/io-pgtable-arm.c | 9 ++++++--- drivers/iommu/io-pgtable.h | 5 +++++ 2 files changed, 11 insertions(+), 3 deletions(-) -- 2.19.0.dirty diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 2f79efd16a05..5b915aab7fd3 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -576,6 +576,7 @@ static size_t arm_lpae_split_blk_unmap(struct arm_lpae_io_pgtable *data, tablep = iopte_deref(pte, data); } else if (unmap_idx >= 0) { io_pgtable_tlb_add_flush(&data->iop, iova, size, size, true); + io_pgtable_tlb_sync(&data->iop); return size; } @@ -609,7 +610,7 @@ static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data, io_pgtable_tlb_sync(iop); ptep = iopte_deref(pte, data); __arm_lpae_free_pgtable(data, lvl + 1, ptep); - } else { + } else if (!(iop->cfg.quirks & IO_PGTABLE_QUIRK_NON_STRICT)) { io_pgtable_tlb_add_flush(iop, iova, size, size, true); } @@ -771,7 +772,8 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg, void *cookie) u64 reg; struct arm_lpae_io_pgtable *data; - if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS | IO_PGTABLE_QUIRK_NO_DMA)) + if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS | IO_PGTABLE_QUIRK_NO_DMA | + IO_PGTABLE_QUIRK_NON_STRICT)) return NULL; data = arm_lpae_alloc_pgtable(cfg); @@ -863,7 +865,8 @@ arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie) struct arm_lpae_io_pgtable *data; /* The NS quirk doesn't apply at stage 2 */ - if (cfg->quirks & ~IO_PGTABLE_QUIRK_NO_DMA) + if (cfg->quirks & ~(IO_PGTABLE_QUIRK_NO_DMA | + IO_PGTABLE_QUIRK_NON_STRICT)) return NULL; data = arm_lpae_alloc_pgtable(cfg); diff --git a/drivers/iommu/io-pgtable.h b/drivers/iommu/io-pgtable.h index 2df79093cad9..47d5ae559329 100644 --- a/drivers/iommu/io-pgtable.h +++ b/drivers/iommu/io-pgtable.h @@ -71,12 +71,17 @@ struct io_pgtable_cfg { * be accessed by a fully cache-coherent IOMMU or CPU (e.g. for a * software-emulated IOMMU), such that pagetable updates need not * be treated as explicit DMA data. + * + * IO_PGTABLE_QUIRK_NON_STRICT: Skip issuing synchronous leaf TLBIs + * on unmap, for DMA domains using the flush queue mechanism for + * delayed invalidation. */ #define IO_PGTABLE_QUIRK_ARM_NS BIT(0) #define IO_PGTABLE_QUIRK_NO_PERMS BIT(1) #define IO_PGTABLE_QUIRK_TLBI_ON_MAP BIT(2) #define IO_PGTABLE_QUIRK_ARM_MTK_4GB BIT(3) #define IO_PGTABLE_QUIRK_NO_DMA BIT(4) + #define IO_PGTABLE_QUIRK_NON_STRICT BIT(5) unsigned long quirks; unsigned long pgsize_bitmap; unsigned int ias; From patchwork Thu Sep 13 16:42:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146634 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp812604ljw; Thu, 13 Sep 2018 09:42:51 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbMBU8IWbnKmZjEESyWOIby/XgFcAnv9zh/E0QOokRVca4Qf38N/oYyMAGtbk0zMA4sEVId X-Received: by 2002:a17:902:ba83:: with SMTP id k3-v6mr7868550pls.251.1536856970956; Thu, 13 Sep 2018 09:42:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536856970; cv=none; d=google.com; s=arc-20160816; b=fkFJU4k8IOUgafeDB6nbcAk03lo/FgXsScQ8IaFxVqU3dx5Nfpy6N4XkzYP5y5D6QO lbOGqxky5PQ8f61U51LHwTGMtIc5a0D1iQK1ga1PnCC6p/bxljz8zUuleqJonK3lHBl8 PBtcAwrGPxveRJpL9K4SjI8xVuc/iXhmy8pexNgTa1JmP9Fi6P34ymJXo4AtS0FQzygx eihD+tgUxZmgvbxBDBx5tX6eU5V388CvkkJaHoEyW+oHwP4mb+hDOYhiZlvlInb+vAmI grZqUNg04uh0hc/nwzVxZD/F7uIPWUsO/cF2oeXR7+5zyJoqQG4QR+WYhnZGtvpFwo4q 9nLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=ufOysDayhCv8ZRLRyWmygAPRHXY2m3YhIpRsoagW8Xk=; b=crLze3wVFSMHEo0AMDR4u596Hz2GOoz38IJizZ5u85RGOTbMxc0xVlFigIrLS1hAAh 0476Kn8o9FyYq4zPPiiqqrqhT7gUuYKYLDMoKGQVU88o9COYwDcbpmS3hfd3nYNYabk8 5OE7GgyPW5NF9C6MDDkvM47CIG5dfaU5UJ8Jpy6WEzNi0RB7HegJEKE/JK5A6gc9ZqOr sDCe5bQkjvO4/YRJd0jishqQdO0lT7iNrqYdPRXdP/fZa/ZstgbOWIqltSCujH8Q/z6k g0z2NsOseR5YYnKakiV7KwspOuaB3nJvgk/7rfZXsiQPQN93GwAJkLsfIwErnQl6bW/E 05ow== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b2-v6si5118969plm.202.2018.09.13.09.42.50; Thu, 13 Sep 2018 09:42:50 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728380AbeIMVxG (ORCPT + 32 others); Thu, 13 Sep 2018 17:53:06 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51144 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726452AbeIMVxE (ORCPT ); Thu, 13 Sep 2018 17:53:04 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 65AD080D; Thu, 13 Sep 2018 09:42:47 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9AC603F703; Thu, 13 Sep 2018 09:42:45 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v6 5/7] iommu/arm-smmu-v3: Add support for non-strict mode Date: Thu, 13 Sep 2018 17:42:22 +0100 Message-Id: <9804a057ac2fcd4d7a5c3878a7e6daa33751c9fc.1536856828.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei Dynamically choose strict or non-strict mode for page table config based on the iommu domain type. Signed-off-by: Zhen Lei [rm: convert to domain attribute] Signed-off-by: Robin Murphy --- drivers/iommu/arm-smmu-v3.c | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) -- 2.19.0.dirty diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index f10c852479fc..e2f0e4a3374d 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -612,6 +612,7 @@ struct arm_smmu_domain { struct mutex init_mutex; /* Protects smmu pointer */ struct io_pgtable_ops *pgtbl_ops; + bool non_strict; enum arm_smmu_domain_stage stage; union { @@ -1633,6 +1634,9 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain) if (smmu->features & ARM_SMMU_FEAT_COHERENCY) pgtbl_cfg.quirks = IO_PGTABLE_QUIRK_NO_DMA; + if (smmu_domain->non_strict) + pgtbl_cfg.quirks |= IO_PGTABLE_QUIRK_NON_STRICT; + pgtbl_ops = alloc_io_pgtable_ops(fmt, &pgtbl_cfg, smmu_domain); if (!pgtbl_ops) return -ENOMEM; @@ -1934,13 +1938,17 @@ static int arm_smmu_domain_get_attr(struct iommu_domain *domain, { struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); - if (domain->type != IOMMU_DOMAIN_UNMANAGED) - return -EINVAL; - switch (attr) { case DOMAIN_ATTR_NESTING: + if (domain->type != IOMMU_DOMAIN_UNMANAGED) + return -EINVAL; *(int *)data = (smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED); return 0; + case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE: + if (domain->type != IOMMU_DOMAIN_DMA) + return -EINVAL; + *(int *)data = smmu_domain->non_strict; + return 0; default: return -ENODEV; } @@ -1952,13 +1960,15 @@ static int arm_smmu_domain_set_attr(struct iommu_domain *domain, int ret = 0; struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); - if (domain->type != IOMMU_DOMAIN_UNMANAGED) - return -EINVAL; - mutex_lock(&smmu_domain->init_mutex); switch (attr) { case DOMAIN_ATTR_NESTING: + if (domain->type != IOMMU_DOMAIN_UNMANAGED) { + ret = -EINVAL; + goto out_unlock; + } + if (smmu_domain->smmu) { ret = -EPERM; goto out_unlock; @@ -1970,6 +1980,17 @@ static int arm_smmu_domain_set_attr(struct iommu_domain *domain, smmu_domain->stage = ARM_SMMU_DOMAIN_S1; break; + case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE: + if (domain->type != IOMMU_DOMAIN_DMA) { + ret = -EINVAL; + goto out_unlock; + } + + smmu_domain->non_strict = *(int *)data; + if (smmu_domain->pgtbl_ops) + io_pgtable_set_non_strict(smmu_domain->pgtbl_ops, + smmu_domain->non_strict); + break; default: ret = -ENODEV; } From patchwork Thu Sep 13 16:42:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Murphy X-Patchwork-Id: 146635 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp812698ljw; Thu, 13 Sep 2018 09:42:56 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZSjKu0cRdAuQ6FQFTrG8ChCUKf1akSmAGv+3YnB+HoMbh4Aj8/KDtEIr8zjJUIn67lwwgN X-Received: by 2002:a63:d857:: with SMTP id k23-v6mr7970772pgj.106.1536856976834; Thu, 13 Sep 2018 09:42:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536856976; cv=none; d=google.com; s=arc-20160816; b=Y/Sj/v4Xnf/oS9ECm/2YQf6Gts9LRizibmegrEHuIM74uxZnBDcTHWtsQ53apC8o2R pEf+CI3s/zM4+QZyoh1aVwVUv+8bOBzRJcWqnczUZPHJjnYqFR7FLlX5X9VTEuGAIAok 3PD05Nk5XzyFvUckZdErgl56WjBIXNMTOFwP60sxYCwRbE0Nt0gDiC+fumkvFH9km1Nh VigsCS3Bdo0uc9k9dxJ/+0hvFNQlAMmUXJ/YiDa7uWDUjZKw/Nh2qxVwWICnwfLcJRqb 8IMYizj3sd9MR05qcXFnluREnOHtalyy5NELgyB1g2MjtqwSTGo9d+t5SDwvUKkTzg7n Q9vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=HlQCIJcDGbf8MlvcgxIrSI2BVoFoO56b4FRl8XMW144=; b=QLQuUfvmJu81/cPM7hJ6oqbP6dhzb2t3Pv6i1PqdC7+mUwqZFqAZFh+t2ETSgUp2HH YfKArqDk5ttOLKNq2xmW/Zwe3RFkhSyRIwPGv6ENjJSA6frg9KKALmg70MVOdLueBsl4 JxnyEf2rwOEj/dhxKZvamHc7oXjPpA2tE/nhQx0WdAIi4uB2geFJPtvua9vG/vIWvPDc RZbPgCg2JeV/IMycPmjjA3pkgSMrJ3L9GGMnTvnz9KnqPAgOVWYXEQ5BroT9aHCQHNbp E+TT1HfAIYZK4gtP2dv/icxjwyTdsYNZiACdgUyqQBbE/+W7kVfKTK810vdtTei+K5z4 J1OQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r84-v6si4440313pfj.355.2018.09.13.09.42.56; Thu, 13 Sep 2018 09:42:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728468AbeIMVxK (ORCPT + 32 others); Thu, 13 Sep 2018 17:53:10 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:51170 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728430AbeIMVxJ (ORCPT ); Thu, 13 Sep 2018 17:53:09 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 76FDD1684; Thu, 13 Sep 2018 09:42:52 -0700 (PDT) Received: from e110467-lin.cambridge.arm.com (e110467-lin.emea.arm.com [10.4.12.131]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AC5243F703; Thu, 13 Sep 2018 09:42:50 -0700 (PDT) From: Robin Murphy To: joro@8bytes.org, will.deacon@arm.com, thunder.leizhen@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: linuxarm@huawei.com, guohanjun@huawei.com, huawei.libin@huawei.com, john.garry@huawei.com Subject: [PATCH v6 7/7] iommu/dma: Add bootup option "iommu.non_strict" Date: Thu, 13 Sep 2018 17:42:24 +0100 Message-Id: <94fe2e15d36565e13218b007b9d5d9d08f190e5e.1536856828.git.robin.murphy@arm.com> X-Mailer: git-send-email 2.19.0.dirty In-Reply-To: References: MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Zhen Lei Add a bootup option to make the system manager can choose which mode to be used. The default mode is strict. Signed-off-by: Zhen Lei [rm: make it a generic iommu-dma feature] Signed-off-by: Robin Murphy --- .../admin-guide/kernel-parameters.txt | 13 +++++++++++++ drivers/iommu/dma-iommu.c | 18 ++++++++++++++++++ 2 files changed, 31 insertions(+) -- 2.19.0.dirty diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 9871e649ffef..406b91759b62 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1749,6 +1749,19 @@ nobypass [PPC/POWERNV] Disable IOMMU bypass, using IOMMU for PCI devices. + iommu.non_strict= [ARM64] + Format: { "0" | "1" } + 0 - strict mode, default. + Release IOVAs after the related TLBs are invalid + completely. + 1 - non-strict mode. + Put off TLBs invalidation and release memory first. + It's good for scatter-gather performance but lacks + full isolation, an untrusted device can access the + reused memory because the TLBs may still valid. + Please take full consideration before choosing this + mode. Note that, VFIO will always use strict mode. + iommu.passthrough= [ARM64] Configure DMA to bypass the IOMMU by default. Format: { "0" | "1" } diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index d91849fe4ebe..04d4c5453acd 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -62,6 +62,24 @@ struct iommu_dma_cookie { static bool iommu_dma_non_strict __read_mostly; +static int __init iommu_dma_setup(char *str) +{ + int ret; + + ret = kstrtobool(str, &iommu_dma_non_strict); + if (ret) + return ret; + + if (iommu_dma_non_strict) { + pr_warn("WARNING: iommu non-strict mode is chosen.\n" + "It's good for scatter-gather performance but lacks full isolation\n"); + add_taint(TAINT_WARN, LOCKDEP_STILL_OK); + } + + return 0; +} +early_param("iommu.non_strict", iommu_dma_setup); + static inline size_t cookie_msi_granule(struct iommu_dma_cookie *cookie) { if (cookie->type == IOMMU_DMA_IOVA_COOKIE)