From patchwork Tue May 19 17:54:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99DAEC433E3 for ; Tue, 19 May 2020 18:01:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 80A56207D3 for ; Tue, 19 May 2020 18:01:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="v3uSYjcT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726059AbgESSBp (ORCPT ); Tue, 19 May 2020 14:01:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726747AbgESSBo (ORCPT ); Tue, 19 May 2020 14:01:44 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DB3DC08C5C1 for ; Tue, 19 May 2020 11:01:44 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id 50so366660wrc.11 for ; Tue, 19 May 2020 11:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GyR/0Ui23R5AHEgwKIBkr1oNL8Gb0ETBbsB9Bvf72kY=; b=v3uSYjcTrWI5/qXsAKPtQitwafYDySNKobI2mT1/dgOS6rzcfLNMmKF8Zt+y0lAIG+ 7NlQH+xDT61+aKM9dVrX/ST+T5q9rAVx5upfWAkNFK1yIBfi0CAVIN2p765NEfOSC08U sFwBSa6m6vidPtQ10csw1xhsYdtx+Clr66gxG7w/lPymxvLly5LhE7Um5QY8fQhUpWaJ xdgGAMGjYoX5ObjkOmVHefDcNQj8HEkXT7/nacUbxTsEOjESUGfhUPoGor1ifWuorzlX L6qNerDYVI4QEctFS67Z/P9rCH1XE1xvD8PSxU2aJEL1im0fmYXq7/33gxfnNONbYidw T/Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GyR/0Ui23R5AHEgwKIBkr1oNL8Gb0ETBbsB9Bvf72kY=; b=jgO6VQiZVTfPuLpZFBRYXGiK+iNXSRaiDgpe7t8LBhfhMcyVmSPUnQmlPHbBRr3IJZ s+ZWFi2CPzasQzMpFNxagagygyLo8Fu+m97H4HyofBI/SLQeMVMlxNKwQRipeLmBS/Ab KERoQjWripqYAakqxvcOYeIg5kSeyfKpr0kaZNgFcT1O0PCq//YU5MCRKmkTVBQM+f+U 2uqvZQeBFHTd2t5CucHq9ACRYT5a2LGc4OYJPldeEGSy52ySQpg5c/pDrvC0LmLILf6i u0qKlJkGAfQKyMct3P0pyRQP45Uxc5zY4oGUjNNwu4cbuZF30mTpGLhncHMyYYrKvoNY V2cA== X-Gm-Message-State: AOAM531ffvU2Ns0W/q9MxWb9fjBaYeJSpi+nBD6ZguTrk5/PQKb+naRa YJDVZG6MHBvNSKv1YKo/YGo6Fu3bT0GrVg== X-Google-Smtp-Source: ABdhPJyPXdWabIumTknX9KHGpAfZMDP+D+8kcHDQrtNtnJy5wEFWx5RISo0BU/ne7okq5g9/m6U84w== X-Received: by 2002:a5d:694d:: with SMTP id r13mr112031wrw.238.1589911302937; Tue, 19 May 2020 11:01:42 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.01.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:01:42 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v7 01/24] mm: Add a PASID field to mm_struct Date: Tue, 19 May 2020 19:54:39 +0200 Message-Id: <20200519175502.2504091-2-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Some devices can tag their DMA requests with a 20-bit Process Address Space ID (PASID), allowing them to access multiple address spaces. In combination with recoverable I/O page faults (for example PCIe PRI), PASID allows the IOMMU to share page tables with the MMU. To make sure that a single PASID is allocated for each address space, as required by Intel ENQCMD, store the PASID in the mm_struct. The IOMMU driver is in charge of serializing modifications to the PASID field. Signed-off-by: Jean-Philippe Brucker --- include/linux/mm_types.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 4aba6c0c2ba8..8db647275817 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -534,6 +534,10 @@ struct mm_struct { atomic_long_t hugetlb_usage; #endif struct work_struct async_put_work; +#ifdef CONFIG_IOMMU_SUPPORT + /* Address space ID used by device DMA */ + unsigned int pasid; +#endif } __randomize_layout; /* From patchwork Tue May 19 17:54:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55F6BC433E5 for ; Tue, 19 May 2020 18:01:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D767207D3 for ; Tue, 19 May 2020 18:01:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="LrVLTpbG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727009AbgESSBr (ORCPT ); Tue, 19 May 2020 14:01:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727045AbgESSBq (ORCPT ); Tue, 19 May 2020 14:01:46 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F377C08C5C0 for ; Tue, 19 May 2020 11:01:45 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id l11so447735wru.0 for ; Tue, 19 May 2020 11:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2PFJF5btJNBPprEd/LjKgNayT4ugYCOJnPARFwzNang=; b=LrVLTpbGU4NiphecFL5M4ouOgZufinFzBCfZAELELYApdrOCEHujSXy+FUdD42qX8l 8vjiQkUpQllNaT36Vs+gVTDsK0mCFuQ4hQ5Y5M23lUpiTU6YsaHYDzi9lNFE4T8WHXeb NZrBW40uyUB/AajvgtfXJEDwUayE2sLlUwgHNn+HnsB/dijnotLjnkjd83m8g012WOOf rdixreNwYbkTse3eCKAZLtShfMQ+jSekNlwF9YLtQB6Hkp2JTm1Q0bhjWGXr1Y9Fvbez rfin57fhrSJY8eRT3Rr+KPlNcOf+8a2Yn81ZFbRYIooZJe/UEU5y2m5zLbjU+BdEftqx g0yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2PFJF5btJNBPprEd/LjKgNayT4ugYCOJnPARFwzNang=; b=QCUvcGDqUWCx21TJV1jZFqvjl2zNI/YSgeCrG9rTVWkOdT/UsQqPjFJX8Zx4JLJAt4 IXQLUAFtFqkL8J5U8JYH8w9jvPDLghfQmf70wqf4Xmm4CNSPQmuKAhs9ZvHgoSRkb1M7 CEugqBkEhFxair7pUH/b2mO4CPD//NadIVmbDAQ9VoGIzqxr7PmLb3kKQeK672mQCfJG F6ym+KcPlCOZfX0e9u37HOB3NJmavMuxZVTVkbDRBk7kvgRLy4J8COR8pM10W3J8Vecg /9TzTd48Q1dvM4aDKELtFGnKrlTPRMWQxCoaczsNGNYD4wMos0TtNrWwy2u21e39qOSH 1wbw== X-Gm-Message-State: AOAM5304l44q26nSW3KL2I4fC/3Hy9NPy2YLzZ3tDZeiPtfmJTlti8gI ZvFCi+V3KQ/m0Ug5Jvzd/7K86Q== X-Google-Smtp-Source: ABdhPJztpovo6qlRG1M54wjBp9zTXl5RwTuJ1SnYrJPYt/7mMDTUOS9LfLeUBDTtOc3yWZxPnijhIQ== X-Received: by 2002:a5d:5048:: with SMTP id h8mr76396wrt.293.1589911303946; Tue, 19 May 2020 11:01:43 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:01:43 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v7 02/24] iommu/ioasid: Add ioasid references Date: Tue, 19 May 2020 19:54:40 +0200 Message-Id: <20200519175502.2504091-3-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Let IOASID users take references to existing ioasids with ioasid_get(). ioasid_put() drops a reference and only frees the ioasid when its reference number is zero. It returns true if the ioasid was freed. For drivers that don't call ioasid_get(), ioasid_put() is the same as ioasid_free(). Signed-off-by: Jean-Philippe Brucker --- v6->v7: rename ioasid_free() to ioasid_put(), add WARN in ioasid_get() --- include/linux/ioasid.h | 10 ++++++++-- drivers/iommu/intel-iommu.c | 4 ++-- drivers/iommu/intel-svm.c | 6 +++--- drivers/iommu/ioasid.c | 38 +++++++++++++++++++++++++++++++++---- 4 files changed, 47 insertions(+), 11 deletions(-) diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h index 6f000d7a0ddc..e9dacd4b9f6b 100644 --- a/include/linux/ioasid.h +++ b/include/linux/ioasid.h @@ -34,7 +34,8 @@ struct ioasid_allocator_ops { #if IS_ENABLED(CONFIG_IOASID) ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, void *private); -void ioasid_free(ioasid_t ioasid); +void ioasid_get(ioasid_t ioasid); +bool ioasid_put(ioasid_t ioasid); void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, bool (*getter)(void *)); int ioasid_register_allocator(struct ioasid_allocator_ops *allocator); @@ -48,10 +49,15 @@ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, return INVALID_IOASID; } -static inline void ioasid_free(ioasid_t ioasid) +static inline void ioasid_get(ioasid_t ioasid) { } +static inline bool ioasid_put(ioasid_t ioasid) +{ + return false; +} + static inline void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, bool (*getter)(void *)) { diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index ed21ce6d1238..0230f35480ee 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -5432,7 +5432,7 @@ static void auxiliary_unlink_device(struct dmar_domain *domain, domain->auxd_refcnt--; if (!domain->auxd_refcnt && domain->default_pasid > 0) - ioasid_free(domain->default_pasid); + ioasid_put(domain->default_pasid); } static int aux_domain_add_dev(struct dmar_domain *domain, @@ -5494,7 +5494,7 @@ static int aux_domain_add_dev(struct dmar_domain *domain, spin_unlock(&iommu->lock); spin_unlock_irqrestore(&device_domain_lock, flags); if (!domain->auxd_refcnt && domain->default_pasid > 0) - ioasid_free(domain->default_pasid); + ioasid_put(domain->default_pasid); return ret; } diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c index 2998418f0a38..86f1264bd07c 100644 --- a/drivers/iommu/intel-svm.c +++ b/drivers/iommu/intel-svm.c @@ -353,7 +353,7 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ if (mm) { ret = mmu_notifier_register(&svm->notifier, mm); if (ret) { - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); kfree(svm); kfree(sdev); goto out; @@ -371,7 +371,7 @@ int intel_svm_bind_mm(struct device *dev, int *pasid, int flags, struct svm_dev_ if (ret) { if (mm) mmu_notifier_unregister(&svm->notifier, mm); - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); kfree(svm); kfree(sdev); goto out; @@ -447,7 +447,7 @@ int intel_svm_unbind_mm(struct device *dev, int pasid) kfree_rcu(sdev, rcu); if (list_empty(&svm->devs)) { - ioasid_free(svm->pasid); + ioasid_put(svm->pasid); if (svm->mm) mmu_notifier_unregister(&svm->notifier, svm->mm); list_del(&svm->list); diff --git a/drivers/iommu/ioasid.c b/drivers/iommu/ioasid.c index 0f8dd377aada..50ee27bbd04e 100644 --- a/drivers/iommu/ioasid.c +++ b/drivers/iommu/ioasid.c @@ -2,7 +2,7 @@ /* * I/O Address Space ID allocator. There is one global IOASID space, split into * subsets. Users create a subset with DECLARE_IOASID_SET, then allocate and - * free IOASIDs with ioasid_alloc and ioasid_free. + * free IOASIDs with ioasid_alloc and ioasid_put. */ #include #include @@ -15,6 +15,7 @@ struct ioasid_data { struct ioasid_set *set; void *private; struct rcu_head rcu; + refcount_t refs; }; /* @@ -314,6 +315,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, data->set = set; data->private = private; + refcount_set(&data->refs, 1); /* * Custom allocator needs allocator data to perform platform specific @@ -346,11 +348,34 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, EXPORT_SYMBOL_GPL(ioasid_alloc); /** - * ioasid_free - Free an IOASID + * ioasid_get - obtain a reference to the IOASID + */ +void ioasid_get(ioasid_t ioasid) +{ + struct ioasid_data *ioasid_data; + + spin_lock(&ioasid_allocator_lock); + ioasid_data = xa_load(&active_allocator->xa, ioasid); + if (ioasid_data) + refcount_inc(&ioasid_data->refs); + else + WARN_ON(1); + spin_unlock(&ioasid_allocator_lock); +} +EXPORT_SYMBOL_GPL(ioasid_get); + +/** + * ioasid_put - Release a reference to an ioasid * @ioasid: the ID to remove + * + * Put a reference to the IOASID, free it when the number of references drops to + * zero. + * + * Return: %true if the IOASID was freed, %false otherwise. */ -void ioasid_free(ioasid_t ioasid) +bool ioasid_put(ioasid_t ioasid) { + bool free = false; struct ioasid_data *ioasid_data; spin_lock(&ioasid_allocator_lock); @@ -360,6 +385,10 @@ void ioasid_free(ioasid_t ioasid) goto exit_unlock; } + free = refcount_dec_and_test(&ioasid_data->refs); + if (!free) + goto exit_unlock; + active_allocator->ops->free(ioasid, active_allocator->ops->pdata); /* Custom allocator needs additional steps to free the xa element */ if (active_allocator->flags & IOASID_ALLOCATOR_CUSTOM) { @@ -369,8 +398,9 @@ void ioasid_free(ioasid_t ioasid) exit_unlock: spin_unlock(&ioasid_allocator_lock); + return free; } -EXPORT_SYMBOL_GPL(ioasid_free); +EXPORT_SYMBOL_GPL(ioasid_put); /** * ioasid_find - Find IOASID data From patchwork Tue May 19 17:54:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7196C433E0 for ; Tue, 19 May 2020 18:01:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AB5EA207D3 for ; Tue, 19 May 2020 18:01:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="nQU00t3D" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728686AbgESSBv (ORCPT ); Tue, 19 May 2020 14:01:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727844AbgESSBu (ORCPT ); Tue, 19 May 2020 14:01:50 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EBB13C08C5C1 for ; Tue, 19 May 2020 11:01:49 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id e16so393855wra.7 for ; Tue, 19 May 2020 11:01:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NSaXkG4DBzNCdQ4OZDMt/0atg6IYfB3L/KPL3V3BCYk=; b=nQU00t3D3kWSaVXfKB6ILMM8HhndFO++HUNfNGnl/4ZnY/Rhi7BHTDXR9BpRR0c0qT z19ba52YPIwtH4OcM+u0Xdf2tw84kXX05CqOe48/inMpbcHCXlyR9H10fSIDK59pInIA 15/+tVGMmO+cfZN3/1D4OlqeHoUN05oqhn13vYpWJHIuNsEADi+xswr96WrPDIde7sN+ wBoY5thPuyrhQlVYF77YHWxFTUSrBCzatmWZFsnpNMn89Pg9cHgJG36M0/aja3AbRcNw VHZVVAUGNBCFwjuZNdBhxmesYIBszgrGeKkhSFOIx+ZJbzmAQ0Bv4FSvtSu2/5TpI/jH MLkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NSaXkG4DBzNCdQ4OZDMt/0atg6IYfB3L/KPL3V3BCYk=; b=Cc2dPf/lJUylQSqOUZFLOAubomle5fQkUvgToIrUu15fPSVMCtCoT/cpyeo1byQ5bR m3/IG2J9fjEeucv9K22a5RQdQzlzBn2A1qG+LJvmG74TE/ubC743o2OS8OQ4of+OSQls alBTQHSqKiZXayqMznq3gMK3BL7+VTxbWeZsxAhkAKYz+7mMtyuzPwFIUuqZGpVEvkK0 vYEYw27a8eudGl+hItndVPdy2490Rr/Yx+lvWbiOEY+PurP+INHW4Cow9PRDx2lSo40V Vz6Zu+QwnqMYAe3U6xHCbvs0Rm6YX8t3Y1TWOPooykR/FQQ7cX9J/CI+pzgsZQXQyELw 513Q== X-Gm-Message-State: AOAM532hPPao3sO8p3hP3SJz6OiwDCQfVMSlVscigrUiuefpDkipX1zA HZ08Qd8aHVvs4tS2e1A9NYK3uA== X-Google-Smtp-Source: ABdhPJwNMcsg8nvKB0kC8qUGaFg91bhfG5z/uwW4vZV0zE+ZApr2q4kMr3hbdFGf3VvkTxQmHPCTfQ== X-Received: by 2002:adf:f981:: with SMTP id f1mr114539wrr.244.1589911308565; Tue, 19 May 2020 11:01:48 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.01.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:01:48 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v7 06/24] arm64: mm: Pin down ASIDs for sharing mm with devices Date: Tue, 19 May 2020 19:54:44 +0200 Message-Id: <20200519175502.2504091-7-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org To enable address space sharing with the IOMMU, introduce mm_context_get() and mm_context_put(), that pin down a context and ensure that it will keep its ASID after a rollover. Export the symbols to let the modular SMMUv3 driver use them. Pinning is necessary because a device constantly needs a valid ASID, unlike tasks that only require one when running. Without pinning, we would need to notify the IOMMU when we're about to use a new ASID for a task, and it would get complicated when a new task is assigned a shared ASID. Consider the following scenario with no ASID pinned: 1. Task t1 is running on CPUx with shared ASID (gen=1, asid=1) 2. Task t2 is scheduled on CPUx, gets ASID (1, 2) 3. Task tn is scheduled on CPUy, a rollover occurs, tn gets ASID (2, 1) We would now have to immediately generate a new ASID for t1, notify the IOMMU, and finally enable task tn. We are holding the lock during all that time, since we can't afford having another CPU trigger a rollover. The IOMMU issues invalidation commands that can take tens of milliseconds. It gets needlessly complicated. All we wanted to do was schedule task tn, that has no business with the IOMMU. By letting the IOMMU pin tasks when needed, we avoid stalling the slow path, and let the pinning fail when we're out of shareable ASIDs. After a rollover, the allocator expects at least one ASID to be available in addition to the reserved ones (one per CPU). So (NR_ASIDS - NR_CPUS - 1) is the maximum number of ASIDs that can be shared with the IOMMU. Signed-off-by: Jean-Philippe Brucker --- arch/arm64/include/asm/mmu.h | 1 + arch/arm64/include/asm/mmu_context.h | 11 +++- arch/arm64/mm/context.c | 95 +++++++++++++++++++++++++++- 3 files changed, 104 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 68140fdd89d6..bbdd291e31d5 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -19,6 +19,7 @@ typedef struct { atomic64_t id; + unsigned long pinned; void *vdso; unsigned long flags; } mm_context_t; diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index ab46187c6300..69599a64945b 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -177,7 +177,13 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp) #define destroy_context(mm) do { } while(0) void check_and_switch_context(struct mm_struct *mm, unsigned int cpu); -#define init_new_context(tsk,mm) ({ atomic64_set(&(mm)->context.id, 0); 0; }) +static inline int +init_new_context(struct task_struct *tsk, struct mm_struct *mm) +{ + atomic64_set(&mm->context.id, 0); + mm->context.pinned = 0; + return 0; +} #ifdef CONFIG_ARM64_SW_TTBR0_PAN static inline void update_saved_ttbr0(struct task_struct *tsk, @@ -250,6 +256,9 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next, void verify_cpu_asid_bits(void); void post_ttbr_update_workaround(void); +unsigned long mm_context_get(struct mm_struct *mm); +void mm_context_put(struct mm_struct *mm); + #endif /* !__ASSEMBLY__ */ #endif /* !__ASM_MMU_CONTEXT_H */ diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index d702d60e64da..d0ddd413f564 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -27,6 +27,10 @@ static DEFINE_PER_CPU(atomic64_t, active_asids); static DEFINE_PER_CPU(u64, reserved_asids); static cpumask_t tlb_flush_pending; +static unsigned long max_pinned_asids; +static unsigned long nr_pinned_asids; +static unsigned long *pinned_asid_map; + #define ASID_MASK (~GENMASK(asid_bits - 1, 0)) #define ASID_FIRST_VERSION (1UL << asid_bits) @@ -74,6 +78,9 @@ void verify_cpu_asid_bits(void) static void set_kpti_asid_bits(void) { + unsigned int k; + u8 *dst = (u8 *)asid_map; + u8 *src = (u8 *)pinned_asid_map; unsigned int len = BITS_TO_LONGS(NUM_USER_ASIDS) * sizeof(unsigned long); /* * In case of KPTI kernel/user ASIDs are allocated in @@ -81,7 +88,8 @@ static void set_kpti_asid_bits(void) * is set, then the ASID will map only userspace. Thus * mark even as reserved for kernel. */ - memset(asid_map, 0xaa, len); + for (k = 0; k < len; k++) + dst[k] = src[k] | 0xaa; } static void set_reserved_asid_bits(void) @@ -89,7 +97,7 @@ static void set_reserved_asid_bits(void) if (arm64_kernel_unmapped_at_el0()) set_kpti_asid_bits(); else - bitmap_clear(asid_map, 0, NUM_USER_ASIDS); + bitmap_copy(asid_map, pinned_asid_map, NUM_USER_ASIDS); } #define asid_gen_match(asid) \ @@ -165,6 +173,14 @@ static u64 new_context(struct mm_struct *mm) if (check_update_reserved_asid(asid, newasid)) return newasid; + /* + * If it is pinned, we can keep using it. Note that reserved + * takes priority, because even if it is also pinned, we need to + * update the generation into the reserved_asids. + */ + if (mm->context.pinned) + return newasid; + /* * We had a valid ASID in a previous life, so try to re-use * it if possible. @@ -254,6 +270,68 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) cpu_switch_mm(mm->pgd, mm); } +unsigned long mm_context_get(struct mm_struct *mm) +{ + unsigned long flags; + u64 asid; + + raw_spin_lock_irqsave(&cpu_asid_lock, flags); + + asid = atomic64_read(&mm->context.id); + + if (mm->context.pinned) { + mm->context.pinned++; + asid &= ~ASID_MASK; + goto out_unlock; + } + + if (nr_pinned_asids >= max_pinned_asids) { + asid = 0; + goto out_unlock; + } + + if (!asid_gen_match(asid)) { + /* + * We went through one or more rollover since that ASID was + * used. Ensure that it is still valid, or generate a new one. + */ + asid = new_context(mm); + atomic64_set(&mm->context.id, asid); + } + + asid &= ~ASID_MASK; + + nr_pinned_asids++; + __set_bit(asid2idx(asid), pinned_asid_map); + mm->context.pinned++; + +out_unlock: + raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); + + /* Set the equivalent of USER_ASID_BIT */ + if (asid && IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0)) + asid |= 1; + + return asid; +} +EXPORT_SYMBOL_GPL(mm_context_get); + +void mm_context_put(struct mm_struct *mm) +{ + unsigned long flags; + u64 asid = atomic64_read(&mm->context.id) & ~ASID_MASK; + + raw_spin_lock_irqsave(&cpu_asid_lock, flags); + + if (--mm->context.pinned == 0) { + __clear_bit(asid2idx(asid), pinned_asid_map); + nr_pinned_asids--; + } + + raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); +} +EXPORT_SYMBOL_GPL(mm_context_put); + /* Errata workaround post TTBRx_EL1 update. */ asmlinkage void post_ttbr_update_workaround(void) { @@ -303,6 +381,13 @@ static int asids_update_limit(void) WARN_ON(num_available_asids - 1 <= num_possible_cpus()); pr_info("ASID allocator initialised with %lu entries\n", num_available_asids); + + /* + * We assume that an ASID is always available after a rollover. This + * means that even if all CPUs have a reserved ASID, there still is at + * least one slot available in the asid map. + */ + max_pinned_asids = num_available_asids - num_possible_cpus() - 2; return 0; } arch_initcall(asids_update_limit); @@ -317,6 +402,12 @@ static int asids_init(void) panic("Failed to allocate bitmap for %lu ASIDs\n", NUM_USER_ASIDS); + pinned_asid_map = kcalloc(BITS_TO_LONGS(NUM_USER_ASIDS), + sizeof(*pinned_asid_map), GFP_KERNEL); + if (!pinned_asid_map) + panic("Failed to allocate pinned ASID bitmap\n"); + nr_pinned_asids = 0; + /* * We cannot call set_reserved_asid_bits() here because CPU * caps are not finalized yet, so it is safer to assume KPTI From patchwork Tue May 19 17:54:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDC7CC433F4 for ; Tue, 19 May 2020 18:01:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C215D207D3 for ; Tue, 19 May 2020 18:01:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="PVltUbDt" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728875AbgESSBx (ORCPT ); Tue, 19 May 2020 14:01:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727066AbgESSBw (ORCPT ); Tue, 19 May 2020 14:01:52 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EFE4C08C5C0 for ; Tue, 19 May 2020 11:01:52 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id u188so220181wmu.1 for ; Tue, 19 May 2020 11:01:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OlE3rnOB9MCKbV81bgfQSo1zIBpkt/AF9XwsE8EZHww=; b=PVltUbDtiUNJUYohQuAJijXOuB5A5bH0xkzVWvUq1H2+e2ibU19HCBKJgHG0LpvJOO Ig0UqvmrBwPjchj2eoYBA6yTE2yZMjt27StiBfvKzwPY0A4BZboeHT5Mq2aJCYWh6O3p Q4Dn6OKJZaLDx+E+zM3jpPFK6EoXV1JbeExIqt2da3BtZrBlufWbqYdyZBpifE02CiWo Dpcps1X5V7yvnFwAb6dgOCso+myoqErPY0GbTB3T56uonYvfNM2sdhN6ZZlzeHJdmz9V sW1KoZznYpI78h/lNulsayaggPntaZw0EAA8Mo+eWW+YEWyHD4MAwpzLho0o134xN2lw mKKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OlE3rnOB9MCKbV81bgfQSo1zIBpkt/AF9XwsE8EZHww=; b=qSufx/X6EgqGq+4KEzg22vl028k66XXoNbWO6CNQlXTvelmUadFr+781ZwtVb9uzwW 486S9E2DVEoHkQm+bKIz7DHwslxuLuYwOPH4S5i00zdQ7WbyYF7OVFmGkIqk14JwF8VW dxCIbiRNo4nVMiUIBDqDI4Zari1f3am3L3dDRM/uC+SwhqB970RjEbZ3LyBZwwEzBW9V TfmyjceFArCnwNuMW8pnzrVWGkgXkGDpa22yQed0iJ+bNdGwYFvByx7i1F7EIUx4ZuQg 8FUZoHaeJt1rq7kd5WMVQ4Teob5Xx9VQIozQkWkybrFa5YsEl+q7lOq5y6RVnC36VKCB WCMg== X-Gm-Message-State: AOAM533IC6nOy/lqysueBPKNuTSB4SWMKjd+gReDOm2mAZNla38KryU9 gI5NshloxFz57tRGFUHbD4gbyw== X-Google-Smtp-Source: ABdhPJxm6LwKarbOgqdR2lNw/X/XowWuMGizAMxxxXwhldyYMSyTlek4+lJ4QVEwMcaSb4ml0HljTw== X-Received: by 2002:a1c:2e41:: with SMTP id u62mr615750wmu.91.1589911310989; Tue, 19 May 2020 11:01:50 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.01.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:01:50 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v7 08/24] iommu/arm-smmu-v3: Manage ASIDs with xarray Date: Tue, 19 May 2020 19:54:46 +0200 Message-Id: <20200519175502.2504091-9-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org In preparation for sharing some ASIDs with the CPU, use a global xarray to store ASIDs and their context. ASID#0 is now reserved, and the ASID space is global. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index b5467e3e9250..847c7de0a93f 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -667,7 +667,6 @@ struct arm_smmu_device { #define ARM_SMMU_MAX_ASIDS (1 << 16) unsigned int asid_bits; - DECLARE_BITMAP(asid_map, ARM_SMMU_MAX_ASIDS); #define ARM_SMMU_MAX_VMIDS (1 << 16) unsigned int vmid_bits; @@ -727,6 +726,8 @@ struct arm_smmu_option_prop { const char *prop; }; +static DEFINE_XARRAY_ALLOC1(asid_xa); + static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -1765,6 +1766,14 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_domain *smmu_domain) cdcfg->cdtab = NULL; } +static void arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) +{ + if (!cd->asid) + return; + + xa_erase(&asid_xa, cd->asid); +} + /* Stream table manipulation functions */ static void arm_smmu_write_strtab_l1_desc(__le64 *dst, struct arm_smmu_strtab_l1_desc *desc) @@ -2450,10 +2459,9 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; - if (cfg->cdcfg.cdtab) { + if (cfg->cdcfg.cdtab) arm_smmu_free_cd_tables(smmu_domain); - arm_smmu_bitmap_free(smmu->asid_map, cfg->cd.asid); - } + arm_smmu_free_asid(&cfg->cd); } else { struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; if (cfg->vmid) @@ -2468,14 +2476,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, struct io_pgtable_cfg *pgtbl_cfg) { int ret; - int asid; + u32 asid; struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; typeof(&pgtbl_cfg->arm_lpae_s1_cfg.tcr) tcr = &pgtbl_cfg->arm_lpae_s1_cfg.tcr; - asid = arm_smmu_bitmap_alloc(smmu->asid_map, smmu->asid_bits); - if (asid < 0) - return asid; + ret = xa_alloc(&asid_xa, &asid, &cfg->cd, + XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); + if (ret) + return ret; cfg->s1cdmax = master->ssid_bits; @@ -2508,7 +2517,7 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, out_free_cd_tables: arm_smmu_free_cd_tables(smmu_domain); out_free_asid: - arm_smmu_bitmap_free(smmu->asid_map, asid); + arm_smmu_free_asid(&cfg->cd); return ret; } From patchwork Tue May 19 17:54:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF18FC433ED for ; Tue, 19 May 2020 18:01:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D60D0207D3 for ; Tue, 19 May 2020 18:01:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="HkelvqOK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728762AbgESSBy (ORCPT ); Tue, 19 May 2020 14:01:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728290AbgESSBx (ORCPT ); Tue, 19 May 2020 14:01:53 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F18CC08C5C0 for ; Tue, 19 May 2020 11:01:53 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id 50so367351wrc.11 for ; Tue, 19 May 2020 11:01:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tdr2nbJDVREjEd6opf3N1PUExNDbBuhp6Y6q/Aa2PEk=; b=HkelvqOK2ASLLh8prk/cDS5hZTdQU9xrr2V3ZxvDu0Sq7ohsTnzBnIMz0gnehYydrd QbNe0r28o6KsNU4c8jG6/OECzBTt1dgbKqKBO3Hy9R096LEbCLBtm5fJCXyPcoLUqC7z nJYdYMckvaByveD0Pss3OR0UJ2rAqvk1aa9NiHwQWgUQzWYyPlH/WdC5rnr0FA7nXTHT wUy4E18JFZwuckNo0n2C9EJ91f2GqeXNIr2Vo1ubIjolcQVPov+bsWVCo6uVm3u+gguw ifDXz+1XuUSPfCuU2aV4GnReJ1qkGO0jRWJcKZ8SusFCa4UK2engyDtkMxBFLorw/oqX 3ZVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tdr2nbJDVREjEd6opf3N1PUExNDbBuhp6Y6q/Aa2PEk=; b=rHmAGDa676Hjkw6E0ESO4ttlc1hoGZtoYvxdTQK4LhcIHiyRbeY+ETdEH0GTOdAC3V 203yaw52X9RgCV7BQe73EehK6ykvtqFFExjXTKq03U8sJZAPf1m1F0+miCE8bVfyVmux 6DwV4xoR/jiXlpMuwZ5Io8oVuIFpODONsCYdaJnoi/1ylrg4Kpp2US29sBL40GC/yQ2B aUzf5x+uPaDUscfj5qx9SoieMhl2gwUqS2Cc2Q21kLVKT9u7WuKKgIk+Br1YYcdPopn6 GpHs73DApHRZp6zkL+VQVzerxFO12qx+G+ZHpkaEhUMRGzVHgQZiMVt1YrMtbjY2UdbO rgvg== X-Gm-Message-State: AOAM530C9++DA6KgvZlK0DtPE/NUw1I4m86Gg7vuwnkgpLychvoHYWOp M+CBEQF0RmCC0QIyqJJy9h2Uug== X-Google-Smtp-Source: ABdhPJynWLyIUGdI5MUqsN8GOzUd3ADLnVQan+95nD7CN0va9fcQVLO9UHUTvmOnyfYBlYSmz6uVWQ== X-Received: by 2002:adf:c508:: with SMTP id q8mr125580wrf.4.1589911312245; Tue, 19 May 2020 11:01:52 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.01.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:01:51 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker , Suzuki K Poulose Subject: [PATCH v7 09/24] arm64: cpufeature: Export symbol read_sanitised_ftr_reg() Date: Tue, 19 May 2020 19:54:47 +0200 Message-Id: <20200519175502.2504091-10-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org The SMMUv3 driver would like to read the MMFR0 PARANGE field in order to share CPU page tables with devices. Allow the driver to be built as module by exporting the read_sanitized_ftr_reg() cpufeature symbol. Acked-by: Suzuki K Poulose Signed-off-by: Jean-Philippe Brucker --- arch/arm64/kernel/cpufeature.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9fac745aa7bb..5f6adbf4ae89 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -841,6 +841,7 @@ u64 read_sanitised_ftr_reg(u32 id) BUG_ON(!regp); return regp->sys_val; } +EXPORT_SYMBOL_GPL(read_sanitised_ftr_reg); #define read_sysreg_case(r) \ case r: return read_sysreg_s(r) From patchwork Tue May 19 17:54:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA791C43446 for ; Tue, 19 May 2020 18:01:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C266B20884 for ; Tue, 19 May 2020 18:01:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="huY/boZ2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728290AbgESSB5 (ORCPT ); Tue, 19 May 2020 14:01:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728898AbgESSB4 (ORCPT ); Tue, 19 May 2020 14:01:56 -0400 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC043C08C5C1 for ; Tue, 19 May 2020 11:01:55 -0700 (PDT) Received: by mail-wr1-x441.google.com with SMTP id s8so379854wrt.9 for ; Tue, 19 May 2020 11:01:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XBKcHNY1n69RPE8RkPPU5z/rlngsZCnS/EuX/PAz4IU=; b=huY/boZ2R0L8m0KTImbAnN89+/2ewwTJYLMEBBCpcua+DalBUsakIScasPz1tAUdut jRLy3r1jNosv70oYwSU9jO/ND777I99T3vTcwtDuvS+z4ts3JBiyyelU4xrccZtfv65M OoGGVHnkE3+NYPIRcGlfJu3caej4l9TJmqKGkN/NstG1NCG63dxqwgHHgFCMfUm4mrj0 z1oeTNy0tARikd0eyqDO4i543e7oQYPFKVv5hf9WWhxCUW3E5Cn6ju0kq0tTnDeE3Ux8 tvYYtC0GsBVoEtefsfPtLyyXRmMJBG1+xMkQ9hQtTSnIhJWUYukG+coDDK6CSirzk1XF Ki2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XBKcHNY1n69RPE8RkPPU5z/rlngsZCnS/EuX/PAz4IU=; b=N5rsQ2pitw2qNdhvx+XNquLO0Or2ZOmtLDkXZN2hbCVwIaar95bWfqdDizwNX+Cdom ekki5ZnZVwJPLzcS3Frp4Gcp7bgZkSUqBP5lGw2iLtDeSacSTQO+ufp9hOmQs4YzFXfs wGutcWV7bIbdVML70c4T8kSmbxXzRrmvjE2puXWxse5hhxgoRC5vor+k1MRxD54MuGeZ BV/h4xHiEpVDATtL3sH7JiM5M8P+tlnx7nEiFLGe4Ui7BsByjNoLNAAHPZZQV+31xJBF 5VJqeIosa3NZZZf/rmYcHQ6wikeu+T75ZEGbBOtBGhtct36Rn5r6uy28vjYN+KgNFvNL FnGw== X-Gm-Message-State: AOAM531K1f0feboPttMzhDp2B9ZTilPiRqf7I5sXfqOMCeo/kCTNX8nN Dv1vCVrPs9aY1fBdOlI15ZhrplDs+mYSeQ== X-Google-Smtp-Source: ABdhPJyaRoZ2ct/6xg1Cpi+87CQAyXoR+7ccakgkN9wxMnUzVPrjQA/FX1kAYbsrPwmNn+nen9KjLQ== X-Received: by 2002:adf:ec88:: with SMTP id z8mr126757wrn.44.1589911314539; Tue, 19 May 2020 11:01:54 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.01.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:01:54 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v7 11/24] iommu/arm-smmu-v3: Seize private ASID Date: Tue, 19 May 2020 19:54:49 +0200 Message-Id: <20200519175502.2504091-12-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org The SMMU has a single ASID space, the union of shared and private ASID sets. This means that the SMMU driver competes with the arch allocator for ASIDs. Shared ASIDs are those of Linux processes, allocated by the arch, and contribute in broadcast TLB maintenance. Private ASIDs are allocated by the SMMU driver and used for "classic" map/unmap DMA. They require command-queue TLB invalidations. When we pin down an mm_context and get an ASID that is already in use by the SMMU, it belongs to a private context. We used to simply abort the bind, but this is unfair to users that would be unable to bind a few seemingly random processes. Try to allocate a new private ASID for the context, and make the old ASID shared. Signed-off-by: Jean-Philippe Brucker --- v6->v7: Replace context_lock spinlock with asid_lock mutex, remove GFP_ATOMIC changes, add comments about locking. --- drivers/iommu/arm-smmu-v3.c | 100 ++++++++++++++++++++++++++++-------- 1 file changed, 80 insertions(+), 20 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 52cbdf08f5e2..403871d36438 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -733,6 +733,7 @@ struct arm_smmu_option_prop { }; static DEFINE_XARRAY_ALLOC1(asid_xa); +static DEFINE_MUTEX(asid_lock); static DEFINE_MUTEX(sva_lock); static struct arm_smmu_option_prop arm_smmu_options[] = { @@ -1537,6 +1538,17 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, } /* Context descriptor manipulation functions */ +static void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) +{ + struct arm_smmu_cmdq_ent cmd = { + .opcode = CMDQ_OP_TLBI_NH_ASID, + .tlbi.asid = asid, + }; + + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); +} + static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, int ssid, bool leaf) { @@ -1795,9 +1807,18 @@ static bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd) return free; } +/* + * Try to reserve this ASID in the SMMU. If it is in use, try to steal it from + * the private entry. Careful here, we may be modifying the context tables of + * another SMMU! + */ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) { + int ret; + u32 new_asid; struct arm_smmu_ctx_desc *cd; + struct arm_smmu_device *smmu; + struct arm_smmu_domain *smmu_domain; cd = xa_load(&asid_xa, asid); if (!cd) @@ -1809,11 +1830,31 @@ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) return cd; } + smmu_domain = container_of(cd, struct arm_smmu_domain, s1_cfg.cd); + smmu = smmu_domain->smmu; + + ret = xa_alloc(&asid_xa, &new_asid, cd, + XA_LIMIT(1, 1 << smmu->asid_bits), GFP_KERNEL); + if (ret) + return ERR_PTR(-ENOSPC); + /* + * Race with unmap: TLB invalidations will start targeting the new ASID, + * which isn't assigned yet. We'll do an invalidate-all on the old ASID + * later, so it doesn't matter. + */ + cd->asid = new_asid; + /* - * Ouch, ASID is already in use for a private cd. - * TODO: seize it. + * Update ASID and invalidate CD in all associated masters. There will + * be some overlap between use of both ASIDs, until we invalidate the + * TLB. */ - return ERR_PTR(-EEXIST); + arm_smmu_write_ctx_desc(smmu_domain, 0, cd); + + /* Invalidate TLB entries previously associated with that context */ + arm_smmu_tlb_inv_asid(smmu, asid); + + return NULL; } __maybe_unused @@ -1839,7 +1880,20 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) arm_smmu_init_cd(cd); + /* + * Serialize against arm_smmu_domain_finalise_s1() and + * arm_smmu_domain_free() as we might need to replace the private ASID + * from an existing CD. + */ + mutex_lock(&asid_lock); old_cd = arm_smmu_share_asid(asid); + if (!old_cd) { + ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL); + if (ret) + old_cd = ERR_PTR(ret); + } + mutex_unlock(&asid_lock); + if (IS_ERR(old_cd)) { ret = PTR_ERR(old_cd); goto err_free_cd; @@ -1853,11 +1907,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) return old_cd; } - /* Fails if a private ASID has been allocated since we last checked */ - ret = xa_insert(&asid_xa, asid, cd, GFP_KERNEL); - if (ret) - goto err_free_cd; - tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - VA_BITS) | FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | @@ -2401,15 +2450,6 @@ static void arm_smmu_tlb_inv_context(void *cookie) struct arm_smmu_device *smmu = smmu_domain->smmu; struct arm_smmu_cmdq_ent cmd; - if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { - cmd.opcode = CMDQ_OP_TLBI_NH_ASID; - cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid; - cmd.tlbi.vmid = 0; - } else { - cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; - cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; - } - /* * NOTE: when io-pgtable is in non-strict mode, we may get here with * PTEs previously cleared by unmaps on the current CPU not yet visible @@ -2417,8 +2457,14 @@ static void arm_smmu_tlb_inv_context(void *cookie) * insertion to guarantee those are observed before the TLBI. Do be * careful, 007. */ - arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); + if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { + arm_smmu_tlb_inv_asid(smmu, smmu_domain->s1_cfg.cd.asid); + } else { + cmd.opcode = CMDQ_OP_TLBI_S12_VMALL; + cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid; + arm_smmu_cmdq_issue_cmd(smmu, &cmd); + arm_smmu_cmdq_issue_sync(smmu); + } arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0); } @@ -2602,9 +2648,15 @@ static void arm_smmu_domain_free(struct iommu_domain *domain) if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) { struct arm_smmu_s1_cfg *cfg = &smmu_domain->s1_cfg; + /* + * Prevent arm_smmu_share_asid() from rewriting CD#0 while we're + * freeing it. + */ + mutex_lock(&asid_lock); if (cfg->cdcfg.cdtab) arm_smmu_free_cd_tables(smmu_domain); arm_smmu_free_asid(&cfg->cd); + mutex_unlock(&asid_lock); } else { struct arm_smmu_s2_cfg *cfg = &smmu_domain->s2_cfg; if (cfg->vmid) @@ -2626,10 +2678,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, arm_smmu_init_cd(&cfg->cd); + /* + * Prevent arm_smmu_share_asid() from seizing the private ASID we're + * allocating here until it is written to the CD. + */ + mutex_lock(&asid_lock); ret = xa_alloc(&asid_xa, &asid, &cfg->cd, XA_LIMIT(1, (1 << smmu->asid_bits) - 1), GFP_KERNEL); if (ret) - return ret; + goto out_unlock; cfg->s1cdmax = master->ssid_bits; @@ -2657,12 +2714,15 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, if (ret) goto out_free_cd_tables; + mutex_unlock(&asid_lock); return 0; out_free_cd_tables: arm_smmu_free_cd_tables(smmu_domain); out_free_asid: arm_smmu_free_asid(&cfg->cd); +out_unlock: + mutex_unlock(&asid_lock); return ret; } From patchwork Tue May 19 17:54:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200348 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E1F8C4345B for ; Tue, 19 May 2020 18:02:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6829F207D3 for ; Tue, 19 May 2020 18:02:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="CVRPa4zD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729166AbgESSCA (ORCPT ); Tue, 19 May 2020 14:02:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728898AbgESSB7 (ORCPT ); Tue, 19 May 2020 14:01:59 -0400 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82854C08C5C0 for ; Tue, 19 May 2020 11:01:58 -0700 (PDT) Received: by mail-wr1-x441.google.com with SMTP id l17so412486wrr.4 for ; Tue, 19 May 2020 11:01:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MElXxFUU4SmUEJg24asfPA/yzXG4FEYdFM842ev0uCE=; b=CVRPa4zD/VPnA7aSmXfjFudzYbQzAkt2fDPLVKyp+6lkKXjtkMxOo9eKi63QXyKhsR idydtFvzsoUMqwomO1K3gnLoko0DRoGb4XjFwFYeNrJ0UnnDf760n4L7Z3iZK2nNjqKc qzEDw7iJt229mzJLqo33oKuUwmUMxFLk7qLzrj/R1ZAUDRNXQUeyYewC12f9znuOrbd1 mESDGceVEL7Eb24fn6dF4WJYPiJtmj8zMhlIHnQk4EyANomkIVzoEIi95nuy6wyFyDyV 2dRxdWZIZL5mseHHVRWCg9eco6IoWkDjQ/BDlsAwgvFkwfRZONYUigvNvwFVmhXBmXIl Ko3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MElXxFUU4SmUEJg24asfPA/yzXG4FEYdFM842ev0uCE=; b=EWrtvK83V2CHHDTf6oDSP7vG6kqRuZ8zn+2NzL3nfIC9ljDhWzEfoiN0Bh7DbpPP2k 6KYZMide40Erwxu2l18k7aNXaNpBqTwirBrTnJUrTd8WEBih32ViSaXhKX7E4UjcnE1w QXnyxxIhOpeJpux+YHwAPCtO1KFkXoXILzJZHdgI0UHsGu+5qbb6ZsJ7USLis6TXRSIi 8dv9EtOnM1aikzm9wsDQFBZQEv7oH8gVytRcs5EDweqHRIbHiM2XZw3FL0DQeG/NbcXL cBSD1OzpfhFR2Ns2jcW+nE/KaUxKl4daO+TQX6etBPDjGWvzk8IaRnMqmVT0SWF0wFjm 4Hjw== X-Gm-Message-State: AOAM533T+cbb4xFIS29FWBXOK/rG0J00K3Io0JJZQU12QCEi1PvZJJHs lvp+Av6VP1hcfWRrpl6ik/VmRA== X-Google-Smtp-Source: ABdhPJyUX7XmnpA8DeuBDVrI8G0XF2iDcX0m67gXquuRRNHmvZchB0Xi1muHbYRZPMsVS4WxVUNdhw== X-Received: by 2002:adf:a1d7:: with SMTP id v23mr90184wrv.155.1589911317146; Tue, 19 May 2020 11:01:57 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.01.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:01:56 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v7 13/24] iommu/arm-smmu-v3: Enable broadcast TLB maintenance Date: Tue, 19 May 2020 19:54:51 +0200 Message-Id: <20200519175502.2504091-14-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org The SMMUv3 can handle invalidation targeted at TLB entries with shared ASIDs. If the implementation supports broadcast TLB maintenance, enable it and keep track of it in a feature bit. The SMMU will then be affected by inner-shareable TLB invalidations from other agents. A major side-effect of this change is that stage-2 translation contexts are now affected by all invalidations by VMID. VMIDs are all shared and the only ways to prevent over-invalidation, since the stage-2 page tables are not shared between CPU and SMMU, are to either disable BTM or allocate different VMIDs. This patch does not address the problem. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 7e1933e7e35f..9332253e3608 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -56,6 +56,7 @@ #define IDR0_ASID16 (1 << 12) #define IDR0_ATS (1 << 10) #define IDR0_HYP (1 << 9) +#define IDR0_BTM (1 << 5) #define IDR0_COHACC (1 << 4) #define IDR0_TTF GENMASK(3, 2) #define IDR0_TTF_AARCH64 2 @@ -658,6 +659,7 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_VAX (1 << 14) #define ARM_SMMU_FEAT_RANGE_INV (1 << 15) #define ARM_SMMU_FEAT_E2H (1 << 16) +#define ARM_SMMU_FEAT_BTM (1 << 17) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0) @@ -3819,11 +3821,14 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass) writel_relaxed(reg, smmu->base + ARM_SMMU_CR1); /* CR2 (random crap) */ - reg = CR2_PTM | CR2_RECINVSID; + reg = CR2_RECINVSID; if (smmu->features & ARM_SMMU_FEAT_E2H) reg |= CR2_E2H; + if (!(smmu->features & ARM_SMMU_FEAT_BTM)) + reg |= CR2_PTM; + writel_relaxed(reg, smmu->base + ARM_SMMU_CR2); /* Stream table */ @@ -3934,6 +3939,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) { u32 reg; bool coherent = smmu->features & ARM_SMMU_FEAT_COHERENCY; + bool vhe = cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN); /* IDR0 */ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR0); @@ -3983,10 +3989,19 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) if (reg & IDR0_HYP) { smmu->features |= ARM_SMMU_FEAT_HYP; - if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) + if (vhe) smmu->features |= ARM_SMMU_FEAT_E2H; } + /* + * If the CPU is using VHE, but the SMMU doesn't support it, the SMMU + * will create TLB entries for NH-EL1 world and will miss the + * broadcasted TLB invalidations that target EL2-E2H world. Don't enable + * BTM in that case. + */ + if (reg & IDR0_BTM && (!vhe || reg & IDR0_HYP)) + smmu->features |= ARM_SMMU_FEAT_BTM; + /* * The coherency feature as set by FW is used in preference to the ID * register, but warn on mismatch. From patchwork Tue May 19 17:54:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A88AC4345D for ; Tue, 19 May 2020 18:02:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D6F7207D3 for ; Tue, 19 May 2020 18:02:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="Y4EdnjwY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729210AbgESSCB (ORCPT ); Tue, 19 May 2020 14:02:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728898AbgESSCB (ORCPT ); Tue, 19 May 2020 14:02:01 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAF55C08C5C1 for ; Tue, 19 May 2020 11:02:00 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id n18so196236wmj.5 for ; Tue, 19 May 2020 11:02:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=w0kdAS+3V4p7jXN1k79MYNGRv9Y+4JMnAcNVoiF0l70=; b=Y4EdnjwYGiovh5kRncy2PI1+5V23I9EidrfOm90dz7a75obSThtBtY4iz/6hLBVCho 5gutgO9+jYJaqa3AP50b579CyD3Xo8Os+g5ZMtz+rvzj3KIv4+JVix3pPmzIdWGBJpUr mRdFDzxwXg8P6rMBwMIB7hmJCUHwiDSjb9udlALkBO4crYuf5bUdIRrst4t2//W5+BIr qDjRWz0EZjqj5ojI8fljkyY6ePbeArnslKBHUJdWTLeNiK8/mjqfUvalHtpMS9aAXXYY 1FNHNPtwsiFRQScSD/UobANFMpQMISm2uWThO2evgtfehAQl/p6SzjFc3OPNw3sSLGHi TQ7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=w0kdAS+3V4p7jXN1k79MYNGRv9Y+4JMnAcNVoiF0l70=; b=IuKuvf6osJS/D1etlNRtQ1K5Gq5BpBJg4YZJH2W2jizoQ6G0fPeSs4SV6/JRvDb6nu tBxLs7ivf7FyIU2GAjdhy+GN5YT70PF5zy8IpP1UPMXxAoD8U86uGEJSbPbjiy5FPj8n GtBl7Ffls2b3gvu1SJkzjT34+g1nvt4cd9l4ABemQTSkyb2bGZOrnucn46FDpOPYEqfC 5FhHl9titezqkl70Zj7ZLyJDgtmw3SXJfSNL8QcT2L3H7+98/3gBA4n45HyTPI9umaGs bEhLaLmzK1IH1pQ/qM9S/+tdU3oC9QMMll11F/4qrQonBVe7wSqGM3lsuIJI2jd6mDnB 3Fxg== X-Gm-Message-State: AOAM5312/olWT7wJITe+qDqNK30vm8LLg3HC8W1FJp75mSrY9vY2hdGm Kq0ewpBOUtn+A2w+MTPwqgs+CQ== X-Google-Smtp-Source: ABdhPJxiur973rQO7P0is4n9bTcqUoLADPVJgaczJU1ETHTcZpVacHBincB+tnOWfVKyZnCnndXCAg== X-Received: by 2002:a7b:c3da:: with SMTP id t26mr575120wmj.146.1589911319392; Tue, 19 May 2020 11:01:59 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.01.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:01:58 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v7 15/24] iommu/arm-smmu-v3: Add SVA device feature Date: Tue, 19 May 2020 19:54:53 +0200 Message-Id: <20200519175502.2504091-16-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Implement the IOMMU device feature callbacks to support the SVA feature. At the moment dev_has_feat() returns false since I/O Page Faults isn't yet implemented. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 124 ++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index a9f6f1d7014e..b016b61cee23 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -703,6 +703,8 @@ struct arm_smmu_master { u32 *sids; unsigned int num_sids; bool ats_enabled; + bool sva_enabled; + struct list_head bonds; unsigned int ssid_bits; }; @@ -3013,6 +3015,19 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) master = dev_iommu_priv_get(dev); smmu = master->smmu; + /* + * Checking that SVA is disabled ensures that this device isn't bound to + * any mm, and can be safely detached from its old domain. Bonds cannot + * be removed concurrently since we're holding the group mutex. + */ + mutex_lock(&sva_lock); + if (master->sva_enabled) { + mutex_unlock(&sva_lock); + dev_err(dev, "cannot attach - SVA enabled\n"); + return -EBUSY; + } + mutex_unlock(&sva_lock); + arm_smmu_detach_dev(master); mutex_lock(&smmu_domain->init_mutex); @@ -3161,6 +3176,7 @@ static int arm_smmu_add_device(struct device *dev) master->smmu = smmu; master->sids = fwspec->ids; master->num_sids = fwspec->num_ids; + INIT_LIST_HEAD(&master->bonds); dev_iommu_priv_set(dev, master); /* Check the SIDs are in range of the SMMU and our stream table */ @@ -3230,6 +3246,7 @@ static void arm_smmu_remove_device(struct device *dev) master = dev_iommu_priv_get(dev); smmu = master->smmu; + WARN_ON(master->sva_enabled); arm_smmu_detach_dev(master); iommu_group_remove_device(dev); iommu_device_unlink(&smmu->iommu, dev); @@ -3349,6 +3366,109 @@ static void arm_smmu_get_resv_regions(struct device *dev, iommu_dma_get_resv_regions(dev, head); } +static bool arm_smmu_iopf_supported(struct arm_smmu_master *master) +{ + return false; +} + +static bool arm_smmu_dev_has_feature(struct device *dev, + enum iommu_dev_features feat) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + if (!master) + return false; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + if (!(master->smmu->features & ARM_SMMU_FEAT_SVA)) + return false; + + /* SSID and IOPF support are mandatory for the moment */ + return master->ssid_bits && arm_smmu_iopf_supported(master); + default: + return false; + } +} + +static bool arm_smmu_dev_feature_enabled(struct device *dev, + enum iommu_dev_features feat) +{ + bool enabled = false; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + if (!master) + return false; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + mutex_lock(&sva_lock); + enabled = master->sva_enabled; + mutex_unlock(&sva_lock); + return enabled; + default: + return false; + } +} + +static int arm_smmu_dev_enable_sva(struct device *dev) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + mutex_lock(&sva_lock); + master->sva_enabled = true; + mutex_unlock(&sva_lock); + + return 0; +} + +static int arm_smmu_dev_disable_sva(struct device *dev) +{ + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + + mutex_lock(&sva_lock); + if (!list_empty(&master->bonds)) { + dev_err(dev, "cannot disable SVA, device is bound\n"); + mutex_unlock(&sva_lock); + return -EBUSY; + } + master->sva_enabled = false; + mutex_unlock(&sva_lock); + + return 0; +} + +static int arm_smmu_dev_enable_feature(struct device *dev, + enum iommu_dev_features feat) +{ + if (!arm_smmu_dev_has_feature(dev, feat)) + return -ENODEV; + + if (arm_smmu_dev_feature_enabled(dev, feat)) + return -EBUSY; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + return arm_smmu_dev_enable_sva(dev); + default: + return -EINVAL; + } +} + +static int arm_smmu_dev_disable_feature(struct device *dev, + enum iommu_dev_features feat) +{ + if (!arm_smmu_dev_feature_enabled(dev, feat)) + return -EINVAL; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + return arm_smmu_dev_disable_sva(dev); + default: + return -EINVAL; + } +} + static struct iommu_ops arm_smmu_ops = { .capable = arm_smmu_capable, .domain_alloc = arm_smmu_domain_alloc, @@ -3367,6 +3487,10 @@ static struct iommu_ops arm_smmu_ops = { .of_xlate = arm_smmu_of_xlate, .get_resv_regions = arm_smmu_get_resv_regions, .put_resv_regions = generic_iommu_put_resv_regions, + .dev_has_feat = arm_smmu_dev_has_feature, + .dev_feat_enabled = arm_smmu_dev_feature_enabled, + .dev_enable_feat = arm_smmu_dev_enable_feature, + .dev_disable_feat = arm_smmu_dev_disable_feature, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; From patchwork Tue May 19 17:54:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47568C43460 for ; Tue, 19 May 2020 18:02:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 26586207D3 for ; Tue, 19 May 2020 18:02:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="afLtAp2i" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729233AbgESSCC (ORCPT ); Tue, 19 May 2020 14:02:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728898AbgESSCC (ORCPT ); Tue, 19 May 2020 14:02:02 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5FD0C08C5C0 for ; Tue, 19 May 2020 11:02:01 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id l18so405600wrn.6 for ; Tue, 19 May 2020 11:02:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4JRCRvDwKBaj6zFWwAyf94DPzbFyDqhKWQ8Qel74T6k=; b=afLtAp2iGh0TSBKhOlXEtgRXoAAe/HVKmjTsSeDv9UxjhRCMaVKuinCcCKSstdsZUZ kdMWspzZ2QsL5Dau2h4mskzs/qU/JpXEbIEasYD0S1PcdPIAajqvdZryUJYwyh97byo3 I9Wwh3Ym7E5J8i7hGztD4+iEGuD60ykADGTl3eKWMyiY2HIf6xDoG7OcUZp981d6hEq6 0TIm+yZWWnnOBVgyRxiN34vur8pq+aRPJrG5M24Zz24eEH8j19WveCVy5CYw4fv399WU TGCaw9JbKY4KIq9k1nnPX6+3+/4+X2jtr691bmfVQXZQwcKQTCWCY+xoNAk8RN14s9Ip D1oQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4JRCRvDwKBaj6zFWwAyf94DPzbFyDqhKWQ8Qel74T6k=; b=ckulLw3EWmehRlTlretzPNmJVzL8k8DpeCOjxrtb5IUuuUoU76tsEAawJBfvVdeAOf +b+6AxPm3C8sKHnPSlUZ23m0yrfVlsxLM2dBqITGq3Tw0uUZ3vmGlO/zhDDneVqI4KDG NGV2xz/EGysXCH6Fuk7L/o876JdrgBn2xyFh0PeBN/HZ5Uhl5wvESU766x6uiLd2I11f zTJM0yfsvOn95KvSwdmkhtzAiFwJz9oKDI3R0Q4CWG+BzIixFyTeFDJ08OKsx/HxnAyF x4SeciI/sPIPioKC8WzH3xPa+39EU0AGDqaoxzmAdNowbf2EugxKYx5ZTlMx/7lngZPW 8ZSg== X-Gm-Message-State: AOAM531zMShz1fiIote3BDl+KRiapYxQF6MUdll83XSSfc/auxkqjnT8 zTcpWU5yDigY7+2RAO0w20NjzA== X-Google-Smtp-Source: ABdhPJx8Z3P30TLSvOFRBCkey5bHCv2HIx/ihXvi35CP5z2ZNK51cJrX3rrbYnMtuijPpPM1UkAWBQ== X-Received: by 2002:adf:ffc2:: with SMTP id x2mr76717wrs.273.1589911320419; Tue, 19 May 2020 11:02:00 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.01.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:01:59 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v7 16/24] iommu/arm-smmu-v3: Implement iommu_sva_bind/unbind() Date: Tue, 19 May 2020 19:54:54 +0200 Message-Id: <20200519175502.2504091-17-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org The sva_bind() function allows devices to access process address spaces using a PASID (aka SSID). (1) bind() allocates or gets an existing MMU notifier tied to the (domain, mm) pair. Each mm gets one PASID. (2) Any change to the address space calls invalidate_range() which sends ATC invalidations (in a subsequent patch). (3) When the process address space dies, the release() notifier disables the CD to allow reclaiming the page tables. Since release() has to be light we do not instruct device drivers to stop DMA here, we just ignore incoming page faults from this point onwards. To avoid any event 0x0a print (C_BAD_CD) we disable translation without clearing CD.V. PCIe Translation Requests and Page Requests are silently denied. Don't clear the R bit because the S bit can't be cleared when STALL_MODEL==0b10 (forced), and clearing R without clearing S is useless. Faulting transactions will stall and will be aborted by the IOPF handler. (4) After stopping DMA, the device driver releases the bond by calling unbind(). We release the MMU notifier, free the PASID and the bond. Three structures keep track of bonds: * arm_smmu_bond: one per {device, mm} pair, the handle returned to the device driver for a bind() request. * arm_smmu_mmu_notifier: one per {domain, mm} pair, deals with ATS/TLB invalidations and clearing the context descriptor on mm exit. * arm_smmu_ctx_desc: one per mm, holds the pinned ASID and pgd. Signed-off-by: Jean-Philippe Brucker --- v6->v7: Keep track of {domains, mm} pairs. Move mmu_notifier_synchronize() to module_exit(). --- drivers/iommu/Kconfig | 2 + drivers/iommu/arm-smmu-v3.c | 272 +++++++++++++++++++++++++++++++++++- 2 files changed, 269 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 15e9dc4e503c..00b517f449ab 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -432,8 +432,10 @@ config ARM_SMMU_V3 tristate "ARM Ltd. System MMU Version 3 (SMMUv3) Support" depends on ARM64 select IOMMU_API + select IOMMU_SVA select IOMMU_IO_PGTABLE_LPAE select GENERIC_MSI_IRQ_DOMAIN + select MMU_NOTIFIER help Support for implementations of the ARM System MMU architecture version 3 providing translation support to a PCIe root complex. diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index b016b61cee23..00a9342eed99 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -36,6 +37,7 @@ #include #include "io-pgtable-arm.h" +#include "iommu-sva.h" /* MMIO registers */ #define ARM_SMMU_IDR0 0x0 @@ -734,8 +736,32 @@ struct arm_smmu_domain { struct list_head devices; spinlock_t devices_lock; + + struct list_head mmu_notifiers; +}; + +struct arm_smmu_mmu_notifier { + struct mmu_notifier mn; + struct arm_smmu_ctx_desc *cd; + bool cleared; + refcount_t refs; + struct list_head list; + struct arm_smmu_domain *domain; }; +#define mn_to_smmu(mn) container_of(mn, struct arm_smmu_mmu_notifier, mn) + +struct arm_smmu_bond { + struct iommu_sva sva; + struct mm_struct *mm; + struct arm_smmu_mmu_notifier *smmu_mn; + struct list_head list; + refcount_t refs; +}; + +#define sva_to_bond(handle) \ + container_of(handle, struct arm_smmu_bond, sva) + struct arm_smmu_option_prop { u32 opt; const char *prop; @@ -745,6 +771,13 @@ static DEFINE_XARRAY_ALLOC1(asid_xa); static DEFINE_MUTEX(asid_lock); static DEFINE_MUTEX(sva_lock); +/* + * When a process dies, DMA is still running but we need to clear the pgd. If we + * simply cleared the valid bit from the context descriptor, we'd get event 0x0a + * which are not recoverable. + */ +static struct arm_smmu_ctx_desc invalid_cd = { 0 }; + static struct arm_smmu_option_prop arm_smmu_options[] = { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -1654,7 +1687,9 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, * (2) Install a secondary CD, for SID+SSID traffic. * (3) Update ASID of a CD. Atomically write the first 64 bits of the * CD, then invalidate the old entry and mappings. - * (4) Remove a secondary CD. + * (4) Quiesce the context without clearing the valid bit. Disable + * translation, and ignore any translation fault. + * (5) Remove a secondary CD. */ u64 val; bool cd_live; @@ -1671,8 +1706,10 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, val = le64_to_cpu(cdptr[0]); cd_live = !!(val & CTXDESC_CD_0_V); - if (!cd) { /* (4) */ + if (!cd) { /* (5) */ val = 0; + } else if (cd == &invalid_cd) { /* (4) */ + val |= CTXDESC_CD_0_TCR_EPD0; } else if (cd_live) { /* (3) */ val &= ~CTXDESC_CD_0_ASID; val |= FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); @@ -1872,7 +1909,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_share_asid(u16 asid) return NULL; } -__maybe_unused static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) { u16 asid; @@ -1969,7 +2005,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) return ERR_PTR(ret); } -__maybe_unused static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) { lockdep_assert_held(&sva_lock); @@ -2606,6 +2641,8 @@ static bool arm_smmu_capable(enum iommu_cap cap) } } +static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops; + static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) { struct arm_smmu_domain *smmu_domain; @@ -2633,6 +2670,7 @@ static struct iommu_domain *arm_smmu_domain_alloc(unsigned type) mutex_init(&smmu_domain->init_mutex); INIT_LIST_HEAD(&smmu_domain->devices); spin_lock_init(&smmu_domain->devices_lock); + INIT_LIST_HEAD(&smmu_domain->mmu_notifiers); return &smmu_domain->domain; } @@ -3127,6 +3165,216 @@ arm_smmu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova) return ops->iova_to_phys(ops, iova); } +static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, unsigned long end) +{ + /* TODO: invalidate ATS */ +} + +static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) +{ + struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn); + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + mutex_lock(&sva_lock); + if (smmu_mn->cleared) { + mutex_unlock(&sva_lock); + return; + } + + /* + * DMA may still be running. Keep the cd valid to avoid C_BAD_CD events, + * but disable translation. + */ + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, &invalid_cd); + + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); + /* TODO: invalidate ATS */ + + smmu_mn->cleared = true; + mutex_unlock(&sva_lock); +} + +static void arm_smmu_mmu_notifier_free(struct mmu_notifier *mn) +{ + kfree(mn_to_smmu(mn)); +} + +static struct mmu_notifier_ops arm_smmu_mmu_notifier_ops = { + .invalidate_range = arm_smmu_mm_invalidate_range, + .release = arm_smmu_mm_release, + .free_notifier = arm_smmu_mmu_notifier_free, +}; + +/* Allocate or get existing MMU notifier for this {domain, mm} pair */ +static struct arm_smmu_mmu_notifier * +arm_smmu_mmu_notifier_get(struct arm_smmu_domain *smmu_domain, + struct mm_struct *mm) +{ + int ret; + struct arm_smmu_ctx_desc *cd; + struct arm_smmu_mmu_notifier *smmu_mn; + + lockdep_assert_held(&sva_lock); + + list_for_each_entry(smmu_mn, &smmu_domain->mmu_notifiers, list) { + if (smmu_mn->mn.mm == mm) { + refcount_inc(&smmu_mn->refs); + return smmu_mn; + } + } + + cd = arm_smmu_alloc_shared_cd(mm); + if (IS_ERR(cd)) + return ERR_CAST(cd); + + smmu_mn = kzalloc(sizeof(*smmu_mn), GFP_KERNEL); + if (!smmu_mn) { + ret = -ENOMEM; + goto err_free_cd; + } + + refcount_set(&smmu_mn->refs, 1); + smmu_mn->cd = cd; + smmu_mn->domain = smmu_domain; + smmu_mn->mn.ops = &arm_smmu_mmu_notifier_ops; + + ret = mmu_notifier_register(&smmu_mn->mn, mm); + if (ret) { + kfree(smmu_mn); + goto err_free_cd; + } + + ret = arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, cd); + if (ret) + goto err_put_notifier; + + list_add(&smmu_mn->list, &smmu_domain->mmu_notifiers); + return smmu_mn; + +err_put_notifier: + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); +err_free_cd: + arm_smmu_free_shared_cd(cd); + return ERR_PTR(ret); +} + +static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) +{ + struct mm_struct *mm = smmu_mn->mn.mm; + struct arm_smmu_ctx_desc *cd = smmu_mn->cd; + struct arm_smmu_domain *smmu_domain = smmu_mn->domain; + + lockdep_assert_held(&sva_lock); + + if (!refcount_dec_and_test(&smmu_mn->refs)) + return; + + list_del(&smmu_mn->list); + arm_smmu_write_ctx_desc(smmu_domain, mm->pasid, NULL); + + /* + * If we went through clear(), we've already invalidated, and no + * new TLB entry can have been formed. + */ + if (!smmu_mn->cleared) { + arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + /* TODO: invalidate ATS */ + } + + /* Frees smmu_mn */ + mmu_notifier_put(&smmu_mn->mn); + arm_smmu_free_shared_cd(cd); +} + +static struct iommu_sva * +__arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm) +{ + int ret; + struct arm_smmu_bond *bond; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + lockdep_assert_held(&sva_lock); + + if (!master || !master->sva_enabled) + return ERR_PTR(-ENODEV); + + /* If bind() was already called for this {dev, mm} pair, reuse it. */ + list_for_each_entry(bond, &master->bonds, list) { + if (bond->mm == mm) { + refcount_inc(&bond->refs); + return &bond->sva; + } + } + + bond = kzalloc(sizeof(*bond), GFP_KERNEL); + if (!bond) + return ERR_PTR(-ENOMEM); + + /* Allocate a PASID for this mm if necessary */ + ret = iommu_sva_alloc_pasid(mm, 1, (1U << master->ssid_bits) - 1); + if (ret) + goto err_free_bond; + + bond->mm = mm; + bond->sva.dev = dev; + refcount_set(&bond->refs, 1); + + bond->smmu_mn = arm_smmu_mmu_notifier_get(smmu_domain, mm); + if (IS_ERR(bond->smmu_mn)) + goto err_free_pasid; + + list_add(&bond->list, &master->bonds); + return &bond->sva; + +err_free_pasid: + iommu_sva_free_pasid(mm); +err_free_bond: + kfree(bond); + return ERR_PTR(ret); +} + +static struct iommu_sva * +arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdata) +{ + struct iommu_sva *handle; + struct iommu_domain *domain = iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain); + + if (smmu_domain->stage != ARM_SMMU_DOMAIN_S1) + return ERR_PTR(-EINVAL); + + mutex_lock(&sva_lock); + handle = __arm_smmu_sva_bind(dev, mm); + mutex_unlock(&sva_lock); + return handle; +} + +static void arm_smmu_sva_unbind(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + mutex_lock(&sva_lock); + if (refcount_dec_and_test(&bond->refs)) { + list_del(&bond->list); + arm_smmu_mmu_notifier_put(bond->smmu_mn); + iommu_sva_free_pasid(bond->mm); + kfree(bond); + } + mutex_unlock(&sva_lock); +} + +static int arm_smmu_sva_get_pasid(struct iommu_sva *handle) +{ + struct arm_smmu_bond *bond = sva_to_bond(handle); + + return bond->mm->pasid; +} + static struct platform_driver arm_smmu_driver; static @@ -3491,6 +3739,9 @@ static struct iommu_ops arm_smmu_ops = { .dev_feat_enabled = arm_smmu_dev_feature_enabled, .dev_enable_feat = arm_smmu_dev_enable_feature, .dev_disable_feat = arm_smmu_dev_disable_feature, + .sva_bind = arm_smmu_sva_bind, + .sva_unbind = arm_smmu_sva_unbind, + .sva_get_pasid = arm_smmu_sva_get_pasid, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; @@ -4598,6 +4849,16 @@ static const struct of_device_id arm_smmu_of_match[] = { }; MODULE_DEVICE_TABLE(of, arm_smmu_of_match); +static void arm_smmu_driver_unregister(struct platform_driver *drv) +{ + /* + * Wait for all notifiers free() RCU callbacks, since they are still + * using the arm_smmu_mmu_notifier_ops. + */ + mmu_notifier_synchronize(); + platform_driver_unregister(drv); +} + static struct platform_driver arm_smmu_driver = { .driver = { .name = "arm-smmu-v3", @@ -4608,7 +4869,8 @@ static struct platform_driver arm_smmu_driver = { .remove = arm_smmu_device_remove, .shutdown = arm_smmu_device_shutdown, }; -module_platform_driver(arm_smmu_driver); +module_driver(arm_smmu_driver, platform_driver_register, + arm_smmu_driver_unregister); MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations"); MODULE_AUTHOR("Will Deacon "); From patchwork Tue May 19 17:54:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77D6FC43466 for ; Tue, 19 May 2020 18:02:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 61B8A20823 for ; Tue, 19 May 2020 18:02:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="K/TabPRr" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729288AbgESSCF (ORCPT ); Tue, 19 May 2020 14:02:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729267AbgESSCF (ORCPT ); Tue, 19 May 2020 14:02:05 -0400 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9750C08C5C2 for ; Tue, 19 May 2020 11:02:03 -0700 (PDT) Received: by mail-wr1-x441.google.com with SMTP id l11so449013wru.0 for ; Tue, 19 May 2020 11:02:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fg8dmjPt1ph2YrEH+2TlM2jE9Xe96yZybVcxaHnAkA0=; b=K/TabPRrtSWRAQ8LXcijRuJOve7Tq3nGl34UfEWB0FFTlEq6ZM5hQ39eTft3KR8VFX tYj4YAy/8VDecjXA3Nm7axX05bTsbZ0mfsWekrw+XvU7lcpYO5YWjd46wmytl8WW6a+k Z//gWmucZD2sLlF6urGmcRdUkCoMDQ/cMlx4drtCEFQ/FhB++ucmI3zsJWWXu1WORVq6 fSYvtpoQVvNqEKXGhlCwMf9sPLGWN1SdT+y8alAtOYcDh9cMEfbKbOVHb4iMaL35u4Vj HU1EqI7zhjzvIkF85VQI5/VV+8/e5baagldL+gSbhByLf6w1ZC6mcq4w3m7E/HM07fC6 fjwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fg8dmjPt1ph2YrEH+2TlM2jE9Xe96yZybVcxaHnAkA0=; b=PluhrqftYy/059i4iMhEQd0Eh1qpWkXsZpBob/18dc+EyKQVRQtCFeRdhjG6WHBLVf m33x7lpJOGXT4z6kuX7iQDjISERjNNo6gXdfZM/OOqdcHzU89Pev7b0GqEnS6yJu5XeN ycBnk+CeXK2nfso3pd8MV65x25cYmKFYXy6tC7Mq5x3wpX9Nzns8uZXnqfpZNLx3ADJ2 UyM3zGFet2l8dPnvKcgUik3pXwNeo8yONjzUpz8cg+sjPpBidsq+DVmbxOezcrBi82kx y7BjB6lhg61ksu8/mT0D51FQbAYqF7KP+ixOnriaJz22Dgj23s3gvNIegGJjOCJ+/qnc 7/0Q== X-Gm-Message-State: AOAM532TvPrDMaHzHK4U1G82FRv/+AHF8rEuKwQdyah0/DhxhwQLXz// cwTjuBpWVHPS2/s85TRH+GcVIg== X-Google-Smtp-Source: ABdhPJyzkXmiYxNm30l0xsYDmklGny2bG167UWZuisgXBYPv8wqLmD/PLaUZ75RNeRP8Sz1G3zk17w== X-Received: by 2002:a5d:6388:: with SMTP id p8mr69668wru.369.1589911322694; Tue, 19 May 2020 11:02:02 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.02.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:02:02 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker Subject: [PATCH v7 18/24] iommu/arm-smmu-v3: Add support for Hardware Translation Table Update Date: Tue, 19 May 2020 19:54:56 +0200 Message-Id: <20200519175502.2504091-19-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org If the SMMU supports it and the kernel was built with HTTU support, enable hardware update of access and dirty flags. This is essential for shared page tables, to reduce the number of access faults on the fault queue. Normal DMA with io-pgtables doesn't currently use the access or dirty flags. We can enable HTTU even if CPUs don't support it, because the kernel always checks for HW dirty bit and updates the PTE flags atomically. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 1386d4d2bc60..6a368218f54c 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -58,6 +58,8 @@ #define IDR0_ASID16 (1 << 12) #define IDR0_ATS (1 << 10) #define IDR0_HYP (1 << 9) +#define IDR0_HD (1 << 7) +#define IDR0_HA (1 << 6) #define IDR0_BTM (1 << 5) #define IDR0_COHACC (1 << 4) #define IDR0_TTF GENMASK(3, 2) @@ -311,6 +313,9 @@ #define CTXDESC_CD_0_TCR_IPS GENMASK_ULL(34, 32) #define CTXDESC_CD_0_TCR_TBI0 (1ULL << 38) +#define CTXDESC_CD_0_TCR_HA (1UL << 43) +#define CTXDESC_CD_0_TCR_HD (1UL << 42) + #define CTXDESC_CD_0_AA64 (1UL << 41) #define CTXDESC_CD_0_S (1UL << 44) #define CTXDESC_CD_0_R (1UL << 45) @@ -663,6 +668,8 @@ struct arm_smmu_device { #define ARM_SMMU_FEAT_E2H (1 << 16) #define ARM_SMMU_FEAT_BTM (1 << 17) #define ARM_SMMU_FEAT_SVA (1 << 18) +#define ARM_SMMU_FEAT_HA (1 << 19) +#define ARM_SMMU_FEAT_HD (1 << 20) u32 features; #define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0) @@ -1718,10 +1725,17 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, * this substream's traffic */ } else { /* (1) and (2) */ + u64 tcr = cd->tcr; + cdptr[1] = cpu_to_le64(cd->ttbr & CTXDESC_CD_1_TTB0_MASK); cdptr[2] = 0; cdptr[3] = cpu_to_le64(cd->mair); + if (!(smmu->features & ARM_SMMU_FEAT_HD)) + tcr &= ~CTXDESC_CD_0_TCR_HD; + if (!(smmu->features & ARM_SMMU_FEAT_HA)) + tcr &= ~CTXDESC_CD_0_TCR_HA; + /* * STE is live, and the SMMU might read dwords of this CD in any * order. Ensure that it observes valid values before reading @@ -1729,7 +1743,7 @@ static int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, */ arm_smmu_sync_cd(smmu_domain, ssid, true); - val = cd->tcr | + val = tcr | #ifdef __BIG_ENDIAN CTXDESC_CD_0_ENDI | #endif @@ -1958,10 +1972,12 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_struct *mm) return old_cd; } + /* HA and HD will be filtered out later if not supported by the SMMU */ tcr = FIELD_PREP(CTXDESC_CD_0_TCR_T0SZ, 64ULL - VA_BITS) | FIELD_PREP(CTXDESC_CD_0_TCR_IRGN0, ARM_LPAE_TCR_RGN_WBWA) | FIELD_PREP(CTXDESC_CD_0_TCR_ORGN0, ARM_LPAE_TCR_RGN_WBWA) | FIELD_PREP(CTXDESC_CD_0_TCR_SH0, ARM_LPAE_TCR_SH_IS) | + CTXDESC_CD_0_TCR_HA | CTXDESC_CD_0_TCR_HD | CTXDESC_CD_0_TCR_EPD1 | CTXDESC_CD_0_AA64; switch (PAGE_SIZE) { @@ -4454,6 +4470,12 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) smmu->features |= ARM_SMMU_FEAT_E2H; } + if (reg & (IDR0_HA | IDR0_HD)) { + smmu->features |= ARM_SMMU_FEAT_HA; + if (reg & IDR0_HD) + smmu->features |= ARM_SMMU_FEAT_HD; + } + /* * If the CPU is using VHE, but the SMMU doesn't support it, the SMMU * will create TLB entries for NH-EL1 world and will miss the From patchwork Tue May 19 17:55:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28B82C43470 for ; Tue, 19 May 2020 18:02:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0763020823 for ; Tue, 19 May 2020 18:02:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="T73/hci8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729312AbgESSCK (ORCPT ); Tue, 19 May 2020 14:02:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33484 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729345AbgESSCK (ORCPT ); Tue, 19 May 2020 14:02:10 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A376FC08C5C0 for ; Tue, 19 May 2020 11:02:08 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id n18so196757wmj.5 for ; Tue, 19 May 2020 11:02:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ItT+TOPL7iHYDvWhuBPZ37dZIwufOCW7iBxWcHWr4y4=; b=T73/hci8m0xaQEonuG+UITtUtGVzOJkMhIY/g1SohiFjmbfVK6w59f+mvqGudusgqn KgSK+DNf5H/EQw1G/d66b7N0XokxEarYo8OFVIa3yO4z8ZUf4rBHRzAX2fY5L09vHu1Y QQzBlmR3/6FRKXAEejV9+0bzWiW/ndlxSsX7EpdL73Qs3tIfuU7JiQ1JABcUgfVkg84v /5x6oY+5isr6RrxYXgLgn/rKf2OVXKo+2aaq/oVwR31vmGqGarJEDUl11onJW/3JsSu6 L760pJcDstfBWOr3+KHNzLdSHCFEX02NpH3IIwq+RgLPlc9O62wk3XBFgwtfUBP2m/Pl pXnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ItT+TOPL7iHYDvWhuBPZ37dZIwufOCW7iBxWcHWr4y4=; b=QYDOp8/otgS2toeHR582MtW595Mn2FDJESRZg8wKpiK31tNbygK7l4OL3lQdldbZqK 7k4m2JKlPN5ktvsvo9pgx8uq+zu+pwTnZYKVaGYMUIBzGCnztskFOY8UAUT3dcf8+g7p HZTNBuWZEdqa7aZimmNVKAQR1alxQ9N/mw+udEvG60OD8mLrTqXWLr558p3pS3xFfIPF DqkDgalOZYY+SPUihyNvg2PhShpgAVMBtDX6FBKQlfFpN0C9itrisfjbYo3ZfedfpunY zxy4CWzL8zp5KA95Q9WOX9e9EtmyXlYWpVdSL8EdGL3VQK4qkbyP52gcPtn4OKBv4+AH FLCw== X-Gm-Message-State: AOAM531C+T/f5t+PveQVMNXrNZHxuOKvGs5VTpH9G5g3uMRaZL4g6AwH 4q2Qex9GcojSAJ2BfeID1uqO/A== X-Google-Smtp-Source: ABdhPJyqmxnjyw5ql53tgyxYBbcNmlPhWAlfjLm7SZrRDqOvDmRTlTWmj7mIiYK7YZi2FXhxDTCLEQ== X-Received: by 2002:a7b:cb96:: with SMTP id m22mr582720wmi.164.1589911327419; Tue, 19 May 2020 11:02:07 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.02.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:02:07 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker , Bjorn Helgaas , Kuppuswamy Sathyanarayanan Subject: [PATCH v7 22/24] PCI/ATS: Add PRI stubs Date: Tue, 19 May 2020 19:55:00 +0200 Message-Id: <20200519175502.2504091-23-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org The SMMUv3 driver, which can be built without CONFIG_PCI, will soon gain support for PRI. Partially revert commit c6e9aefbf9db ("PCI/ATS: Remove unused PRI and PASID stubs") to re-introduce the PRI stubs, and avoid adding more #ifdefs to the SMMU driver. Acked-by: Bjorn Helgaas Reviewed-by: Kuppuswamy Sathyanarayanan Signed-off-by: Jean-Philippe Brucker --- include/linux/pci-ats.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/linux/pci-ats.h b/include/linux/pci-ats.h index f75c307f346d..e9e266df9b37 100644 --- a/include/linux/pci-ats.h +++ b/include/linux/pci-ats.h @@ -28,6 +28,14 @@ int pci_enable_pri(struct pci_dev *pdev, u32 reqs); void pci_disable_pri(struct pci_dev *pdev); int pci_reset_pri(struct pci_dev *pdev); int pci_prg_resp_pasid_required(struct pci_dev *pdev); +#else /* CONFIG_PCI_PRI */ +static inline int pci_enable_pri(struct pci_dev *pdev, u32 reqs) +{ return -ENODEV; } +static inline void pci_disable_pri(struct pci_dev *pdev) { } +static inline int pci_reset_pri(struct pci_dev *pdev) +{ return -ENODEV; } +static inline int pci_prg_resp_pasid_required(struct pci_dev *pdev) +{ return 0; } #endif /* CONFIG_PCI_PRI */ #ifdef CONFIG_PCI_PASID From patchwork Tue May 19 17:55:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 200343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CDC9C433EA for ; Tue, 19 May 2020 18:02:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1EB68207D3 for ; Tue, 19 May 2020 18:02:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="RaBfqvZB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729356AbgESSCL (ORCPT ); Tue, 19 May 2020 14:02:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33478 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728188AbgESSCL (ORCPT ); Tue, 19 May 2020 14:02:11 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C202BC08C5C1 for ; Tue, 19 May 2020 11:02:09 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id t8so1288629wmi.0 for ; Tue, 19 May 2020 11:02:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hORlUUicNTnmubGRwNiUffCoZPiWJG9HuOL8osSXdzo=; b=RaBfqvZBLG2+O2ys2+0Ikq5Wp+XHH/rqPMkeuqJRaQX3zplSn8zIVMkUTAPhGK28u9 wAEiv6Yy2jClvwowLeJGvp9nz/q2xjW3B1m8ef6AweRUFIRQce+1qhbFZu9Iqxpgb93T HQS/ayRa/QKZAcPdong1IOD4jlzFFccRioNNAzKo4Svis/lAW9o32Xi6WC9+XqJP5jVD 95wb3prQy7RhU+V9NvD9Uimr2HAe66cN0BSaK01ShE8K4Vh5b6WaVDUh0vqD6yii3Akb VvWJQ/nNMjB1sgj37nXUu8itMP/AiTtQykovHtt0BFwZVr2yuqgX+iT4E9CLWqxkgpmY SZRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hORlUUicNTnmubGRwNiUffCoZPiWJG9HuOL8osSXdzo=; b=YlJzYiUTNxI0WqkKzUtkIEUgmK8IVjBf9TpEJ7/XHzEmwn6NhBraJUE0Zoo/wxSwZj 1YByDaXIRp8a1a4dLSbOoaHe3pMYvfNAp1+NFJ46sxx683yFD9x218Uur3Y8Bl3PE9Zp KSSWB7Cc3JGvgo27mCEZ4fLA+wuGsUSP/rmX1dYjPKxub8hcZZevO1nUyhPfHqi4xaop HtnaCoH1MXzzci+Q/nA7KDhSG2a6rjOBYZ9BW7QflOnq1CwihdKmiq/2xslitmvB4yzt 4zCgvxqKImjo6Q3pu9wXeyh3Ex1SPN1r8ikvLXUY/wbOhwk40DKGhlh2REPEX1Eea/7Z Iiyw== X-Gm-Message-State: AOAM530JPjzFuCG5jIIOTjAj53Tq6UDMODTfVkA0Op8V8gMYmWTmUV3r 5kXIU/1kx1rftajx53kd+0ZKKw== X-Google-Smtp-Source: ABdhPJxkJn6Qhu9nn0EKvZWG6V4NMT7DJRYrUqG7GGKCgCc/g26MRyJ+LKLXQeOruPfp1Lim/+kFJw== X-Received: by 2002:a05:600c:29a:: with SMTP id 26mr615194wmk.151.1589911328505; Tue, 19 May 2020 11:02:08 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id 1sm510496wmz.13.2020.05.19.11.02.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 11:02:08 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, felix.kuehling@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, fenghua.yu@intel.com, hch@infradead.org, Jean-Philippe Brucker , Bjorn Helgaas , Kuppuswamy Sathyanarayanan Subject: [PATCH v7 23/24] PCI/ATS: Export PRI functions Date: Tue, 19 May 2020 19:55:01 +0200 Message-Id: <20200519175502.2504091-24-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519175502.2504091-1-jean-philippe@linaro.org> References: <20200519175502.2504091-1-jean-philippe@linaro.org> MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org The SMMUv3 driver uses pci_{enable,disable}_pri() and related functions. Export those functions to allow the driver to be built as a module. Acked-by: Bjorn Helgaas Reviewed-by: Kuppuswamy Sathyanarayanan Signed-off-by: Jean-Philippe Brucker --- drivers/pci/ats.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index a4722e8b6a51..418737a3c2c2 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -191,6 +191,7 @@ void pci_pri_init(struct pci_dev *pdev) if (status & PCI_PRI_STATUS_PASID) pdev->pasid_required = 1; } +EXPORT_SYMBOL_GPL(pci_pri_init); /** * pci_enable_pri - Enable PRI capability @@ -237,6 +238,7 @@ int pci_enable_pri(struct pci_dev *pdev, u32 reqs) return 0; } +EXPORT_SYMBOL_GPL(pci_enable_pri); /** * pci_disable_pri - Disable PRI capability @@ -316,6 +318,7 @@ int pci_reset_pri(struct pci_dev *pdev) return 0; } +EXPORT_SYMBOL_GPL(pci_reset_pri); /** * pci_prg_resp_pasid_required - Return PRG Response PASID Required bit @@ -331,6 +334,7 @@ int pci_prg_resp_pasid_required(struct pci_dev *pdev) return pdev->pasid_required; } +EXPORT_SYMBOL_GPL(pci_prg_resp_pasid_required); #endif /* CONFIG_PCI_PRI */ #ifdef CONFIG_PCI_PASID