From patchwork Thu Nov 12 12:55:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 324187 Delivered-To: patch@linaro.org Received: by 2002:a92:cc8a:0:0:0:0:0 with SMTP id x10csp323770ilo; Thu, 12 Nov 2020 05:03:53 -0800 (PST) X-Google-Smtp-Source: ABdhPJz9ZdtD7sS4M5mJmzFYS3jybWtAJaXKGNSgyz6/wnhxpSTmfnygYKAc/Qyytf1BF6mdM6Jz X-Received: by 2002:a17:906:2581:: with SMTP id m1mr29621807ejb.28.1605186233030; Thu, 12 Nov 2020 05:03:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1605186233; cv=none; d=google.com; s=arc-20160816; b=Jjb6cjoo1TSw8txBPPmfR6N6/yNjbNWrliyEfJlypPFdSLMEgthJ9LIAQtJpmAlHZ1 7n1OhVMBLMj9ZT3zEaj909q9FpjMnYD9SaIGRiFBrnPnrn9j9XApRdD3orzQKZJmh89Z qqJN/PCt9Ihd/D5XLec0Xksu/7eCqpjPVhRERJqnHTB96b65V580hbeBD7TW00Vx+Cju xrQtk9G33EbNc5z/1ofM6UIAjLVIGlsgLKOg+bqKI8mEN8KhX7/vi9k8qfXm1wSadhur T1K8rpp0BoJIbK9vwrRTn11WzaXfTq14iiAifHpm/QlDwAtY2LptMoDOot+ciydoNw+v 2WfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ct/A3+BqtZ1mh7NbSkQbTndKvraebCJIuStSjYZJj6w=; b=WdrdvYsHdbFVac1BbG1otpkyEdo99xLABWlR8q0xl74DrDidbeCzo4gtTwdlhTP5nR tGwiVX04FelP4Bi/oNf2j56lrZqHE0XwxMqs+WDxL14RqvgHhkRa/oVHtZAg42vBZ6Qh f25Pe5P4OjhQvPzMwynskQTNE0hRTDVomaawJ/3VjPaozo1awTgDhP8DzH0M8BV59zmJ TJjU/LJ5HX0Lfhmm3+h8Qg5HHn3J7cPRvm2O39jsPIiw1LA2reWS9pfcDBKfTDT3hc19 FMDDuE4cAKkGlD7GgweKeJ0BQeVoO2y+XxCzwCc4vZS4M5cCzJmHociOQepurzu6PMSD rDWQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Dn59G2jx; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u20si3713663edr.287.2020.11.12.05.03.51; Thu, 12 Nov 2020 05:03:53 -0800 (PST) Received-SPF: pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Dn59G2jx; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728144AbgKLNDm (ORCPT + 6 others); Thu, 12 Nov 2020 08:03:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728179AbgKLNDT (ORCPT ); Thu, 12 Nov 2020 08:03:19 -0500 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76180C0613D4 for ; Thu, 12 Nov 2020 05:03:18 -0800 (PST) Received: by mail-wr1-x443.google.com with SMTP id r17so5945812wrw.1 for ; Thu, 12 Nov 2020 05:03:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ct/A3+BqtZ1mh7NbSkQbTndKvraebCJIuStSjYZJj6w=; b=Dn59G2jxCJpXQrWrRlrml739Y0UGynYN7lH9lyw+1yaOF++vjgKKIvvlKYEGsBjX5C 7bdJR7rJAuBNA6tM+LlGlnNHHmCoaTyD4NWrpKiOjUIhCkls45V8Se7r6SFaJcMeKREy ODKuG3eMApLS+UrxG7p2Rcg/bO3sqLM8QwGCYaFJO+3NlqESiIm3vY68Cvs5nj2xrMeD dlNhxa1IyOUWrAQ7Uo1BnZ0Tj6HXA5JKm9KJDaeeMJSSnG8yPwAA3+IUcziyR/s3DN3v nlgfSNdjoxc/NeIWDE7/RF1ZTAOZSxAbW5VF+X0d1jOT8pO+zEOA1bjRrngz4uSQbQBP YIhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ct/A3+BqtZ1mh7NbSkQbTndKvraebCJIuStSjYZJj6w=; b=RfelQvmBo8DDw5Sp4l9DgAIfqJOS6RE2P6R/GcWORN3ZviT5Yxpm0q8mg0Dv/GTh0R BtBi3PKagRQ0+6qt5KwpFjpzRpvruI4cEqzYacSK3bhGTayJnjmrAfvHqMx+qne/6hj/ gkLJcU/daSoe+dc6x3L66cDL2roKnkT/1NhItM/fEsk8Kji0TprKKrdTM7plvewyYwcU dc9g0nXaOju6KuN00cwSvBFR8eHOQgZ6gtlhcQ3Pp6/8gG6ilBgfSkRw0lDnQsBmmT1a ZN81XwWvt0I0xEtHIjopHhQ81C3mE+293Hdirkp+QZsfbBdBDTBq6phQXLxQfUknJhEF yfrw== X-Gm-Message-State: AOAM532K0dcQYAHaSkkJNP9L14VTL013riDS4Kz0/P5ZX8G+aWkf1U3l gvZn8E/XYToFUU52W+9KoP2ekg== X-Received: by 2002:adf:eb47:: with SMTP id u7mr35100618wrn.163.1605186197169; Thu, 12 Nov 2020 05:03:17 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:16 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Cc: guohanjun@huawei.com, sudeep.holla@arm.com, rjw@rjwysocki.net, lenb@kernel.org, robin.murphy@arm.com, bhelgaas@google.com, Jonathan.Cameron@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, baolu.lu@linux.intel.com, zhangfei.gao@linaro.org, shameerali.kolothum.thodi@huawei.com, vivek.gautam@arm.com, Jean-Philippe Brucker Subject: [PATCH v8 2/9] iommu/arm-smmu-v3: Maintain a SID->device structure Date: Thu, 12 Nov 2020 13:55:14 +0100 Message-Id: <20201112125519.3987595-3-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org When handling faults from the event or PRI queue, we need to find the struct device associated to a SID. Add a rb_tree to keep track of SIDs. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 13 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 161 ++++++++++++++++---- 2 files changed, 144 insertions(+), 30 deletions(-) -- 2.29.1 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 58868a5677b6..7bd98fdce5c3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -648,6 +648,15 @@ struct arm_smmu_device { /* IOMMU core code handle */ struct iommu_device iommu; + + struct rb_root streams; + struct mutex streams_mutex; +}; + +struct arm_smmu_stream { + u32 id; + struct arm_smmu_master *master; + struct rb_node node; }; /* SMMU private data for each master */ @@ -656,8 +665,8 @@ struct arm_smmu_master { struct device *dev; struct arm_smmu_domain *domain; struct list_head domain_head; - u32 *sids; - unsigned int num_sids; + struct arm_smmu_stream *streams; + unsigned int num_streams; bool ats_enabled; bool sva_enabled; struct list_head bonds; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 453478c83933..d87c87136d63 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -918,8 +918,8 @@ static void arm_smmu_sync_cd(struct arm_smmu_domain *smmu_domain, spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_for_each_entry(master, &smmu_domain->devices, domain_head) { - for (i = 0; i < master->num_sids; i++) { - cmd.cfgi.sid = master->sids[i]; + for (i = 0; i < master->num_streams; i++) { + cmd.cfgi.sid = master->streams[i].id; arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd); } } @@ -1371,6 +1371,32 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid) return 0; } +__maybe_unused +static struct arm_smmu_master * +arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid) +{ + struct rb_node *node; + struct arm_smmu_stream *stream; + struct arm_smmu_master *master = NULL; + + mutex_lock(&smmu->streams_mutex); + node = smmu->streams.rb_node; + while (node) { + stream = rb_entry(node, struct arm_smmu_stream, node); + if (stream->id < sid) { + node = node->rb_right; + } else if (stream->id > sid) { + node = node->rb_left; + } else { + master = stream->master; + break; + } + } + mutex_unlock(&smmu->streams_mutex); + + return master; +} + /* IRQ and event handlers */ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) { @@ -1604,8 +1630,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) arm_smmu_atc_inv_to_cmd(0, 0, 0, &cmd); - for (i = 0; i < master->num_sids; i++) { - cmd.atc.sid = master->sids[i]; + for (i = 0; i < master->num_streams; i++) { + cmd.atc.sid = master->streams[i].id; arm_smmu_cmdq_issue_cmd(master->smmu, &cmd); } @@ -1648,8 +1674,8 @@ int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, if (!master->ats_enabled) continue; - for (i = 0; i < master->num_sids; i++) { - cmd.atc.sid = master->sids[i]; + for (i = 0; i < master->num_streams; i++) { + cmd.atc.sid = master->streams[i].id; arm_smmu_cmdq_batch_add(smmu_domain->smmu, &cmds, &cmd); } } @@ -2064,13 +2090,13 @@ static void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master) int i, j; struct arm_smmu_device *smmu = master->smmu; - for (i = 0; i < master->num_sids; ++i) { - u32 sid = master->sids[i]; + for (i = 0; i < master->num_streams; ++i) { + u32 sid = master->streams[i].id; __le64 *step = arm_smmu_get_step_for_sid(smmu, sid); /* Bridged PCI devices may end up with duplicated IDs */ for (j = 0; j < i; j++) - if (master->sids[j] == sid) + if (master->streams[j].id == sid) break; if (j < i) continue; @@ -2339,11 +2365,101 @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid) return sid < limit; } +static int arm_smmu_insert_master(struct arm_smmu_device *smmu, + struct arm_smmu_master *master) +{ + int i; + int ret = 0; + struct arm_smmu_stream *new_stream, *cur_stream; + struct rb_node **new_node, *parent_node = NULL; + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(master->dev); + + master->streams = kcalloc(fwspec->num_ids, + sizeof(struct arm_smmu_stream), GFP_KERNEL); + if (!master->streams) + return -ENOMEM; + master->num_streams = fwspec->num_ids; + + mutex_lock(&smmu->streams_mutex); + for (i = 0; i < fwspec->num_ids && !ret; i++) { + u32 sid = fwspec->ids[i]; + + new_stream = &master->streams[i]; + new_stream->id = sid; + new_stream->master = master; + + /* + * Check the SIDs are in range of the SMMU and our stream table + */ + if (!arm_smmu_sid_in_range(smmu, sid)) { + ret = -ERANGE; + break; + } + + /* Ensure l2 strtab is initialised */ + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + ret = arm_smmu_init_l2_strtab(smmu, sid); + if (ret) + break; + } + + /* Insert into SID tree */ + new_node = &(smmu->streams.rb_node); + while (*new_node) { + cur_stream = rb_entry(*new_node, struct arm_smmu_stream, + node); + parent_node = *new_node; + if (cur_stream->id > new_stream->id) { + new_node = &((*new_node)->rb_left); + } else if (cur_stream->id < new_stream->id) { + new_node = &((*new_node)->rb_right); + } else { + dev_warn(master->dev, + "stream %u already in tree\n", + cur_stream->id); + ret = -EINVAL; + break; + } + } + + if (!ret) { + rb_link_node(&new_stream->node, parent_node, new_node); + rb_insert_color(&new_stream->node, &smmu->streams); + } + } + + if (ret) { + for (; i > 0; i--) + rb_erase(&master->streams[i].node, &smmu->streams); + kfree(master->streams); + } + mutex_unlock(&smmu->streams_mutex); + + return ret; +} + +static void arm_smmu_remove_master(struct arm_smmu_master *master) +{ + int i; + struct arm_smmu_device *smmu = master->smmu; + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(master->dev); + + if (!smmu || !master->streams) + return; + + mutex_lock(&smmu->streams_mutex); + for (i = 0; i < fwspec->num_ids; i++) + rb_erase(&master->streams[i].node, &smmu->streams); + mutex_unlock(&smmu->streams_mutex); + + kfree(master->streams); +} + static struct iommu_ops arm_smmu_ops; static struct iommu_device *arm_smmu_probe_device(struct device *dev) { - int i, ret; + int ret; struct arm_smmu_device *smmu; struct arm_smmu_master *master; struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); @@ -2364,27 +2480,12 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->dev = dev; master->smmu = smmu; - master->sids = fwspec->ids; - master->num_sids = fwspec->num_ids; INIT_LIST_HEAD(&master->bonds); dev_iommu_priv_set(dev, master); - /* Check the SIDs are in range of the SMMU and our stream table */ - for (i = 0; i < master->num_sids; i++) { - u32 sid = master->sids[i]; - - if (!arm_smmu_sid_in_range(smmu, sid)) { - ret = -ERANGE; - goto err_free_master; - } - - /* Ensure l2 strtab is initialised */ - if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { - ret = arm_smmu_init_l2_strtab(smmu, sid); - if (ret) - goto err_free_master; - } - } + ret = arm_smmu_insert_master(smmu, master); + if (ret) + goto err_free_master; master->ssid_bits = min(smmu->ssid_bits, fwspec->num_pasid_bits); @@ -2422,6 +2523,7 @@ static void arm_smmu_release_device(struct device *dev) WARN_ON(arm_smmu_master_sva_enabled(master)); arm_smmu_detach_dev(master); arm_smmu_disable_pasid(master); + arm_smmu_remove_master(master); kfree(master); iommu_fwspec_free(dev); } @@ -2844,6 +2946,9 @@ static int arm_smmu_init_structures(struct arm_smmu_device *smmu) { int ret; + mutex_init(&smmu->streams_mutex); + smmu->streams = RB_ROOT; + ret = arm_smmu_init_queues(smmu); if (ret) return ret; From patchwork Thu Nov 12 12:55:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 324182 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:5ce:0:0:0:0 with SMTP id l14csp313299ils; Thu, 12 Nov 2020 05:03:26 -0800 (PST) X-Google-Smtp-Source: ABdhPJymWXKX1Tf57UfbcJ5RsmndTkE54VvrqOexsg+HOAoFW7I+6zyRMrTX296ZNtsxhcMj2Dd2 X-Received: by 2002:a5d:69d1:: with SMTP id s17mr25417636wrw.104.1605186206489; Thu, 12 Nov 2020 05:03:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1605186206; cv=none; d=google.com; s=arc-20160816; b=07FPkyNHGNsaSROSCxreUpAYAmeOmBVi+cNexGETP2ZBKCFzFUIJVj9G+BctKylVU/ 1MWwl0oumWXOAvVol2W8UDvMd1KS+VFrY5aRPBBxOguAmU/g3UNeyczAz5dM48i7Z1o9 h9bVsHbkD5PnCjxDhybbUqTgWpwwUF+UZS/rUUT1LICsGOBKHCA7jQz7MuzqI1/GCuK0 MqjlQstjVrlelhlVzlDhkZHu8uRqMqQlQZbhwLhJzjZBYf2e/XGTy4ZVSdk8k0O8rJlS /wFVtU0o4a+7DpOHS9XVVB3LiJZVDbCLRe04VLCKS+gXujCcNAenBhquOYpr5tzKkHAr bqRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Tid3zbIJRKyj0Bgjxj2sf+RSwZO5YPfaeHYqLr0g5iY=; b=HVa9waYTkY+7L5P2Ai6JCCOF/tnhY3kFGlUvX9/VvxipltlkyBF0tIScgYeOUVymqy PoYqwNHakjz+eTBwusRUJ5yiuYznCu7hc8u26JhK4cDl5SUa3+zslU+aqHkvhh7xSTOd 8Q2hCSmQ/QenSbg9aVjQuo+NCvDg0rqdKkiegmv8ag1bd91moCCs/LQIbnE//F9jjQ4y VoNnxzqbBkwCyuCyv/BvxbDD62zNFk8E0KTdSMw0unYpRgAZAP+Tc21Rc3JYg3ZZzxco NxAdDDxNzrFUO+Z0qS19s6kAWAyNnaRds0f2mqTSChxQPvwovj8yCDsheVlPiAIUx9/Y a6CA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=oJG4X6CI; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u20si3713663edr.287.2020.11.12.05.03.26; Thu, 12 Nov 2020 05:03:26 -0800 (PST) Received-SPF: pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=oJG4X6CI; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728298AbgKLNDX (ORCPT + 6 others); Thu, 12 Nov 2020 08:03:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728271AbgKLNDV (ORCPT ); Thu, 12 Nov 2020 08:03:21 -0500 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FEF9C0617A7 for ; Thu, 12 Nov 2020 05:03:20 -0800 (PST) Received: by mail-wr1-x444.google.com with SMTP id 23so5898411wrc.8 for ; Thu, 12 Nov 2020 05:03:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Tid3zbIJRKyj0Bgjxj2sf+RSwZO5YPfaeHYqLr0g5iY=; b=oJG4X6CI7enblSZUp3EgWunBFmyQv6mCwtptSww+yPpxypkzQpLOkXizEPseu9wI2t 2uZpYfOj9eUqkc5Mc5tlt3K7jx+TWvIflBQqX1SEYsvTIjIegzyCiXcmEFzhPjxvoA47 6Dq4ROrha3r07LiOgqCfR01hIVv7++Ib5t5K0yXdpav8K1LR8Zat1PWjsoKfQ809OKnH 1k5v7fX1Ma0DDtnaTPNqdNu6iefW98IR6j56uIkzclVoD8+TGJgwKYfDHt4NXx+25cGn SRyX77nEr1L/nTzdOO1Yo2Myq9+BofVGSodnPeoKBdMbtEUWw45R7NEYQRJNFh3Men13 MQBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Tid3zbIJRKyj0Bgjxj2sf+RSwZO5YPfaeHYqLr0g5iY=; b=I+91IsE4fUaYnbxWGSjz9aPIwCQhcPFmXP6R4EhwhIvomCZkQ+jqfADAnikVo+dSE2 nkmOEPn1P4VaCQkIEQZfjJ1pw427ONbqoUhHUIFpsssfZDrqF/9h9pX2YBK1j3vtdKaN QAdK0NBTLLxfmr4pDVcRS22fISavk7lao+3wcuynHx8D6JPTfg/yjyBz8C4k8PmY506T +52hQSpVlSLZIVLVFCoZBhw1U+nbGVD/fMOBlsvEofCVc91gDKfvnjPumMmW62s83EDI 9ROTn68zuuZWPG0ZjejF2bVtXCWTV8kqGB675Qe0/Uo86O2Lv91i3Xf2TTxMUTY1cGOS 8ORQ== X-Gm-Message-State: AOAM531brhWrbv9nZh6rrHoQgImheFoNJc9HrNkcRAxO5x03J/XRQHs4 j7xihg6o0Pu8z5daka3+Ct7fXg== X-Received: by 2002:adf:df88:: with SMTP id z8mr35234477wrl.113.1605186198680; Thu, 12 Nov 2020 05:03:18 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:17 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Cc: guohanjun@huawei.com, sudeep.holla@arm.com, rjw@rjwysocki.net, lenb@kernel.org, robin.murphy@arm.com, bhelgaas@google.com, Jonathan.Cameron@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, baolu.lu@linux.intel.com, zhangfei.gao@linaro.org, shameerali.kolothum.thodi@huawei.com, vivek.gautam@arm.com, Jean-Philippe Brucker , Rob Herring Subject: [PATCH v8 3/9] dt-bindings: document stall property for IOMMU masters Date: Thu, 12 Nov 2020 13:55:15 +0100 Message-Id: <20201112125519.3987595-4-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org On ARM systems, some platform devices behind an IOMMU may support stall, which is the ability to recover from page faults. Let the firmware tell us when a device supports stall. Reviewed-by: Rob Herring Signed-off-by: Jean-Philippe Brucker --- .../devicetree/bindings/iommu/iommu.txt | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) -- 2.29.1 diff --git a/Documentation/devicetree/bindings/iommu/iommu.txt b/Documentation/devicetree/bindings/iommu/iommu.txt index 3c36334e4f94..26ba9e530f13 100644 --- a/Documentation/devicetree/bindings/iommu/iommu.txt +++ b/Documentation/devicetree/bindings/iommu/iommu.txt @@ -92,6 +92,24 @@ Optional properties: tagging DMA transactions with an address space identifier. By default, this is 0, which means that the device only has one address space. +- dma-can-stall: When present, the master can wait for a transaction to + complete for an indefinite amount of time. Upon translation fault some + IOMMUs, instead of aborting the translation immediately, may first + notify the driver and keep the transaction in flight. This allows the OS + to inspect the fault and, for example, make physical pages resident + before updating the mappings and completing the transaction. Such IOMMU + accepts a limited number of simultaneous stalled transactions before + having to either put back-pressure on the master, or abort new faulting + transactions. + + Firmware has to opt-in stalling, because most buses and masters don't + support it. In particular it isn't compatible with PCI, where + transactions have to complete before a time limit. More generally it + won't work in systems and masters that haven't been designed for + stalling. For example the OS, in order to handle a stalled transaction, + may attempt to retrieve pages from secondary storage in a stalled + domain, leading to a deadlock. + Notes: ====== From patchwork Thu Nov 12 12:55:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 324183 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:5ce:0:0:0:0 with SMTP id l14csp313465ils; Thu, 12 Nov 2020 05:03:36 -0800 (PST) X-Google-Smtp-Source: ABdhPJxQyKFEcz/ww1OiuJ5qCycM69oWW+0igpNaczR4YG6IhHdKVD0Lq1k0cBb/g+VRqd5+oXcM X-Received: by 2002:a17:906:14d1:: with SMTP id y17mr30792905ejc.15.1605186215954; Thu, 12 Nov 2020 05:03:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1605186215; cv=none; d=google.com; s=arc-20160816; b=LIoXlC8Q6bMxhvOv5iGuVzYRusD5lUOEeJjAf6ND7L2ApHvWhhdNx6bd1Z8zNj/r7S 3GK8q1bBizH86bOjD5z/uTtyvZWaEWxlsO1hQfScFivAWCg+8rOhmT7M0Dit65V9Na2u bp/p1EKtwzQFCWhdfKfH9xHghOUUsQry/O/juUoheC+jQ6q3YIayLtqM+ctbKuIbTMjq VOS94iz1Pqw6Zg1mCPzl45Gr6KY5i7T0j/aBdw0Nab/X3kxijYR7WJShR69InAh2RoHk C0g6tRQCPduDeRz/FqFgtuZnw3G7/fAidQlywschKCSv4Pt8PBLjhbil2fhrBzi6Kt9x JOcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3ygm1AXTLu+53w4Gri7BYEJbuhvyMFezCUVbiVOA1cI=; b=Alvi6sZ1w3f5MrJ/mE+ErqxuF9EBQgzo7Aff+AyamDdhlUmTZomAn259vtnHRI5ZHG 1+zQ1VkAj33dX30ivaFoFqHatyPa36VwZppiAkeq4o14KdKuOCIWou0PlgwMKiqbCfB/ puneFXu5G5tc04H0CsExL0ZjXmF4IXB1XZgdD1kdPXEUTaRkd2Y1NOVsR4Sy8JabY3c5 LVGwP3M5+8lvoSPGEeSyrYmGBbjtL7hQiBmnHHXMQuDA1LdV0M4WTD9qnOrLEbMYK7cr B/5uzKfytqeNOrEBKx6HSSFyL89PVbARtpdL+JCiNf0qm/UNsoJZJ6+tThnWdCknniUE Mcfg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=uA63WGlR; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u20si3713663edr.287.2020.11.12.05.03.35; Thu, 12 Nov 2020 05:03:35 -0800 (PST) Received-SPF: pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=uA63WGlR; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728294AbgKLND0 (ORCPT + 6 others); Thu, 12 Nov 2020 08:03:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728291AbgKLNDX (ORCPT ); Thu, 12 Nov 2020 08:03:23 -0500 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70649C061A49 for ; Thu, 12 Nov 2020 05:03:21 -0800 (PST) Received: by mail-wr1-x442.google.com with SMTP id b8so5942427wrn.0 for ; Thu, 12 Nov 2020 05:03:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3ygm1AXTLu+53w4Gri7BYEJbuhvyMFezCUVbiVOA1cI=; b=uA63WGlRDXWDF9rIZV4RZkVq6oJU37wcFuSQmD8BzO159BANMh6M+qpRZeX3/9SDUX NgjC/6kMJoth33QcDclO+ynaJlZA5DobFNRQNYs96uO5BecxpNBWMkORbZcRWO5P6qD/ BJkF/jlsL/X45r34/KxlPQbUjbKshwp4elCUfaZGnESVEwpxokGBOQy9Uvj/8ap3OCIH X4XkbC3JjCUSG2bj/0UjM+bplVf6wDID7rR7qluxsv54s4WElIx8dWnQLCnODfH4+4TN 1fQtFlyLdLdG2hhM+mcaKhRK/XdUJ1cFPfj1ecKiWJDeJr3prD0gWxlUnB4H0tf5eUhK 2bmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3ygm1AXTLu+53w4Gri7BYEJbuhvyMFezCUVbiVOA1cI=; b=PP638mlKC7m8VEkPqFnaQczGl4i1IdXG3fE17zTThUYVuiIlnylnowmnWsiSK+aY83 ULujfopDpScpn9UYj4LlzWE12FgTfp8yeFoJgq+xVZIKiFGSnYCe/5USZ7jCVAUlyijj SFZxtkDQEPeKFc2Gob1BsTSLsufX/B1J46pwOXf/Vq3eGIdiVsmGIn4belr+3yeMevOp 2Lz0BazKKGE35d7WcsakwtLAeaQFbGnUc/C7pa3AenNPRtnHA4jQ7gaXDul8FTKnPEEJ QK/079Ga9KnXKSR7NCe9J96rLTmzMD+Z7Dq7VFo2XUjWdfMjoIGwPPb/ie56xpi02ALZ myEA== X-Gm-Message-State: AOAM5305duZs/pXD5ApF2SVXxeyPLIh9dyKYcKwJ7ecIa8HdK/xBdk0u +1HyBcSaHgqWENEH3ZQjygPg4w== X-Received: by 2002:a5d:660f:: with SMTP id n15mr30481293wru.345.1605186200191; Thu, 12 Nov 2020 05:03:20 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:19 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Cc: guohanjun@huawei.com, sudeep.holla@arm.com, rjw@rjwysocki.net, lenb@kernel.org, robin.murphy@arm.com, bhelgaas@google.com, Jonathan.Cameron@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, baolu.lu@linux.intel.com, zhangfei.gao@linaro.org, shameerali.kolothum.thodi@huawei.com, vivek.gautam@arm.com, Jean-Philippe Brucker Subject: [PATCH v8 4/9] of/iommu: Support dma-can-stall property Date: Thu, 12 Nov 2020 13:55:16 +0100 Message-Id: <20201112125519.3987595-5-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Copy the dma-can-stall property into the fwspec structure. Signed-off-by: Jean-Philippe Brucker --- include/linux/iommu.h | 2 ++ drivers/iommu/of_iommu.c | 5 ++++- 2 files changed, 6 insertions(+), 1 deletion(-) -- 2.29.1 diff --git a/include/linux/iommu.h b/include/linux/iommu.h index a1c78c4cdeb1..9076fb592c8f 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -572,6 +572,7 @@ struct iommu_group *fsl_mc_device_group(struct device *dev); * @iommu_fwnode: firmware handle for this device's IOMMU * @iommu_priv: IOMMU driver private data for this device * @num_pasid_bits: number of PASID bits supported by this device + * @can_stall: the device is allowed to stall * @num_ids: number of associated device IDs * @ids: IDs which this device may present to the IOMMU */ @@ -579,6 +580,7 @@ struct iommu_fwspec { const struct iommu_ops *ops; struct fwnode_handle *iommu_fwnode; u32 num_pasid_bits; + bool can_stall; unsigned int num_ids; u32 ids[]; }; diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c index e505b9130a1c..d6255ca823d8 100644 --- a/drivers/iommu/of_iommu.c +++ b/drivers/iommu/of_iommu.c @@ -212,9 +212,12 @@ const struct iommu_ops *of_iommu_configure(struct device *dev, err = of_iommu_configure_device(master_np, dev, id); fwspec = dev_iommu_fwspec_get(dev); - if (!err && fwspec) + if (!err && fwspec) { of_property_read_u32(master_np, "pasid-num-bits", &fwspec->num_pasid_bits); + fwspec->can_stall = of_property_read_bool(master_np, + "dma-can-stall"); + } } /* From patchwork Thu Nov 12 12:55:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 324185 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:5ce:0:0:0:0 with SMTP id l14csp313505ils; Thu, 12 Nov 2020 05:03:38 -0800 (PST) X-Google-Smtp-Source: ABdhPJyVU4D7dhY5xGigHDbUct1RV5jXNHn9WvAz6sDkzHFQTrbQ3PjKrYW/QLDtD3pmOCpZlgay X-Received: by 2002:a50:9fc1:: with SMTP id c59mr4776776edf.59.1605186218092; Thu, 12 Nov 2020 05:03:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1605186218; cv=none; d=google.com; s=arc-20160816; b=vcRoVvu7y0aGstI5YWlPZPdm97BTzmjw9B8LMpPad5cKqbrdwo8AnjikJ2TnkPyZiY LT3e29EedI9PaK2TKu4gGoxlV8ELmpaP8nGc6ihbUTw/XjL67vj4wvpAoZGO0fn6vVwl dRSa1FFD4iKFNYRJZCmEyKTqCi4fG7VTR2VADbdwMCLbHFexEvtnJtFXss+Ra4/5JfRm r10YYl/Ie5IX4+bwD2pIWUViYqRkAHjGs/Dyi5QmpDjk8eUhkxiiCJGdGux75yVArFcM /XbEoCwF+cAzhFF2wL7TwnrDcoNxncrLfeak4NqfZuEibLxbqXGwiYxMVROcBDFrw03+ 4lew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cfEAlXk9Q/1m3LFYgSiXuEmltFiYPYDgoNOZyZUs8uU=; b=xaWrP8upUQHvi2LtslVZt2TQ6F9lVOK7PWcl3MB36aMjUtAOyp58KUctAB6hadSv9x CXCfNAVVO8k2Hpid56qa1o0nMrWCe2CxHMdaRgZdZBTPYqZGQGo/5yE2mRIk3gP+VKmh CLksArBJUOF6rYfZHwREPCszl34Yw785IOqP/oFQSiTbIT/UYH5KPLhDW+cSfTlCK1e2 JXmNlwO+LTwBE1NYaVLFHlLkaDIFWDj6hEXOk4Wi6yBMD9BiuSgWhehiQdGfaS00TPR6 zAEPeoQTj4K8XXAFn/j3R0u5u4pn398AIKp3Y39vN12qkCYJlaCXL7XXHAEtV1UmPtI9 n4oQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=xRmdctiu; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u20si3713663edr.287.2020.11.12.05.03.37; Thu, 12 Nov 2020 05:03:38 -0800 (PST) Received-SPF: pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=xRmdctiu; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728265AbgKLND2 (ORCPT + 6 others); Thu, 12 Nov 2020 08:03:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728309AbgKLND1 (ORCPT ); Thu, 12 Nov 2020 08:03:27 -0500 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83183C0617A6 for ; Thu, 12 Nov 2020 05:03:24 -0800 (PST) Received: by mail-wm1-x342.google.com with SMTP id c16so5441944wmd.2 for ; Thu, 12 Nov 2020 05:03:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cfEAlXk9Q/1m3LFYgSiXuEmltFiYPYDgoNOZyZUs8uU=; b=xRmdctiu68lUscLl4F5kD3aAMn/mNPne77+RMTMrwtYwTU5hY0lUgKs+DCAsBfckH5 h+jbd1Q2MJvWNTpvmGwglKXdTsy5ETmzHwulfMVXUHZR0vh+1V+D1cl4xXAmhC5bUuNf 3PnrpXxoAnAKR9uNbj7eH9MVZPZfXWyiF8Yr5Bj4Bmo8BlNgNM+jqKKxSv7jlVII/dYx 8Mm04Zag62Qd34vkI7TIxiCV4QukQy4csUJKTzUp8kGquqoLEma5QVzgRwCJl1N6s6sG 1en0i9zujF5wJHHHYWMHSAEO7BWE5l8f4u7J8uurlWNynBzacQk4+KKS/ILL/RZZU0hZ /qJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cfEAlXk9Q/1m3LFYgSiXuEmltFiYPYDgoNOZyZUs8uU=; b=hqxfDCPM30k2MGSeHocJe/B8cpSopWTcF0Uml2SzUe1Tsq3tPjDNRBUECLLi1Xx6pj FZPgT9uO1/N6c8uZHhH7Wg0mP6mrZo0TjTzNTtnYHUMIQpDJbAwyZOftz7aY7wnT4/La 8aiQhcE5QH3Cr6SNA3iMM3h01zQmjtqy1CJ2MXeILPYLCMEAW428potdhf/3eA1faYLs wE8tDyHB73F2sI2Ht4FHKqI/j80teCWPt1cc+dZQ6fQPE36p88hNXcbipLWQvwJisg8H Ah+7km7aTutAkuR/4TFbU96q0EtBqy1SJ6GonMnaLlNR7oLiLzCq6pZV/lUJUZRAHZ6M tvfw== X-Gm-Message-State: AOAM5306JEx6W2tRzzXgCpaXr4VUPYA240KVvdp466KySZE2eohdB4C0 kvl55PTPBJCQN2Y4iBBhfzMYNQ== X-Received: by 2002:a1c:ddc4:: with SMTP id u187mr9985624wmg.55.1605186203154; Thu, 12 Nov 2020 05:03:23 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:22 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Cc: guohanjun@huawei.com, sudeep.holla@arm.com, rjw@rjwysocki.net, lenb@kernel.org, robin.murphy@arm.com, bhelgaas@google.com, Jonathan.Cameron@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, baolu.lu@linux.intel.com, zhangfei.gao@linaro.org, shameerali.kolothum.thodi@huawei.com, vivek.gautam@arm.com, Jean-Philippe Brucker Subject: [PATCH v8 6/9] iommu/arm-smmu-v3: Add stall support for platform devices Date: Thu, 12 Nov 2020 13:55:18 +0100 Message-Id: <20201112125519.3987595-7-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org The SMMU provides a Stall model for handling page faults in platform devices. It is similar to PCIe PRI, but doesn't require devices to have their own translation cache. Instead, faulting transactions are parked and the OS is given a chance to fix the page tables and retry the transaction. Enable stall for devices that support it (opt-in by firmware). When an event corresponds to a translation error, call the IOMMU fault handler. If the fault is recoverable, it will call us back to terminate or continue the stall. Signed-off-by: Jean-Philippe Brucker --- v8: * Extract firwmare setting into separate patch * Don't drain event queue on unbind() --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 36 ++++ .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 26 ++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 173 +++++++++++++++++- 3 files changed, 224 insertions(+), 11 deletions(-) -- 2.29.1 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 7bd98fdce5c3..124f604ed677 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -361,6 +361,13 @@ #define CMDQ_PRI_1_GRPID GENMASK_ULL(8, 0) #define CMDQ_PRI_1_RESP GENMASK_ULL(13, 12) +#define CMDQ_RESUME_0_SID GENMASK_ULL(63, 32) +#define CMDQ_RESUME_0_RESP_TERM 0UL +#define CMDQ_RESUME_0_RESP_RETRY 1UL +#define CMDQ_RESUME_0_RESP_ABORT 2UL +#define CMDQ_RESUME_0_RESP GENMASK_ULL(13, 12) +#define CMDQ_RESUME_1_STAG GENMASK_ULL(15, 0) + #define CMDQ_SYNC_0_CS GENMASK_ULL(13, 12) #define CMDQ_SYNC_0_CS_NONE 0 #define CMDQ_SYNC_0_CS_IRQ 1 @@ -377,6 +384,25 @@ #define EVTQ_0_ID GENMASK_ULL(7, 0) +#define EVT_ID_TRANSLATION_FAULT 0x10 +#define EVT_ID_ADDR_SIZE_FAULT 0x11 +#define EVT_ID_ACCESS_FAULT 0x12 +#define EVT_ID_PERMISSION_FAULT 0x13 + +#define EVTQ_0_SSV (1UL << 11) +#define EVTQ_0_SSID GENMASK_ULL(31, 12) +#define EVTQ_0_SID GENMASK_ULL(63, 32) +#define EVTQ_1_STAG GENMASK_ULL(15, 0) +#define EVTQ_1_STALL (1UL << 31) +#define EVTQ_1_PRIV (1UL << 33) +#define EVTQ_1_EXEC (1UL << 34) +#define EVTQ_1_READ (1UL << 35) +#define EVTQ_1_S2 (1UL << 39) +#define EVTQ_1_CLASS GENMASK_ULL(41, 40) +#define EVTQ_1_TT_READ (1UL << 44) +#define EVTQ_2_ADDR GENMASK_ULL(63, 0) +#define EVTQ_3_IPA GENMASK_ULL(51, 12) + /* PRI queue */ #define PRIQ_ENT_SZ_SHIFT 4 #define PRIQ_ENT_DWORDS ((1 << PRIQ_ENT_SZ_SHIFT) >> 3) @@ -471,6 +497,13 @@ struct arm_smmu_cmdq_ent { enum pri_resp resp; } pri; + #define CMDQ_OP_RESUME 0x44 + struct { + u32 sid; + u16 stag; + u8 resp; + } resume; + #define CMDQ_OP_CMD_SYNC 0x46 struct { u64 msiaddr; @@ -529,6 +562,7 @@ struct arm_smmu_cmdq_batch { struct arm_smmu_evtq { struct arm_smmu_queue q; + struct iopf_queue *iopf; u32 max_stalls; }; @@ -668,6 +702,7 @@ struct arm_smmu_master { struct arm_smmu_stream *streams; unsigned int num_streams; bool ats_enabled; + bool stall_enabled; bool sva_enabled; struct list_head bonds; unsigned int ssid_bits; @@ -687,6 +722,7 @@ struct arm_smmu_domain { struct io_pgtable_ops *pgtbl_ops; bool non_strict; + bool stall_enabled; atomic_t nr_ats_masters; enum arm_smmu_domain_stage stage; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index f734797ea07a..64e2082ef9ed 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -435,7 +435,7 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) static bool arm_smmu_iopf_supported(struct arm_smmu_master *master) { - return false; + return master->stall_enabled; } bool arm_smmu_master_sva_supported(struct arm_smmu_master *master) @@ -459,24 +459,46 @@ bool arm_smmu_master_sva_enabled(struct arm_smmu_master *master) int arm_smmu_master_enable_sva(struct arm_smmu_master *master) { + int ret; + struct device *dev = master->dev; + + if (master->stall_enabled) { + ret = iopf_queue_add_device(master->smmu->evtq.iopf, dev); + if (ret) + return ret; + } + + ret = iommu_register_device_fault_handler(dev, iommu_queue_iopf, dev); + if (ret) + goto err_disable_iopf; + mutex_lock(&sva_lock); master->sva_enabled = true; mutex_unlock(&sva_lock); return 0; + +err_disable_iopf: + iopf_queue_remove_device(master->smmu->evtq.iopf, dev); + return ret; } int arm_smmu_master_disable_sva(struct arm_smmu_master *master) { + struct device *dev = master->dev; + mutex_lock(&sva_lock); if (!list_empty(&master->bonds)) { - dev_err(master->dev, "cannot disable SVA, device is bound\n"); + dev_err(dev, "cannot disable SVA, device is bound\n"); mutex_unlock(&sva_lock); return -EBUSY; } master->sva_enabled = false; mutex_unlock(&sva_lock); + iommu_unregister_device_fault_handler(dev); + iopf_queue_remove_device(master->smmu->evtq.iopf, dev); + return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index d87c87136d63..d412d063d3b6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -32,6 +32,7 @@ #include #include "arm-smmu-v3.h" +#include "../../iommu-sva-lib.h" static bool disable_bypass = 1; module_param(disable_bypass, bool, 0444); @@ -324,6 +325,11 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) } cmd[1] |= FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); break; + case CMDQ_OP_RESUME: + cmd[0] |= FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid); + cmd[0] |= FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp); + cmd[1] |= FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag); + break; case CMDQ_OP_CMD_SYNC: if (ent->sync.msiaddr) { cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); @@ -887,6 +893,44 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, return arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmds, cmds->num, true); } +static int arm_smmu_page_response(struct device *dev, + struct iommu_fault_event *unused, + struct iommu_page_response *resp) +{ + struct arm_smmu_cmdq_ent cmd = {0}; + struct arm_smmu_master *master = dev_iommu_priv_get(dev); + int sid = master->streams[0].id; + + if (master->stall_enabled) { + cmd.opcode = CMDQ_OP_RESUME; + cmd.resume.sid = sid; + cmd.resume.stag = resp->grpid; + switch (resp->code) { + case IOMMU_PAGE_RESP_INVALID: + case IOMMU_PAGE_RESP_FAILURE: + cmd.resume.resp = CMDQ_RESUME_0_RESP_ABORT; + break; + case IOMMU_PAGE_RESP_SUCCESS: + cmd.resume.resp = CMDQ_RESUME_0_RESP_RETRY; + break; + default: + return -EINVAL; + } + } else { + return -ENODEV; + } + + arm_smmu_cmdq_issue_cmd(master->smmu, &cmd); + /* + * Don't send a SYNC, it doesn't do anything for RESUME or PRI_RESP. + * RESUME consumption guarantees that the stalled transaction will be + * terminated... at some point in the future. PRI_RESP is fire and + * forget. + */ + + return 0; +} + /* Context descriptor manipulation functions */ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid) { @@ -1049,8 +1093,7 @@ int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid, FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid) | CTXDESC_CD_0_V; - /* STALL_MODEL==0b10 && CD.S==0 is ILLEGAL */ - if (smmu->features & ARM_SMMU_FEAT_STALL_FORCE) + if (smmu_domain->stall_enabled) val |= CTXDESC_CD_0_S; } @@ -1294,7 +1337,7 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, FIELD_PREP(STRTAB_STE_1_STRW, strw)); if (smmu->features & ARM_SMMU_FEAT_STALLS && - !(smmu->features & ARM_SMMU_FEAT_STALL_FORCE)) + !master->stall_enabled) dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD); val |= (s1_cfg->cdcfg.cdtab_dma & STRTAB_STE_0_S1CTXPTR_MASK) | @@ -1371,7 +1414,6 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid) return 0; } -__maybe_unused static struct arm_smmu_master * arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid) { @@ -1398,9 +1440,96 @@ arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid) } /* IRQ and event handlers */ +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 *evt) +{ + int ret; + u32 perm = 0; + struct arm_smmu_master *master; + bool ssid_valid = evt[0] & EVTQ_0_SSV; + u8 type = FIELD_GET(EVTQ_0_ID, evt[0]); + u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]); + struct iommu_fault_event fault_evt = { }; + struct iommu_fault *flt = &fault_evt.fault; + + /* Stage-2 is always pinned at the moment */ + if (evt[1] & EVTQ_1_S2) + return -EFAULT; + + master = arm_smmu_find_master(smmu, sid); + if (!master) + return -EINVAL; + + if (evt[1] & EVTQ_1_READ) + perm |= IOMMU_FAULT_PERM_READ; + else + perm |= IOMMU_FAULT_PERM_WRITE; + + if (evt[1] & EVTQ_1_EXEC) + perm |= IOMMU_FAULT_PERM_EXEC; + + if (evt[1] & EVTQ_1_PRIV) + perm |= IOMMU_FAULT_PERM_PRIV; + + if (evt[1] & EVTQ_1_STALL) { + flt->type = IOMMU_FAULT_PAGE_REQ; + flt->prm = (struct iommu_fault_page_request) { + .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, + .grpid = FIELD_GET(EVTQ_1_STAG, evt[1]), + .perm = perm, + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), + }; + + if (ssid_valid) { + flt->prm.flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + flt->prm.pasid = FIELD_GET(EVTQ_0_SSID, evt[0]); + } + } else { + flt->type = IOMMU_FAULT_DMA_UNRECOV; + flt->event = (struct iommu_fault_unrecoverable) { + .flags = IOMMU_FAULT_UNRECOV_ADDR_VALID | + IOMMU_FAULT_UNRECOV_FETCH_ADDR_VALID, + .perm = perm, + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), + .fetch_addr = FIELD_GET(EVTQ_3_IPA, evt[3]), + }; + + if (ssid_valid) { + flt->event.flags |= IOMMU_FAULT_UNRECOV_PASID_VALID; + flt->event.pasid = FIELD_GET(EVTQ_0_SSID, evt[0]); + } + + switch (type) { + case EVT_ID_TRANSLATION_FAULT: + case EVT_ID_ADDR_SIZE_FAULT: + case EVT_ID_ACCESS_FAULT: + flt->event.reason = IOMMU_FAULT_REASON_PTE_FETCH; + break; + case EVT_ID_PERMISSION_FAULT: + flt->event.reason = IOMMU_FAULT_REASON_PERMISSION; + break; + default: + /* TODO: report other unrecoverable faults. */ + return -EFAULT; + } + } + + ret = iommu_report_device_fault(master->dev, &fault_evt); + if (ret && flt->type == IOMMU_FAULT_PAGE_REQ) { + /* Nobody cared, abort the access */ + struct iommu_page_response resp = { + .pasid = flt->prm.pasid, + .grpid = flt->prm.grpid, + .code = IOMMU_PAGE_RESP_FAILURE, + }; + arm_smmu_page_response(master->dev, NULL, &resp); + } + + return ret; +} + static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) { - int i; + int i, ret; struct arm_smmu_device *smmu = dev; struct arm_smmu_queue *q = &smmu->evtq.q; struct arm_smmu_ll_queue *llq = &q->llq; @@ -1410,11 +1539,15 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) while (!queue_remove_raw(q, evt)) { u8 id = FIELD_GET(EVTQ_0_ID, evt[0]); - dev_info(smmu->dev, "event 0x%02x received:\n", id); - for (i = 0; i < ARRAY_SIZE(evt); ++i) - dev_info(smmu->dev, "\t0x%016llx\n", - (unsigned long long)evt[i]); + ret = arm_smmu_handle_evt(smmu, evt); + if (ret) { + dev_info(smmu->dev, "event 0x%02x received:\n", + id); + for (i = 0; i < ARRAY_SIZE(evt); ++i) + dev_info(smmu->dev, "\t0x%016llx\n", + (unsigned long long)evt[i]); + } } /* @@ -1927,6 +2060,8 @@ static int arm_smmu_domain_finalise_s1(struct arm_smmu_domain *smmu_domain, cfg->s1cdmax = master->ssid_bits; + smmu_domain->stall_enabled = master->stall_enabled; + ret = arm_smmu_alloc_cd_tables(smmu_domain); if (ret) goto out_free_asid; @@ -2270,6 +2405,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) smmu_domain->s1_cfg.s1cdmax, master->ssid_bits); ret = -EINVAL; goto out_unlock; + } else if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1 && + smmu_domain->stall_enabled != master->stall_enabled) { + dev_err(dev, "cannot attach to stall-%s domain\n", + smmu_domain->stall_enabled ? "enabled" : "disabled"); + ret = -EINVAL; + goto out_unlock; } master->domain = smmu_domain; @@ -2503,6 +2644,10 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) master->ssid_bits = min_t(u8, master->ssid_bits, CTXDESC_LINEAR_CDMAX); + if ((smmu->features & ARM_SMMU_FEAT_STALLS && fwspec->can_stall) || + smmu->features & ARM_SMMU_FEAT_STALL_FORCE) + master->stall_enabled = true; + return &smmu->iommu; err_free_master: @@ -2521,6 +2666,7 @@ static void arm_smmu_release_device(struct device *dev) master = dev_iommu_priv_get(dev); WARN_ON(arm_smmu_master_sva_enabled(master)); + iopf_queue_remove_device(master->smmu->evtq.iopf, dev); arm_smmu_detach_dev(master); arm_smmu_disable_pasid(master); arm_smmu_remove_master(master); @@ -2727,6 +2873,7 @@ static struct iommu_ops arm_smmu_ops = { .sva_bind = arm_smmu_sva_bind, .sva_unbind = arm_smmu_sva_unbind, .sva_get_pasid = arm_smmu_sva_get_pasid, + .page_response = arm_smmu_page_response, .pgsize_bitmap = -1UL, /* Restricted during device attach */ }; @@ -2804,6 +2951,7 @@ static int arm_smmu_cmdq_init(struct arm_smmu_device *smmu) static int arm_smmu_init_queues(struct arm_smmu_device *smmu) { int ret; + bool sva = arm_smmu_sva_supported(smmu); /* cmdq */ ret = arm_smmu_init_one_queue(smmu, &smmu->cmdq.q, ARM_SMMU_CMDQ_PROD, @@ -2823,6 +2971,12 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu) if (ret) return ret; + if (sva && smmu->features & ARM_SMMU_FEAT_STALLS) { + smmu->evtq.iopf = iopf_queue_alloc(dev_name(smmu->dev)); + if (!smmu->evtq.iopf) + return -ENOMEM; + } + /* priq */ if (!(smmu->features & ARM_SMMU_FEAT_PRI)) return 0; @@ -3789,6 +3943,7 @@ static int arm_smmu_device_remove(struct platform_device *pdev) iommu_device_unregister(&smmu->iommu); iommu_device_sysfs_remove(&smmu->iommu); arm_smmu_device_disable(smmu); + iopf_queue_free(smmu->evtq.iopf); return 0; } From patchwork Thu Nov 12 12:55:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 324189 Delivered-To: patch@linaro.org Received: by 2002:a05:6e02:5ce:0:0:0:0 with SMTP id l14csp313681ils; Thu, 12 Nov 2020 05:03:50 -0800 (PST) X-Google-Smtp-Source: ABdhPJy+aTfnlzHnZDrq+FuTefFXi0pXn07VexabgSI+mCAwzYNnGQE98lXjox57/5T8urKCpiUq X-Received: by 2002:a17:907:d1e:: with SMTP id gn30mr31422003ejc.148.1605186230415; Thu, 12 Nov 2020 05:03:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1605186230; cv=none; d=google.com; s=arc-20160816; b=PihCeNClmrCQHMCNRU1w/YBtiw4LDpPTekY8VPjOf2u19kujhtUKtivxWOjl1+XzCL mW6IUIXqHwv0Iwvanld/k1GwQLaVFiQxz+moe3Ci6gBlTr/sa9+YOu3TOLx53k59ANiY mR38QNAL0boLZN5px1pKqp7EmfbIeNVxHPF0EspMUZ0lKtvwHF69DNqrdGsM0Iltqqj2 cK5ysLwVAMPSSOtzLJo8xNjMyQploEuJ8UlekvwfzyozmIz79MeAMHlWM7w6KLacIdWt kMc0WxH1VdlBFtN7Uuy0FaFk7yqLJdwcQ+ymceHPaj3X/SyeIu5kNsq9Dt1dEkCyET1T zw1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KzHG2PweYWkDaJQol3GXcq+4FrqBl3EkRXm7+VNwUmc=; b=J23IaKeaeHvvAp0DMvXuiVB592zW12P+yw6+2GA9iX4bI1Xeuo9GS/wz4fWKxfUCJL v+Dg5gsXvpZ0L/7SVaDhr/0K2rGGupog2D8jeNChdYhC6+NKAc7KU7vwXsZ7XwiAEebi nk1F6SlMYGh8tDteRe7LM066J1tHLAhcVPm+F5USp3AE+vK2p0xdftDW7jljJ/Z9L50u Y8gW4omCDnP39KV3DZWxjrPnfV3cueN3DvQv3TZzhsn3KIxOHyu0D92FAesE2Rcj/61d bZ+Nnf5wSTM2nzBD+Pw7MRcTZ9hev0NXm5UPpnEcsyU1nqRB0oomN71iatL7fnpZAT8x gCzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QzsCjHvg; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u20si3713663edr.287.2020.11.12.05.03.50; Thu, 12 Nov 2020 05:03:50 -0800 (PST) Received-SPF: pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=QzsCjHvg; spf=pass (google.com: domain of devicetree-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=devicetree-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728327AbgKLNDj (ORCPT + 6 others); Thu, 12 Nov 2020 08:03:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728326AbgKLNDa (ORCPT ); Thu, 12 Nov 2020 08:03:30 -0500 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2233AC0617A7 for ; Thu, 12 Nov 2020 05:03:30 -0800 (PST) Received: by mail-wr1-x443.google.com with SMTP id b6so5922124wrt.4 for ; Thu, 12 Nov 2020 05:03:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KzHG2PweYWkDaJQol3GXcq+4FrqBl3EkRXm7+VNwUmc=; b=QzsCjHvgwTaDtXJ9+Q3O62Yd7pbjZCU3265TBqqZMjaMxMwjrdRp4jHXltzEEU07rs 8mmfQkm8fEnqYqM0E0REckDPxMoROlKoFtzVSydnxin5V/rRTpPx8c5+TuZBGtaTYEuO f4LcmtfroCNh/UOj+hBVBMGKXKu9b7RngKKy6U2RaRfTApWKsBcZ42osFB8fh4Pp9Nqs W67apIz096ig0sdMkpZ2/bsUrG70tIHfMCGQW8OLPOyP3hEhSDWYdVJKLEoDu4+aCb4V Ds1lnR59Y3qMAHjkDFnLi9V4r9sGgD5xjVu1aE0drSG6GBMw+oN9u15GPn+X/9k+IF7F km9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KzHG2PweYWkDaJQol3GXcq+4FrqBl3EkRXm7+VNwUmc=; b=WTmM33Sgrtnb/d+CqglzfLRS1k9wcRMxSN+zWu+NLrm4LLPloXhpvW2gl2fYqonX0I jYegxzXYNGiqhBIO4TKP2+pEyNQ6mJ6pnE9qb8WpwZueh75QeU606NUGKhW2fS2q1wnK oWhi3iVniSjn2gNpDFREiXM6ihwHxLoleSKLbPjbByj119Zc6TkVhoXUiC/YinPPgj22 VODn9iBIp2+Czb6eeGsKQvRwATMUYfXf4DxFQe/S7Z65u2m3GGhfQn5+eTdeV3lj6G8E BWf5Ph42clqc6VKcr4ZVh1bKHHf96Zr/DB+84IAe0MFp87SksRHYpe6zU19lj0k4pfz0 RS9g== X-Gm-Message-State: AOAM533qGDNkR/pIC8vd1O8mN0RIvsfcLbdZAxKXSr+8wdXl/wyBPmjk Z76I+03uObHXwxE/n+LRcufGTQ== X-Received: by 2002:adf:f041:: with SMTP id t1mr12188827wro.139.1605186208754; Thu, 12 Nov 2020 05:03:28 -0800 (PST) Received: from localhost.localdomain ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id m22sm6877508wrb.97.2020.11.12.05.03.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Nov 2020 05:03:28 -0800 (PST) From: Jean-Philippe Brucker To: joro@8bytes.org, will@kernel.org, lorenzo.pieralisi@arm.com, robh+dt@kernel.org Cc: guohanjun@huawei.com, sudeep.holla@arm.com, rjw@rjwysocki.net, lenb@kernel.org, robin.murphy@arm.com, bhelgaas@google.com, Jonathan.Cameron@huawei.com, eric.auger@redhat.com, iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, baolu.lu@linux.intel.com, zhangfei.gao@linaro.org, shameerali.kolothum.thodi@huawei.com, vivek.gautam@arm.com, Jean-Philippe Brucker Subject: [PATCH v8 9/9] iommu/arm-smmu-v3: Add support for PRI Date: Thu, 12 Nov 2020 13:55:21 +0100 Message-Id: <20201112125519.3987595-10-jean-philippe@linaro.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201112125519.3987595-1-jean-philippe@linaro.org> References: <20201112125519.3987595-1-jean-philippe@linaro.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org For PCI devices that support it, enable the PRI capability and handle PRI Page Requests with the generic fault handler. It is enabled on demand by iommu_sva_device_init(). Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 20 +- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 28 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 275 +++++++++++++++--- 3 files changed, 272 insertions(+), 51 deletions(-) -- 2.29.1 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h index 124f604ed677..7c2d31133148 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -230,6 +230,7 @@ #define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4) #define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6) +#define STRTAB_STE_1_PPAR (1UL << 18) #define STRTAB_STE_1_S1STALLD (1UL << 27) #define STRTAB_STE_1_EATS GENMASK_ULL(29, 28) @@ -360,6 +361,9 @@ #define CMDQ_PRI_0_SID GENMASK_ULL(63, 32) #define CMDQ_PRI_1_GRPID GENMASK_ULL(8, 0) #define CMDQ_PRI_1_RESP GENMASK_ULL(13, 12) +#define CMDQ_PRI_1_RESP_FAILURE 0UL +#define CMDQ_PRI_1_RESP_INVALID 1UL +#define CMDQ_PRI_1_RESP_SUCCESS 2UL #define CMDQ_RESUME_0_SID GENMASK_ULL(63, 32) #define CMDQ_RESUME_0_RESP_TERM 0UL @@ -427,12 +431,6 @@ #define MSI_IOVA_BASE 0x8000000 #define MSI_IOVA_LENGTH 0x100000 -enum pri_resp { - PRI_RESP_DENY = 0, - PRI_RESP_FAIL = 1, - PRI_RESP_SUCC = 2, -}; - struct arm_smmu_cmdq_ent { /* Common fields */ u8 opcode; @@ -494,7 +492,7 @@ struct arm_smmu_cmdq_ent { u32 sid; u32 ssid; u16 grpid; - enum pri_resp resp; + u8 resp; } pri; #define CMDQ_OP_RESUME 0x44 @@ -568,6 +566,9 @@ struct arm_smmu_evtq { struct arm_smmu_priq { struct arm_smmu_queue q; + struct iopf_queue *iopf; + u64 batch; + wait_queue_head_t wq; }; /* High-level stream table and context descriptor structures */ @@ -703,6 +704,8 @@ struct arm_smmu_master { unsigned int num_streams; bool ats_enabled; bool stall_enabled; + bool pri_supported; + bool prg_resp_needs_ssid; bool sva_enabled; struct list_head bonds; unsigned int ssid_bits; @@ -754,6 +757,9 @@ void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid); bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd); int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid, unsigned long iova, size_t size); +int arm_smmu_enable_pri(struct arm_smmu_master *master); +void arm_smmu_disable_pri(struct arm_smmu_master *master); +int arm_smmu_flush_priq(struct arm_smmu_device *smmu); #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c index 64e2082ef9ed..1fdfd40f70fd 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c @@ -370,6 +370,19 @@ arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdata) void arm_smmu_sva_unbind(struct iommu_sva *handle) { struct arm_smmu_bond *bond = sva_to_bond(handle); + struct arm_smmu_master *master = dev_iommu_priv_get(handle->dev); + + /* + * For stall, the event queue does not need to be flushed since the + * device driver ensured all transaction are complete. For PRI however, + * although the device driver has stopped all DMA for this PASID, it may + * have left Page Requests in flight (if using the Stop Marker Message + * to stop PASID). Complete them. + */ + if (master->pri_supported) { + arm_smmu_flush_priq(master->smmu); + iopf_queue_flush_dev(handle->dev); + } mutex_lock(&sva_lock); if (refcount_dec_and_test(&bond->refs)) { @@ -435,7 +448,7 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu) static bool arm_smmu_iopf_supported(struct arm_smmu_master *master) { - return master->stall_enabled; + return master->stall_enabled || master->pri_supported; } bool arm_smmu_master_sva_supported(struct arm_smmu_master *master) @@ -466,6 +479,15 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master) ret = iopf_queue_add_device(master->smmu->evtq.iopf, dev); if (ret) return ret; + } else if (master->pri_supported) { + ret = iopf_queue_add_device(master->smmu->priq.iopf, dev); + if (ret) + return ret; + + if (arm_smmu_enable_pri(master)) { + iopf_queue_remove_device(master->smmu->priq.iopf, dev); + return ret; + } } ret = iommu_register_device_fault_handler(dev, iommu_queue_iopf, dev); @@ -479,6 +501,8 @@ int arm_smmu_master_enable_sva(struct arm_smmu_master *master) return 0; err_disable_iopf: + arm_smmu_disable_pri(master); + iopf_queue_remove_device(master->smmu->priq.iopf, dev); iopf_queue_remove_device(master->smmu->evtq.iopf, dev); return ret; } @@ -497,6 +521,8 @@ int arm_smmu_master_disable_sva(struct arm_smmu_master *master) mutex_unlock(&sva_lock); iommu_unregister_device_fault_handler(dev); + arm_smmu_disable_pri(master); + iopf_queue_remove_device(master->smmu->priq.iopf, dev); iopf_queue_remove_device(master->smmu->evtq.iopf, dev); return 0; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index d412d063d3b6..b4d58f3a77bc 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -315,14 +315,6 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) cmd[0] |= FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid); cmd[0] |= FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid); cmd[1] |= FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid); - switch (ent->pri.resp) { - case PRI_RESP_DENY: - case PRI_RESP_FAIL: - case PRI_RESP_SUCC: - break; - default: - return -EINVAL; - } cmd[1] |= FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); break; case CMDQ_OP_RESUME: @@ -894,11 +886,12 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, } static int arm_smmu_page_response(struct device *dev, - struct iommu_fault_event *unused, + struct iommu_fault_event *evt, struct iommu_page_response *resp) { struct arm_smmu_cmdq_ent cmd = {0}; struct arm_smmu_master *master = dev_iommu_priv_get(dev); + bool pasid_valid = resp->flags & IOMMU_PAGE_RESP_PASID_VALID; int sid = master->streams[0].id; if (master->stall_enabled) { @@ -916,6 +909,28 @@ static int arm_smmu_page_response(struct device *dev, default: return -EINVAL; } + } else if (master->pri_supported) { + bool needs_pasid = (evt->fault.prm.flags & + IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID); + + cmd.opcode = CMDQ_OP_PRI_RESP; + cmd.substream_valid = needs_pasid && pasid_valid; + cmd.pri.sid = sid; + cmd.pri.ssid = resp->pasid; + cmd.pri.grpid = resp->grpid; + switch (resp->code) { + case IOMMU_PAGE_RESP_FAILURE: + cmd.pri.resp = CMDQ_PRI_1_RESP_FAILURE; + break; + case IOMMU_PAGE_RESP_INVALID: + cmd.pri.resp = CMDQ_PRI_1_RESP_INVALID; + break; + case IOMMU_PAGE_RESP_SUCCESS: + cmd.pri.resp = CMDQ_PRI_1_RESP_SUCCESS; + break; + default: + return -EINVAL; + } } else { return -ENODEV; } @@ -1336,6 +1351,9 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) | FIELD_PREP(STRTAB_STE_1_STRW, strw)); + if (master->prg_resp_needs_ssid) + dst[1] |= cpu_to_le64(STRTAB_STE_1_PPAR); + if (smmu->features & ARM_SMMU_FEAT_STALLS && !master->stall_enabled) dst[1] |= cpu_to_le64(STRTAB_STE_1_S1STALLD); @@ -1566,64 +1584,155 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) static void arm_smmu_handle_ppr(struct arm_smmu_device *smmu, u64 *evt) { - u32 sid, ssid; - u16 grpid; - bool ssv, last; - - sid = FIELD_GET(PRIQ_0_SID, evt[0]); - ssv = FIELD_GET(PRIQ_0_SSID_V, evt[0]); - ssid = ssv ? FIELD_GET(PRIQ_0_SSID, evt[0]) : 0; - last = FIELD_GET(PRIQ_0_PRG_LAST, evt[0]); - grpid = FIELD_GET(PRIQ_1_PRG_IDX, evt[1]); - - dev_info(smmu->dev, "unexpected PRI request received:\n"); - dev_info(smmu->dev, - "\tsid 0x%08x.0x%05x: [%u%s] %sprivileged %s%s%s access at iova 0x%016llx\n", - sid, ssid, grpid, last ? "L" : "", - evt[0] & PRIQ_0_PERM_PRIV ? "" : "un", - evt[0] & PRIQ_0_PERM_READ ? "R" : "", - evt[0] & PRIQ_0_PERM_WRITE ? "W" : "", - evt[0] & PRIQ_0_PERM_EXEC ? "X" : "", - evt[1] & PRIQ_1_ADDR_MASK); - - if (last) { - struct arm_smmu_cmdq_ent cmd = { - .opcode = CMDQ_OP_PRI_RESP, - .substream_valid = ssv, - .pri = { - .sid = sid, - .ssid = ssid, - .grpid = grpid, - .resp = PRI_RESP_DENY, - }, + bool pasid_valid, last; + struct arm_smmu_master *master; + u32 sid = FIELD_PREP(PRIQ_0_SID, evt[0]); + struct iommu_fault_event fault_evt = { + .fault.type = IOMMU_FAULT_PAGE_REQ, + .fault.prm = { + .grpid = FIELD_GET(PRIQ_1_PRG_IDX, evt[1]), + .addr = evt[1] & PRIQ_1_ADDR_MASK, + }, + }; + struct iommu_fault_page_request *pr = &fault_evt.fault.prm; + + pasid_valid = evt[0] & PRIQ_0_SSID_V; + last = evt[0] & PRIQ_0_PRG_LAST; + + /* Discard Stop PASID marker, it isn't used */ + if (!(evt[0] & (PRIQ_0_PERM_READ | PRIQ_0_PERM_WRITE)) && last) + return; + + if (last) + pr->flags |= IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE; + if (pasid_valid) { + pr->flags |= IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + pr->pasid = FIELD_GET(PRIQ_0_SSID, evt[0]); + } + if (evt[0] & PRIQ_0_PERM_READ) + pr->perm |= IOMMU_FAULT_PERM_READ; + if (evt[0] & PRIQ_0_PERM_WRITE) + pr->perm |= IOMMU_FAULT_PERM_WRITE; + if (evt[0] & PRIQ_0_PERM_EXEC) + pr->perm |= IOMMU_FAULT_PERM_EXEC; + if (evt[0] & PRIQ_0_PERM_PRIV) + pr->perm |= IOMMU_FAULT_PERM_PRIV; + + master = arm_smmu_find_master(smmu, sid); + if (WARN_ON(!master)) + return; + + if (pasid_valid && master->prg_resp_needs_ssid) + pr->flags |= IOMMU_FAULT_PAGE_RESPONSE_NEEDS_PASID; + + if (iommu_report_device_fault(master->dev, &fault_evt)) { + /* + * No handler registered, so subsequent faults won't produce + * better results. Try to disable PRI. + */ + struct iommu_page_response resp = { + .flags = pasid_valid ? + IOMMU_PAGE_RESP_PASID_VALID : 0, + .pasid = pr->pasid, + .grpid = pr->grpid, + .code = IOMMU_PAGE_RESP_FAILURE, }; - arm_smmu_cmdq_issue_cmd(smmu, &cmd); + dev_warn(master->dev, + "PPR 0x%x:0x%llx 0x%x: nobody cared, disabling PRI\n", + pasid_valid ? pr->pasid : 0, pr->addr, pr->perm); + if (last) + arm_smmu_page_response(master->dev, NULL, &resp); } } static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) { + int num_handled = 0; + bool overflow = false; struct arm_smmu_device *smmu = dev; - struct arm_smmu_queue *q = &smmu->priq.q; + struct arm_smmu_priq *priq = &smmu->priq; + struct arm_smmu_queue *q = &priq->q; struct arm_smmu_ll_queue *llq = &q->llq; + size_t queue_size = 1 << llq->max_n_shift; u64 evt[PRIQ_ENT_DWORDS]; + spin_lock(&priq->wq.lock); do { - while (!queue_remove_raw(q, evt)) + while (!queue_remove_raw(q, evt)) { + spin_unlock(&priq->wq.lock); arm_smmu_handle_ppr(smmu, evt); + spin_lock(&priq->wq.lock); + if (++num_handled == queue_size) { + priq->batch++; + wake_up_all_locked(&priq->wq); + num_handled = 0; + } + } - if (queue_sync_prod_in(q) == -EOVERFLOW) + if (queue_sync_prod_in(q) == -EOVERFLOW) { dev_err(smmu->dev, "PRIQ overflow detected -- requests lost\n"); + overflow = true; + } } while (!queue_empty(llq)); /* Sync our overflow flag, as we believe we're up to speed */ llq->cons = Q_OVF(llq->prod) | Q_WRP(llq, llq->cons) | Q_IDX(llq, llq->cons); queue_sync_cons_out(q); + + wake_up_all_locked(&priq->wq); + spin_unlock(&priq->wq.lock); + + /* + * On overflow, the SMMU might have discarded the last PPR in a group. + * There is no way to know more about it, so we have to discard all + * partial faults already queued. + */ + if (overflow) + iopf_queue_discard_partial(priq->iopf); + return IRQ_HANDLED; } +/* + * arm_smmu_flush_priq - wait until all events currently in the queue have been + * consumed. + * + * When unbinding a PASID, ensure there aren't any pending page requests for + * that PASID in the queue. + * + * Wait either that the queue becomes empty or, if new events are continually + * added the queue, that the event queue thread has handled a full batch (where + * one batch corresponds to the queue size). For that we take the batch number + * when entering flush() and wait for the event queue thread to increment it + * twice. Note that we don't handle overflows on q->batch. If it occurs, just + * wait for the queue to become empty. + */ +int arm_smmu_flush_priq(struct arm_smmu_device *smmu) +{ + int ret; + u64 batch; + bool overflow = false; + struct arm_smmu_priq *priq = &smmu->priq; + struct arm_smmu_queue *q = &priq->q; + + spin_lock(&priq->wq.lock); + if (queue_sync_prod_in(q) == -EOVERFLOW) { + dev_err(smmu->dev, "priq overflow detected -- requests lost\n"); + overflow = true; + } + + batch = priq->batch; + ret = wait_event_interruptible_locked(priq->wq, queue_empty(&q->llq) || + priq->batch >= batch + 2); + spin_unlock(&priq->wq.lock); + + if (overflow) + iopf_queue_discard_partial(priq->iopf); + return ret; +} + static int arm_smmu_device_disable(struct arm_smmu_device *smmu); static irqreturn_t arm_smmu_gerror_handler(int irq, void *dev) @@ -2336,6 +2445,73 @@ static void arm_smmu_disable_pasid(struct arm_smmu_master *master) pci_disable_pasid(pdev); } +static int arm_smmu_init_pri(struct arm_smmu_master *master) +{ + struct pci_dev *pdev; + + if (!dev_is_pci(master->dev)) + return -EINVAL; + + if (!(master->smmu->features & ARM_SMMU_FEAT_PRI)) + return 0; + + pdev = to_pci_dev(master->dev); + if (!pci_pri_supported(pdev)) + return 0; + + /* If the device supports PASID and PRI, set STE.PPAR */ + if (master->ssid_bits) + master->prg_resp_needs_ssid = pci_prg_resp_pasid_required(pdev); + + master->pri_supported = true; + return 0; +} + +int arm_smmu_enable_pri(struct arm_smmu_master *master) +{ + int ret; + struct pci_dev *pdev; + /* + * TODO: find a good inflight PPR number. According to the SMMU spec we + * should divide the PRI queue by the number of PRI-capable devices, but + * it's impossible to know about future (probed late or hotplugged) + * devices. So we might miss some PPRs due to queue overflow. + */ + size_t max_inflight_pprs = 16; + + if (!master->pri_supported || !master->ats_enabled) + return -ENODEV; + + pdev = to_pci_dev(master->dev); + + ret = pci_reset_pri(pdev); + if (ret) + return ret; + + ret = pci_enable_pri(pdev, max_inflight_pprs); + if (ret) { + dev_err(master->dev, "cannot enable PRI: %d\n", ret); + return ret; + } + + return 0; +} + +void arm_smmu_disable_pri(struct arm_smmu_master *master) +{ + struct pci_dev *pdev; + + if (!dev_is_pci(master->dev)) + return; + + pdev = to_pci_dev(master->dev); + + if (!pdev->pri_enabled) + return; + + pci_disable_pri(pdev); +} + static void arm_smmu_detach_dev(struct arm_smmu_master *master) { unsigned long flags; @@ -2648,6 +2824,8 @@ static struct iommu_device *arm_smmu_probe_device(struct device *dev) smmu->features & ARM_SMMU_FEAT_STALL_FORCE) master->stall_enabled = true; + arm_smmu_init_pri(master); + return &smmu->iommu; err_free_master: @@ -2666,6 +2844,7 @@ static void arm_smmu_release_device(struct device *dev) master = dev_iommu_priv_get(dev); WARN_ON(arm_smmu_master_sva_enabled(master)); + iopf_queue_remove_device(master->smmu->priq.iopf, dev); iopf_queue_remove_device(master->smmu->evtq.iopf, dev); arm_smmu_detach_dev(master); arm_smmu_disable_pasid(master); @@ -2981,6 +3160,15 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu) if (!(smmu->features & ARM_SMMU_FEAT_PRI)) return 0; + if (sva) { + smmu->priq.iopf = iopf_queue_alloc(dev_name(smmu->dev)); + if (!smmu->priq.iopf) + return -ENOMEM; + } + + init_waitqueue_head(&smmu->priq.wq); + smmu->priq.batch = 0; + return arm_smmu_init_one_queue(smmu, &smmu->priq.q, ARM_SMMU_PRIQ_PROD, ARM_SMMU_PRIQ_CONS, PRIQ_ENT_DWORDS, "priq"); @@ -3944,6 +4132,7 @@ static int arm_smmu_device_remove(struct platform_device *pdev) iommu_device_sysfs_remove(&smmu->iommu); arm_smmu_device_disable(smmu); iopf_queue_free(smmu->evtq.iopf); + iopf_queue_free(smmu->priq.iopf); return 0; }