From patchwork Fri Apr 7 17:12:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671043 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBC0DC6FD1D for ; Fri, 7 Apr 2023 17:12:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229549AbjDGRME (ORCPT ); Fri, 7 Apr 2023 13:12:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229767AbjDGRMD (ORCPT ); Fri, 7 Apr 2023 13:12:03 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60300A5CB; Fri, 7 Apr 2023 10:12:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887522; x=1712423522; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=doVbLk9NItOWliHputOuSYkc3OnaCATg8PMe+fC+ld8=; b=PkoDct51fA3NB7pdDQr+15KGwzOffC/uvIQdHJ4lhErY1X9tz61tGuLK 7pwRMtAMrEE6KGk3OhWzCKkw27wy57vwzh7sLcUbyPqxcVfvq85V48lHO DOjnL0+R4gQGakMkhEEfB4Kzv0LTEgYLh9rPeEEd8ZwdWCIyvI7gp1Uq6 mslWWZv2vxmDhM7shFhqzHzMaOfXORN1vlJVBmxvwPWcSt/cPgTXezm5+ 2qf6Nra2iUqeh9eX1n7if18Jtkxx966LWEndG02eoIx8xocDJXG9JfazZ ipt2NlTXiBS90PoKsXh0i4xduFxnBt6AsM8zsEhbDl5iMNcHJgGDRwfwL Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="323402058" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="323402058" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="717902604" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="717902604" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:01 -0700 Subject: [PATCH 3 01/23] cxl: Export QTG ids from CFMWS to sysfs From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: Ira Weiny , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:00 -0700 Message-ID: <168088752059.1441063.13985364801901477140.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Export the QoS Throttling Group ID from the CXL Fixed Memory Window Structure (CFMWS) under the root decoder sysfs attributes. CXL rev3.0 9.17.1.3 CXL Fixed Memory Window Structure (CFMWS) cxl cli will use this QTG ID to match with the _DSM retrieved QTG ID for a hot-plugged CXL memory device DPA memory range to make sure that the DPA range is under the right CFMWS window. Reviewed-by: Ira Weiny Signed-off-by: Dave Jiang --- v2: - Add explanation commit header (Jonathan) --- Documentation/ABI/testing/sysfs-bus-cxl | 9 +++++++++ drivers/cxl/acpi.c | 3 +++ drivers/cxl/core/port.c | 14 ++++++++++++++ drivers/cxl/cxl.h | 3 +++ 4 files changed, 29 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index 3acf2f17a73f..471ac9a37078 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -309,6 +309,15 @@ Description: (WO) Write a string in the form 'regionZ' to delete that region, provided it is currently idle / not bound to a driver. +What: /sys/bus/cxl/devices/decoderX.Y/qtg_id +Date: Jan, 2023 +KernelVersion: v6.4 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) Shows the QoS Throttling Group ID. The QTG ID for a root + decoder comes from the CFMWS structure of the CEDT. A value of + -1 indicates that no QTG ID was retrieved. The QTG ID is used as + guidance to match against the QTG ID of a hot-plugged device. What: /sys/bus/cxl/devices/regionZ/uuid Date: May, 2022 diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c index 7e1765b09e04..abc24137c291 100644 --- a/drivers/cxl/acpi.c +++ b/drivers/cxl/acpi.c @@ -289,6 +289,9 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, } } } + + cxld->qtg_id = cfmws->qtg_id; + rc = cxl_decoder_add(cxld, target_map); err_xormap: if (rc) diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 8ee6b6e2e2a4..5ec48dddb2f9 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -276,6 +276,16 @@ static ssize_t interleave_ways_show(struct device *dev, static DEVICE_ATTR_RO(interleave_ways); +static ssize_t qtg_id_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct cxl_decoder *cxld = to_cxl_decoder(dev); + + return sysfs_emit(buf, "%d\n", cxld->qtg_id); +} + +static DEVICE_ATTR_RO(qtg_id); + static struct attribute *cxl_decoder_base_attrs[] = { &dev_attr_start.attr, &dev_attr_size.attr, @@ -295,6 +305,7 @@ static struct attribute *cxl_decoder_root_attrs[] = { &dev_attr_cap_type2.attr, &dev_attr_cap_type3.attr, &dev_attr_target_list.attr, + &dev_attr_qtg_id.attr, SET_CXL_REGION_ATTR(create_pmem_region) SET_CXL_REGION_ATTR(create_ram_region) SET_CXL_REGION_ATTR(delete_region) @@ -1649,6 +1660,7 @@ struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port, } atomic_set(&cxlrd->region_id, rc); + cxld->qtg_id = CXL_QTG_ID_INVALID; return cxlrd; } EXPORT_SYMBOL_NS_GPL(cxl_root_decoder_alloc, CXL); @@ -1686,6 +1698,7 @@ struct cxl_switch_decoder *cxl_switch_decoder_alloc(struct cxl_port *port, cxld = &cxlsd->cxld; cxld->dev.type = &cxl_decoder_switch_type; + cxld->qtg_id = CXL_QTG_ID_INVALID; return cxlsd; } EXPORT_SYMBOL_NS_GPL(cxl_switch_decoder_alloc, CXL); @@ -1718,6 +1731,7 @@ struct cxl_endpoint_decoder *cxl_endpoint_decoder_alloc(struct cxl_port *port) } cxld->dev.type = &cxl_decoder_endpoint_type; + cxld->qtg_id = CXL_QTG_ID_INVALID; return cxled; } EXPORT_SYMBOL_NS_GPL(cxl_endpoint_decoder_alloc, CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index f2b0962a552d..cc3309794b45 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -300,6 +300,7 @@ enum cxl_decoder_type { */ #define CXL_DECODER_MAX_INTERLEAVE 16 +#define CXL_QTG_ID_INVALID -1 /** * struct cxl_decoder - Common CXL HDM Decoder Attributes @@ -311,6 +312,7 @@ enum cxl_decoder_type { * @target_type: accelerator vs expander (type2 vs type3) selector * @region: currently assigned region for this decoder * @flags: memory type capabilities and locking + * @qtg_id: QoS Throttling Group ID * @commit: device/decoder-type specific callback to commit settings to hw * @reset: device/decoder-type specific callback to reset hw settings */ @@ -323,6 +325,7 @@ struct cxl_decoder { enum cxl_decoder_type target_type; struct cxl_region *region; unsigned long flags; + int qtg_id; int (*commit)(struct cxl_decoder *cxld); int (*reset)(struct cxl_decoder *cxld); }; From patchwork Fri Apr 7 17:12:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C51D4C77B61 for ; Fri, 7 Apr 2023 17:12:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230002AbjDGRMK (ORCPT ); Fri, 7 Apr 2023 13:12:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229767AbjDGRMJ (ORCPT ); Fri, 7 Apr 2023 13:12:09 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A60789ED1; Fri, 7 Apr 2023 10:12:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887528; x=1712423528; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8MxlJ1+PFYsT4zJD2rLD6eOXsfO1SJ5tDzA0hk12vHw=; b=Z/cJA3R29yUCWMlsIGEnv43NJealSX1eAwdNvdQpVGaPN72nZaKeox5Z efDcZr8+4wUzgYT/ssthxKRAz7rNE3etdBzGWYb4L6H3dhil03JOY4Nd/ HcswJL0q0QpHcw4mFdb9eEYRs8aR80pOWXhn4c6+RfQEFM9ElsFadW6Qp wBOEoD3cElmSRN5KbuLgOvpixVpfW+T5PfhNmeFn0EkX5bVgsxbBWtF0m 22QRo7kfIjf1ZlvLsHMiJ10XJxDKGilAf3E0/B+0/tQJf8k0OOi+cHCr4 KFnm2yxwE2ClVpkRNSELkL4cnjL0tzxGp/YSF/d5VWPrvRkfMkEwZVceL g==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="323402089" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="323402089" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="717902620" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="717902620" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:07 -0700 Subject: [PATCH 3 02/23] cxl: Add checksum verification to CDAT from CXL From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: Ira Weiny , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:06 -0700 Message-ID: <168088752692.1441063.12748870422089681966.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org A CDAT table is available from a CXL device. The table is read by the driver and cached in software. With the CXL subsystem needing to parse the CDAT table, the checksum should be verified. Add checksum verification after the CDAT table is read from device. Reviewed-by: Ira Weiny Signed-off-by: Dave Jiang --- v3: - Just return the final sum. (Alison) v2: - Drop ACPI checksum export and just use local verification. (Dan) --- drivers/cxl/core/pci.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 25b7e8125d5d..9c7e2f69d9ca 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -528,6 +528,16 @@ static int cxl_cdat_read_table(struct device *dev, return 0; } +static unsigned char cdat_checksum(void *buf, size_t size) +{ + unsigned char sum, *data = buf; + size_t i; + + for (sum = 0, i = 0; i < size; i++) + sum += data[i]; + return sum; +} + /** * read_cdat_data - Read the CDAT data on this port * @port: Port to read data from @@ -573,6 +583,12 @@ void read_cdat_data(struct cxl_port *port) } port->cdat.table = cdat_table + sizeof(__le32); + if (cdat_checksum(port->cdat.table, cdat_length)) { + /* Don't leave table data allocated on error */ + devm_kfree(dev, cdat_table); + dev_err(dev, "CDAT data checksum error\n"); + } + port->cdat.length = cdat_length; } EXPORT_SYMBOL_NS_GPL(read_cdat_data, CXL); From patchwork Fri Apr 7 17:12:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671042 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62DEFC77B61 for ; Fri, 7 Apr 2023 17:12:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230010AbjDGRMQ (ORCPT ); Fri, 7 Apr 2023 13:12:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229767AbjDGRMP (ORCPT ); Fri, 7 Apr 2023 13:12:15 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1DB70A5CB; Fri, 7 Apr 2023 10:12:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887535; x=1712423535; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KkQPMKg/m4KPZlcMeM+2KUn0G82HH+sivmcx9DEURSw=; b=HVp8W2XOzODUVqQVCCI2jn9VCoEHgz02Yvv+SCuRvPlSvtViDnvGQgzv fbiRVbNAeWZ4M41eknw74Bh/yyJ8ovXzB2gYxTU+Jro5aD+rkG719B5uG aQhYTMYtUuoDOK4heF8yx5beywGWzCLTZI89AnMRM32XooI0eI61AHhaA KXCxIVa4GfKVVum2WD8dGRxItd8hOh5hnG3P/TYzhPcjXAEgJo4XqrvS7 v+SBRZJrAtGkeIKcICAnb3m2F/nKLUum14QVpAahiORsimB/mo7B7ea9a iEJxF6PwcEpZBrUbRQJ16DnX1q2ncW0ws1bBAU1pkIDbvMrInY0/pzBVB w==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="323402110" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="323402110" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="717902647" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="717902647" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:13 -0700 Subject: [PATCH 3 03/23] cxl: Add support for reading CXL switch CDAT table From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: Ira Weiny , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:13 -0700 Message-ID: <168088753327.1441063.12840367865204790886.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Move read_cdat_data() from endpoint probe to general port probe to allow reading of CDAT data for CXL switches as well as CXL device. Add wrapper support for cxl_test to bypass the cdat reading. Reviewed-by: Ira Weiny Signed-off-by: Dave Jiang --- drivers/cxl/core/pci.c | 20 +++++++++++++++----- drivers/cxl/port.c | 6 +++--- tools/testing/cxl/Kbuild | 1 + tools/testing/cxl/test/mock.c | 5 +++++ 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 9c7e2f69d9ca..1c415b26e866 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -546,16 +546,26 @@ static unsigned char cdat_checksum(void *buf, size_t size) */ void read_cdat_data(struct cxl_port *port) { - struct pci_doe_mb *cdat_doe; - struct device *dev = &port->dev; struct device *uport = port->uport; - struct cxl_memdev *cxlmd = to_cxl_memdev(uport); - struct cxl_dev_state *cxlds = cxlmd->cxlds; - struct pci_dev *pdev = to_pci_dev(cxlds->dev); + struct device *dev = &port->dev; + struct cxl_dev_state *cxlds; + struct pci_doe_mb *cdat_doe; + struct cxl_memdev *cxlmd; + struct pci_dev *pdev; size_t cdat_length; void *cdat_table; int rc; + if (is_cxl_memdev(uport)) { + cxlmd = to_cxl_memdev(uport); + cxlds = cxlmd->cxlds; + pdev = to_pci_dev(cxlds->dev); + } else if (dev_is_pci(uport)) { + pdev = to_pci_dev(uport); + } else { + return; + } + cdat_doe = pci_find_doe_mailbox(pdev, PCI_DVSEC_VENDOR_ID_CXL, CXL_DOE_PROTOCOL_TABLE_ACCESS); if (!cdat_doe) { diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 1049bb5ea496..60a865680e22 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -93,9 +93,6 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) if (IS_ERR(cxlhdm)) return PTR_ERR(cxlhdm); - /* Cache the data early to ensure is_visible() works */ - read_cdat_data(port); - get_device(&cxlmd->dev); rc = devm_add_action_or_reset(&port->dev, schedule_detach, cxlmd); if (rc) @@ -135,6 +132,9 @@ static int cxl_port_probe(struct device *dev) { struct cxl_port *port = to_cxl_port(dev); + /* Cache the data early to ensure is_visible() works */ + read_cdat_data(port); + if (is_cxl_endpoint(port)) return cxl_endpoint_port_probe(port); return cxl_switch_port_probe(port); diff --git a/tools/testing/cxl/Kbuild b/tools/testing/cxl/Kbuild index fba7bec96acd..2637c71f3378 100644 --- a/tools/testing/cxl/Kbuild +++ b/tools/testing/cxl/Kbuild @@ -12,6 +12,7 @@ ldflags-y += --wrap=cxl_await_media_ready ldflags-y += --wrap=cxl_hdm_decode_init ldflags-y += --wrap=cxl_dvsec_rr_decode ldflags-y += --wrap=cxl_rcrb_to_component +ldflags-y += --wrap=read_cdat_data DRIVERS := ../../../drivers CXL_SRC := $(DRIVERS)/cxl diff --git a/tools/testing/cxl/test/mock.c b/tools/testing/cxl/test/mock.c index c4e53f22e421..3a75909b2aae 100644 --- a/tools/testing/cxl/test/mock.c +++ b/tools/testing/cxl/test/mock.c @@ -263,6 +263,11 @@ resource_size_t __wrap_cxl_rcrb_to_component(struct device *dev, } EXPORT_SYMBOL_NS_GPL(__wrap_cxl_rcrb_to_component, CXL); +void __wrap_read_cdat_data(struct cxl_port *port) +{ +} +EXPORT_SYMBOL_NS_GPL(__wrap_read_cdat_data, CXL); + MODULE_LICENSE("GPL v2"); MODULE_IMPORT_NS(ACPI); MODULE_IMPORT_NS(CXL); From patchwork Fri Apr 7 17:12:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99F2FC6FD1D for ; Fri, 7 Apr 2023 17:12:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230015AbjDGRMX (ORCPT ); Fri, 7 Apr 2023 13:12:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229767AbjDGRMW (ORCPT ); Fri, 7 Apr 2023 13:12:22 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 532569ED1; Fri, 7 Apr 2023 10:12:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887541; x=1712423541; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=B9VBVgS4nybtHfZE9p+drAXT3PxU7FrJ0X5i9nulIVQ=; b=ZY3+yEjDX1xRbEVdEdco/6BvFMKbljjmx1zj6YYFeDI3lwmmgIcuFnEw di7eSkmXiP8bLQBjFvqXG/dCWR6SnH71AmAM5bHa4X9lHFdfdc2sYxHrT usFlzX/mtgpHjdzkvm4oy1VAz8yOyZrt6rD0U7AJ1kPvfAdCLvjpj6rpU WaJNHYsZ7K/r4mSmeGWKWpEZG2gfU12m5O83vhmmlwOICOgS03hdhfSAc VPLpqHh+4VqIDwC5oFbPmZ7Q9BFKt/UiChtrd5gkeA1M9Ukq7l5PBKrru tcZnWXo1c3bSbDUaUR0lL/WbQ3x2n1V5P7m4FM0jxzXJjmigB6hkbSktm g==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="323402135" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="323402135" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="717902655" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="717902655" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:20 -0700 Subject: [PATCH 3 04/23] cxl: Add common helpers for cdat parsing From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:19 -0700 Message-ID: <168088753960.1441063.5248646258780465828.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Add helper functions to parse the CDAT table and provide a callback to parse the sub-table. Helpers are provided for DSMAS and DSLBIS sub-table parsing. The code is patterned after the ACPI table parsing helpers. Signed-off-by: Dave Jiang --- v2: - Use local headers to handle LE instead of ACPI header - Reduce complexity of parser function. (Jonathan) - Directly access header type. (Jonathan) - Simplify header ptr math. (Jonathan) - Move parsed counter to the correct location. (Jonathan) - Add LE to host conversion for entry length --- drivers/cxl/core/Makefile | 1 drivers/cxl/core/cdat.c | 100 +++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxlpci.h | 29 +++++++++++++ 3 files changed, 130 insertions(+) create mode 100644 drivers/cxl/core/cdat.c diff --git a/drivers/cxl/core/Makefile b/drivers/cxl/core/Makefile index ca4ae31d8f57..867a8014b462 100644 --- a/drivers/cxl/core/Makefile +++ b/drivers/cxl/core/Makefile @@ -12,5 +12,6 @@ cxl_core-y += memdev.o cxl_core-y += mbox.o cxl_core-y += pci.o cxl_core-y += hdm.o +cxl_core-y += cdat.o cxl_core-$(CONFIG_TRACING) += trace.o cxl_core-$(CONFIG_CXL_REGION) += region.o diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c new file mode 100644 index 000000000000..210f4499bddb --- /dev/null +++ b/drivers/cxl/core/cdat.c @@ -0,0 +1,100 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2023 Intel Corporation. All rights reserved. */ +#include "cxlpci.h" +#include "cxl.h" + +static bool has_handler(struct cdat_subtable_proc *proc) +{ + return proc->handler; +} + +static int call_handler(struct cdat_subtable_proc *proc, + struct cdat_subtable_entry *ent) +{ + if (has_handler(proc)) + return proc->handler(ent->hdr, proc->arg); + return -EINVAL; +} + +static bool cdat_is_subtable_match(struct cdat_subtable_entry *ent) +{ + return ent->hdr->type == ent->type; +} + +static int cdat_table_parse_entries(enum cdat_type type, + struct cdat_header *table_header, + struct cdat_subtable_proc *proc) +{ + unsigned long table_end, entry_len; + struct cdat_subtable_entry entry; + int count = 0; + int rc; + + if (!has_handler(proc)) + return -EINVAL; + + table_end = (unsigned long)table_header + table_header->length; + + if (type >= CDAT_TYPE_RESERVED) + return -EINVAL; + + entry.type = type; + entry.hdr = (struct cdat_entry_header *)(table_header + 1); + + while ((unsigned long)entry.hdr < table_end) { + entry_len = le16_to_cpu(entry.hdr->length); + + if ((unsigned long)entry.hdr + entry_len > table_end) + return -EINVAL; + + if (entry_len == 0) + return -EINVAL; + + if (cdat_is_subtable_match(&entry)) { + rc = call_handler(proc, &entry); + if (rc) + return rc; + count++; + } + + entry.hdr = (struct cdat_entry_header *)((unsigned long)entry.hdr + entry_len); + } + + return count; +} + +int cdat_table_parse_dsmas(struct cdat_header *table, + cdat_tbl_entry_handler handler, void *arg) +{ + struct cdat_subtable_proc proc = { + .handler = handler, + .arg = arg, + }; + + return cdat_table_parse_entries(CDAT_TYPE_DSMAS, table, &proc); +} +EXPORT_SYMBOL_NS_GPL(cdat_table_parse_dsmas, CXL); + +int cdat_table_parse_dslbis(struct cdat_header *table, + cdat_tbl_entry_handler handler, void *arg) +{ + struct cdat_subtable_proc proc = { + .handler = handler, + .arg = arg, + }; + + return cdat_table_parse_entries(CDAT_TYPE_DSLBIS, table, &proc); +} +EXPORT_SYMBOL_NS_GPL(cdat_table_parse_dslbis, CXL); + +int cdat_table_parse_sslbis(struct cdat_header *table, + cdat_tbl_entry_handler handler, void *arg) +{ + struct cdat_subtable_proc proc = { + .handler = handler, + .arg = arg, + }; + + return cdat_table_parse_entries(CDAT_TYPE_SSLBIS, table, &proc); +} +EXPORT_SYMBOL_NS_GPL(cdat_table_parse_sslbis, CXL); diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 0465ef963cd6..45e2f2bf5ef8 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -76,12 +76,34 @@ struct cdat_header { __le32 sequence; } __packed; +enum cdat_type { + CDAT_TYPE_DSMAS = 0, + CDAT_TYPE_DSLBIS, + CDAT_TYPE_DSMSCIS, + CDAT_TYPE_DSIS, + CDAT_TYPE_DSEMTS, + CDAT_TYPE_SSLBIS, + CDAT_TYPE_RESERVED +}; + struct cdat_entry_header { u8 type; u8 reserved; __le16 length; } __packed; +typedef int (*cdat_tbl_entry_handler)(struct cdat_entry_header *header, void *arg); + +struct cdat_subtable_proc { + cdat_tbl_entry_handler handler; + void *arg; +}; + +struct cdat_subtable_entry { + struct cdat_entry_header *hdr; + enum cdat_type type; +}; + int devm_cxl_port_enumerate_dports(struct cxl_port *port); struct cxl_dev_state; int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, @@ -90,4 +112,11 @@ void read_cdat_data(struct cxl_port *port); void cxl_cor_error_detected(struct pci_dev *pdev); pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, pci_channel_state_t state); + +#define cdat_table_parse(x) \ +int cdat_table_parse_##x(struct cdat_header *table, \ + cdat_tbl_entry_handler handler, void *arg) +cdat_table_parse(dsmas); +cdat_table_parse(dslbis); +cdat_table_parse(sslbis); #endif /* __CXL_PCI_H__ */ From patchwork Fri Apr 7 17:12:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56B7BC77B6C for ; Fri, 7 Apr 2023 17:12:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230039AbjDGRMa (ORCPT ); Fri, 7 Apr 2023 13:12:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48094 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230035AbjDGRM3 (ORCPT ); Fri, 7 Apr 2023 13:12:29 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAA5F9ED1; Fri, 7 Apr 2023 10:12:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887548; x=1712423548; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=s8rvJ7QszvsHUKLtT53f6fv6GKMN8X/kehNdsili22E=; b=hMmBvo+dxlsmuYKzj4ifGXZXN99z9t+bH3CNFfqyqAKSkj8EcT940YCp iLwlcEGqZw/4bVoW7v25IueNplhpNA+YdpRW6lzuENvR1CE1IgCjpT/Qc Cc1qmEPBLmX28vuKLog1QHXfpRqjOdo3eDfWmopl3+vxVWvXhw4VvtF1Z xDKsLU3nbTvvqxMaQ11jSi/kGuUypg83dgdYMJ+4tX+FD5WZFUhHzYbpt flh73IQ4UrHQImlv0cAhFJ6u5p+h7XB4FdW7FSVewT9tcQFbsDo6YPqTK d8nCWH44wpCQ4SbmtaELy1Hlq7HB51WJBcY0n9YohFcS0qrlkjAfHEYdS w==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="331684728" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="331684728" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="681086965" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="681086965" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:26 -0700 Subject: [PATCH 3 05/23] cxl: Add callback to parse the DSMAS subtables from CDAT From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:25 -0700 Message-ID: <168088754594.1441063.15913679905112812064.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Provide a callback function to the CDAT parser in order to parse the Device Scoped Memory Affinity Structure (DSMAS). Each DSMAS structure contains the DPA range and its associated attributes in each entry. See the CDAT specification for details. Coherent Device Attribute Table 1.03 2.1 Device Scoped memory Affinity Structure (DSMAS) Signed-off-by: Dave Jiang --- v3: - Add spec section number. (Alison) - Remove cast from void *. (Alison) - Refactor cxl_port_probe() block. (Alison) - Move CDAT parse to cxl_endpoint_port_probe() v2: - Add DSMAS table size check. (Lukas) - Use local DSMAS header for LE handling. - Remove dsmas lock. (Jonathan) - Fix handle size (Jonathan) - Add LE to host conversion for DSMAS address and length. - Make dsmas_list local --- drivers/cxl/core/cdat.c | 26 ++++++++++++++++++++++++++ drivers/cxl/cxl.h | 1 + drivers/cxl/cxlpci.h | 18 ++++++++++++++++++ drivers/cxl/port.c | 22 ++++++++++++++++++++++ 4 files changed, 67 insertions(+) diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c index 210f4499bddb..6f20af83a3ed 100644 --- a/drivers/cxl/core/cdat.c +++ b/drivers/cxl/core/cdat.c @@ -98,3 +98,29 @@ int cdat_table_parse_sslbis(struct cdat_header *table, return cdat_table_parse_entries(CDAT_TYPE_SSLBIS, table, &proc); } EXPORT_SYMBOL_NS_GPL(cdat_table_parse_sslbis, CXL); + +int cxl_dsmas_parse_entry(struct cdat_entry_header *header, void *arg) +{ + struct cdat_dsmas *dsmas = (struct cdat_dsmas *)header; + struct list_head *dsmas_list = arg; + struct dsmas_entry *dent; + + if (dsmas->hdr.length != sizeof(*dsmas)) { + pr_warn("Malformed DSMAS table length: (%lu:%u)\n", + (unsigned long)sizeof(*dsmas), dsmas->hdr.length); + return -EINVAL; + } + + dent = kzalloc(sizeof(*dent), GFP_KERNEL); + if (!dent) + return -ENOMEM; + + dent->handle = dsmas->dsmad_handle; + dent->dpa_range.start = le64_to_cpu(dsmas->dpa_base_address); + dent->dpa_range.end = le64_to_cpu(dsmas->dpa_base_address) + + le64_to_cpu(dsmas->dpa_length) - 1; + list_add_tail(&dent->list, dsmas_list); + + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_dsmas_parse_entry, CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index cc3309794b45..9d0e22fe72c0 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -8,6 +8,7 @@ #include #include #include +#include #include /** diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 45e2f2bf5ef8..9a2468a93d83 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -104,6 +104,22 @@ struct cdat_subtable_entry { enum cdat_type type; }; +struct dsmas_entry { + struct list_head list; + struct range dpa_range; + u8 handle; +}; + +/* Sub-table 0: Device Scoped Memory Affinity Structure (DSMAS) */ +struct cdat_dsmas { + struct cdat_entry_header hdr; + u8 dsmad_handle; + u8 flags; + __u16 reserved; + __le64 dpa_base_address; + __le64 dpa_length; +} __packed; + int devm_cxl_port_enumerate_dports(struct cxl_port *port); struct cxl_dev_state; int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, @@ -119,4 +135,6 @@ int cdat_table_parse_##x(struct cdat_header *table, \ cdat_table_parse(dsmas); cdat_table_parse(dslbis); cdat_table_parse(sslbis); + +int cxl_dsmas_parse_entry(struct cdat_entry_header *header, void *arg); #endif /* __CXL_PCI_H__ */ diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 60a865680e22..ec9051f1dd0f 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -57,6 +57,16 @@ static int discover_region(struct device *dev, void *root) return 0; } +static void dsmas_list_destroy(struct list_head *dsmas_list) +{ + struct dsmas_entry *dentry, *n; + + list_for_each_entry_safe(dentry, n, dsmas_list, list) { + list_del(&dentry->list); + kfree(dentry); + } +} + static int cxl_switch_port_probe(struct cxl_port *port) { struct cxl_hdm *cxlhdm; @@ -125,6 +135,18 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) device_for_each_child(&port->dev, root, discover_region); put_device(&root->dev); + if (port->cdat.table) { + LIST_HEAD(dsmas_list); + + rc = cdat_table_parse_dsmas(port->cdat.table, + cxl_dsmas_parse_entry, + (void *)&dsmas_list); + if (rc < 0) + dev_warn(&port->dev, "Failed to parse DSMAS: %d\n", rc); + + dsmas_list_destroy(&dsmas_list); + } + return 0; } From patchwork Fri Apr 7 17:12:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 358DEC77B61 for ; Fri, 7 Apr 2023 17:12:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230044AbjDGRMh (ORCPT ); Fri, 7 Apr 2023 13:12:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230048AbjDGRMe (ORCPT ); Fri, 7 Apr 2023 13:12:34 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F8E0AD32; Fri, 7 Apr 2023 10:12:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887553; x=1712423553; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dKdXj5i+OJHFV56/MwG69nLfaS3ZF3t5HAH+8I0B8ec=; b=E4dzC/3Yv6aWnpg7GdpHOlNjUoKxxqe/mO5y86of8QYA+Q99iO/dwoVJ 1jrBg1rcoZX/kHdwQfSnpoAjETpSsDTDuhCTesCXBVPDb8GbE4HHRxZlb OqzB4Us3XcxJjx0SckY8tIH2ff403FNWDFD08icWiDCgVQLTmMlfSxCN7 Ysm9gltgSTagi/5PqhK+pteYxgTi1J4Vk6lzSOf50W4lvBMDJff+cDojc Kh/TU1rLyGFGcumyEVyyDbgtrCZg3KNVq1Uq6a6DOyqJ5fsNHp/3cNCFR xyoHMb2WO8olXN4PiVCjHhHW5B2b1gPbi0hsTlDFkkhVetp/PuxPwIhqj A==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="331684753" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="331684753" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="681086976" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="681086976" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:32 -0700 Subject: [PATCH 3 06/23] cxl: Add callback to parse the DSLBIS subtable from CDAT From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:32 -0700 Message-ID: <168088755202.1441063.7051663951287310170.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Provide a callback to parse the Device Scoped Latency and Bandwidth Information Structure (DSLBIS) in the CDAT structures. The DSLBIS contains the bandwidth and latency information that's tied to a DSMAS handle. The driver will retrieve the read and write latency and bandwidth associated with the DSMAS which is tied to a DPA range. Coherent Device Attribute Table 1.03 2.1 Device Scoped Latency and Bandwidth Information Structure (DSLBIS) Signed-off-by: Dave Jiang --- v3: - Added spec section in commit header. (Alison) - Remove void * recast. (Alison) - Rework comment. (Alison) - Move CDAT parse to cxl_endpoint_port_probe() - Convert to use 'struct node_hmem_attrs' v2: - Add size check to DSLIBIS table. (Lukas) - Remove unnecessary entry type check. (Jonathan) - Move data_type check to after match. (Jonathan) - Skip unknown data type. (Jonathan) - Add overflow check for unit multiply. (Jonathan) - Use dev_warn() when entries parsing fail. (Jonathan) --- drivers/cxl/core/cdat.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxlpci.h | 34 +++++++++++++++++++++++- drivers/cxl/port.c | 11 +++++++- 3 files changed, 111 insertions(+), 2 deletions(-) diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c index 6f20af83a3ed..e8b9bb99dfdf 100644 --- a/drivers/cxl/core/cdat.c +++ b/drivers/cxl/core/cdat.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2023 Intel Corporation. All rights reserved. */ +#include #include "cxlpci.h" #include "cxl.h" @@ -124,3 +125,70 @@ int cxl_dsmas_parse_entry(struct cdat_entry_header *header, void *arg) return 0; } EXPORT_SYMBOL_NS_GPL(cxl_dsmas_parse_entry, CXL); + +static void cxl_hmem_attrs_set(struct node_hmem_attrs *attrs, + int access, unsigned int val) +{ + switch (access) { + case HMAT_SLLBIS_ACCESS_LATENCY: + attrs->read_latency = val; + attrs->write_latency = val; + break; + case HMAT_SLLBIS_READ_LATENCY: + attrs->read_latency = val; + break; + case HMAT_SLLBIS_WRITE_LATENCY: + attrs->write_latency = val; + break; + case HMAT_SLLBIS_ACCESS_BANDWIDTH: + attrs->read_bandwidth = val; + attrs->write_bandwidth = val; + break; + case HMAT_SLLBIS_READ_BANDWIDTH: + attrs->read_bandwidth = val; + break; + case HMAT_SLLBIS_WRITE_BANDWIDTH: + attrs->write_bandwidth = val; + break; + } +} + +int cxl_dslbis_parse_entry(struct cdat_entry_header *header, void *arg) +{ + struct cdat_dslbis *dslbis = (struct cdat_dslbis *)header; + struct list_head *dsmas_list = arg; + struct dsmas_entry *dent; + + if (dslbis->hdr.length != sizeof(*dslbis)) { + pr_warn("Malformed DSLBIS table length: (%lu:%u)\n", + (unsigned long)sizeof(*dslbis), dslbis->hdr.length); + return -EINVAL; + } + + /* Skip unrecognized data type */ + if (dslbis->data_type >= HMAT_SLLBIS_DATA_TYPE_MAX) + return 0; + + list_for_each_entry(dent, dsmas_list, list) { + u64 val; + int rc; + + if (dslbis->handle != dent->handle) + continue; + + /* Not a memory type, skip */ + if ((dslbis->flags & DSLBIS_MEM_MASK) != DSLBIS_MEM_MEMORY) + return 0; + + rc = check_mul_overflow(le64_to_cpu(dslbis->entry_base_unit), + le16_to_cpu(dslbis->entry[0]), &val); + if (unlikely(rc)) + pr_warn("DSLBIS value overflowed.\n"); + + cxl_hmem_attrs_set(&dent->hmem_attrs, dslbis->data_type, val); + break; + } + + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_dslbis_parse_entry, CXL); diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 9a2468a93d83..0f36fb23055c 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -2,6 +2,7 @@ /* Copyright(c) 2020 Intel Corporation. All rights reserved. */ #ifndef __CXL_PCI_H__ #define __CXL_PCI_H__ +#include #include #include "cxl.h" @@ -104,10 +105,21 @@ struct cdat_subtable_entry { enum cdat_type type; }; +enum { + HMAT_SLLBIS_ACCESS_LATENCY = 0, + HMAT_SLLBIS_READ_LATENCY, + HMAT_SLLBIS_WRITE_LATENCY, + HMAT_SLLBIS_ACCESS_BANDWIDTH, + HMAT_SLLBIS_READ_BANDWIDTH, + HMAT_SLLBIS_WRITE_BANDWIDTH, + HMAT_SLLBIS_DATA_TYPE_MAX +}; + struct dsmas_entry { struct list_head list; struct range dpa_range; u8 handle; + struct node_hmem_attrs hmem_attrs; }; /* Sub-table 0: Device Scoped Memory Affinity Structure (DSMAS) */ @@ -120,6 +132,22 @@ struct cdat_dsmas { __le64 dpa_length; } __packed; +/* Sub-table 1: Device Scoped Latency and Bandwidth Information Structure (DSLBIS) */ +struct cdat_dslbis { + struct cdat_entry_header hdr; + u8 handle; + u8 flags; + u8 data_type; + u8 reserved; + __le64 entry_base_unit; + __le16 entry[3]; + __le16 reserved2; +} __packed; + +/* Flags for DSLBIS subtable */ +#define DSLBIS_MEM_MASK GENMASK(3, 0) +#define DSLBIS_MEM_MEMORY 0 + int devm_cxl_port_enumerate_dports(struct cxl_port *port); struct cxl_dev_state; int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, @@ -136,5 +164,9 @@ cdat_table_parse(dsmas); cdat_table_parse(dslbis); cdat_table_parse(sslbis); -int cxl_dsmas_parse_entry(struct cdat_entry_header *header, void *arg); +#define cxl_parse_entry(x) \ +int cxl_##x##_parse_entry(struct cdat_entry_header *header, void *arg) + +cxl_parse_entry(dsmas); +cxl_parse_entry(dslbis); #endif /* __CXL_PCI_H__ */ diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index ec9051f1dd0f..36e32f2e2ca9 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -141,8 +141,17 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) rc = cdat_table_parse_dsmas(port->cdat.table, cxl_dsmas_parse_entry, (void *)&dsmas_list); - if (rc < 0) + if (rc > 0) { + rc = cdat_table_parse_dslbis(port->cdat.table, + cxl_dslbis_parse_entry, + (void *)&dsmas_list); + if (rc <= 0) { + dev_warn(&port->dev, + "Failed to parse DSLBIS: %d\n", rc); + } + } else { dev_warn(&port->dev, "Failed to parse DSMAS: %d\n", rc); + } dsmas_list_destroy(&dsmas_list); } From patchwork Fri Apr 7 17:12:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CED3C77B6F for ; Fri, 7 Apr 2023 17:12:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230048AbjDGRMn (ORCPT ); Fri, 7 Apr 2023 13:12:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230058AbjDGRMk (ORCPT ); Fri, 7 Apr 2023 13:12:40 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C2149ED1; Fri, 7 Apr 2023 10:12:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887559; x=1712423559; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=s/5q7hD3kDo/MhZZY11ggYZY7tBQkcA4hXAh+76w5QI=; b=XsXszAAYYogqAiO0isa/DaK7lC9iJgyLGNi0fgk65wGT4fALutzE5mk+ L+Zucz4uU8Egku2z6m3ASnMRXExPK56Kkm8VSb39UGLgKqQXs9aZibYaw HLyfvI65uJFd5j9jmbKBeCBoZ4WzdoLAufCpc6c8/tf0EvpK2nboT1yW/ pVL4aHCkN3HH/LJ0zULh8pSvCHJ5A3uprHHlF5g3SST6gJFA1uXXo5ZBE utFCQEyLLBWFNsJHpTmnX9WYikrHHPvVLVR07qDf2SbQ0Y5C4o3bZe1Pe u1cL9SEnx+WjX5BaiVV0e2V1lZwZizmP6RqqPn/bQxV8qp6oSFYzGJdVE A==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="331684771" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="331684771" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="681086984" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="681086984" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:38 -0700 Subject: [PATCH 3 07/23] cxl: Add callback to parse the SSLBIS subtable from CDAT From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:38 -0700 Message-ID: <168088755812.1441063.16029524574559629006.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Provide a callback to parse the Switched Scoped Latency and Bandwidth Information Structure (DSLBIS) in the CDAT structures. The SSLBIS contains the bandwidth and latency information that's tied to the CLX switch that the data table has been read from. The extracted values are indexed by the downstream port id. It is possible the downstream port id is 0xffff which is a wildcard value for any port id. Coherent Device Attribute Table 1.03 2.1 Switched Scoped Latency and Bandwidth Information Structure (DSLBIS) Signed-off-by: Dave Jiang --- v3: - Add spec section in commit header (Alison) - Move CDAT parse to cxl_switch_port_probe() - Use 'struct node_hmem_attrs' --- drivers/cxl/core/cdat.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/core/port.c | 5 +++ drivers/cxl/cxl.h | 1 + drivers/cxl/cxlpci.h | 20 ++++++++++++ drivers/cxl/port.c | 14 ++++++++- 5 files changed, 115 insertions(+), 1 deletion(-) diff --git a/drivers/cxl/core/cdat.c b/drivers/cxl/core/cdat.c index e8b9bb99dfdf..ec3420dddf27 100644 --- a/drivers/cxl/core/cdat.c +++ b/drivers/cxl/core/cdat.c @@ -192,3 +192,79 @@ int cxl_dslbis_parse_entry(struct cdat_entry_header *header, void *arg) return 0; } EXPORT_SYMBOL_NS_GPL(cxl_dslbis_parse_entry, CXL); + +int cxl_sslbis_parse_entry(struct cdat_entry_header *header, void *arg) +{ + struct cdat_sslbis *sslbis = (struct cdat_sslbis *)header; + struct xarray *sslbis_xa = arg; + int remain, entries, i; + + remain = sslbis->hdr.length - sizeof(*sslbis); + if (!remain || remain % sizeof(struct sslbis_sslbe)) { + pr_warn("Malformed SSLBIS table length: (%u)\n", + sslbis->hdr.length); + return -EINVAL; + } + + /* Unrecognized data type, we can skip */ + if (sslbis->data_type >= HMAT_SLLBIS_DATA_TYPE_MAX) + return 0; + + entries = remain / sizeof(*sslbis); + + for (i = 0; i < entries; i++) { + struct sslbis_sslbe *sslbe = &sslbis->sslbe[i]; + u16 x = le16_to_cpu(sslbe->port_x_id); + u16 y = le16_to_cpu(sslbe->port_y_id); + struct node_hmem_attrs *hmem_attrs; + u16 dsp_id; + u64 val; + int rc; + + switch (x) { + case SSLBIS_US_PORT: + dsp_id = y; + break; + case SSLBIS_ANY_PORT: + switch (y) { + case SSLBIS_US_PORT: + dsp_id = x; + break; + case SSLBIS_ANY_PORT: + dsp_id = SSLBIS_ANY_PORT; + break; + default: + dsp_id = y; + break; + } + break; + default: + dsp_id = x; + break; + } + + hmem_attrs = xa_load(sslbis_xa, dsp_id); + if (xa_is_err(hmem_attrs)) + return xa_err(hmem_attrs); + if (!hmem_attrs) { + hmem_attrs = kzalloc(sizeof(*hmem_attrs), GFP_KERNEL); + if (!hmem_attrs) + return -ENOMEM; + } + + rc = check_mul_overflow(le64_to_cpu(sslbis->entry_base_unit), + le16_to_cpu(sslbe->value), &val); + if (unlikely(rc)) + pr_warn("SSLBIS value overflowed!\n"); + + cxl_hmem_attrs_set(hmem_attrs, sslbis->data_type, val); + rc = xa_insert(sslbis_xa, dsp_id, hmem_attrs, GFP_KERNEL); + if (rc < 0 && rc != -EBUSY) { + kfree(hmem_attrs); + return rc; + } + } + + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_sslbis_parse_entry, CXL); diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 5ec48dddb2f9..8febb0a2d116 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -518,6 +518,7 @@ static void cxl_ep_remove(struct cxl_port *port, struct cxl_ep *ep) static void cxl_port_release(struct device *dev) { struct cxl_port *port = to_cxl_port(dev); + struct node_hmem_attrs *hmem_attrs; unsigned long index; struct cxl_ep *ep; @@ -526,6 +527,9 @@ static void cxl_port_release(struct device *dev) xa_destroy(&port->endpoints); xa_destroy(&port->dports); xa_destroy(&port->regions); + xa_for_each(&port->cdat.sslbis_xa, index, hmem_attrs) + kfree(hmem_attrs); + xa_destroy(&port->cdat.sslbis_xa); ida_free(&cxl_port_ida, port->id); kfree(port); } @@ -684,6 +688,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport, xa_init(&port->dports); xa_init(&port->endpoints); xa_init(&port->regions); + xa_init(&port->cdat.sslbis_xa); device_initialize(dev); lockdep_set_class_and_subclass(&dev->mutex, &cxl_port_key, port->depth); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 9d0e22fe72c0..50ac74f66cbd 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -581,6 +581,7 @@ struct cxl_port { struct cxl_cdat { void *table; size_t length; + struct xarray sslbis_xa; } cdat; bool cdat_available; }; diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 0f36fb23055c..1bca1c0e4b40 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -148,6 +148,25 @@ struct cdat_dslbis { #define DSLBIS_MEM_MASK GENMASK(3, 0) #define DSLBIS_MEM_MEMORY 0 +struct sslbis_sslbe { + __le16 port_x_id; + __le16 port_y_id; + __le16 value; /* latency or bandwidth */ + __le16 reserved; +} __packed; + +/* Sub-table 5: Switch Scoped Latency and Bandwidth Information Structure (SSLBIS) */ +struct cdat_sslbis { + struct cdat_entry_header hdr; + u8 data_type; + u8 reserved[3]; + __le64 entry_base_unit; + struct sslbis_sslbe sslbe[]; +} __packed; + +#define SSLBIS_US_PORT 0x0100 +#define SSLBIS_ANY_PORT 0xffff + int devm_cxl_port_enumerate_dports(struct cxl_port *port); struct cxl_dev_state; int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, @@ -169,4 +188,5 @@ int cxl_##x##_parse_entry(struct cdat_entry_header *header, void *arg) cxl_parse_entry(dsmas); cxl_parse_entry(dslbis); +cxl_parse_entry(sslbis); #endif /* __CXL_PCI_H__ */ diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 36e32f2e2ca9..164383ed5b7a 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -83,7 +83,19 @@ static int cxl_switch_port_probe(struct cxl_port *port) if (IS_ERR(cxlhdm)) return PTR_ERR(cxlhdm); - return devm_cxl_enumerate_decoders(cxlhdm, NULL); + rc = devm_cxl_enumerate_decoders(cxlhdm, NULL); + if (rc < 0) + return rc; + + if (port->cdat.table) { + rc = cdat_table_parse_sslbis(port->cdat.table, + cxl_sslbis_parse_entry, + (void *)&port->cdat.sslbis_xa); + if (rc <= 0) + dev_warn(&port->dev, "Failed to parse SSLBIS: %d\n", rc); + } + + return 0; } static int cxl_endpoint_port_probe(struct cxl_port *port) From patchwork Fri Apr 7 17:12:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B39D8C77B6C for ; Fri, 7 Apr 2023 17:12:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230071AbjDGRMr (ORCPT ); Fri, 7 Apr 2023 13:12:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230046AbjDGRMq (ORCPT ); Fri, 7 Apr 2023 13:12:46 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E29B9ED1; Fri, 7 Apr 2023 10:12:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887565; x=1712423565; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=H7zfqOcVJ02CJZ2p/NGllYcgfY1NukX2REpKJgA7P1A=; b=ixgz4OQQYwSmjEGujcqlyJU1JyiUE9Hognr28LdcXgpOMw7/PtaIhObh 187cxEtozmKa18xm00xbKZ8la4QK9CWVRKb2VqGHsh/yOVUIaccJ/W3ad Ucx+oBbVxSO8bGa3PtIgYc+EwpLnMDqI5vDOmrU9StbysyAu5CBSMa5TG IC1A6IxsNp3Ozh9yz/8YPNkep2yz3DbGDaTsGrALfexOLa8DwO/SNgnmD GRR3XLHRfLKigPT5cmbeTdgIDJAu7h57tGa8LOWOtiWuGF6sLw/PDf4fg EVFNpU8qbtX3bslymhUKyDawYjysJ7kmAXJNjXo/2qq8vWGrChV1f29RL w==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="331684796" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="331684796" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="681086991" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="681086991" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:44 -0700 Subject: [PATCH 3 08/23] cxl: Add support for _DSM Function for retrieving QTG ID From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:44 -0700 Message-ID: <168088756427.1441063.2551375099982005342.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org CXL spec v3.0 9.17.3 CXL Root Device Specific Methods (_DSM) Add support to retrieve QTG ID via ACPI _DSM call. The _DSM call requires an input of an ACPI package with 4 dwords (read latency, write latency, read bandwidth, write bandwidth). The call returns a package with 1 WORD that provides the max supported QTG ID and a package that may contain 0 or more WORDs as the recommended QTG IDs in the recommended order. Signed-off-by: Dave Jiang --- v2: - Reorder var declaration and use C99 style. (Jonathan) - Allow >2 ACPI objects in package for future expansion. (Jonathan) - Check QTG IDs against MAX QTG ID provided by output package. (Jonathan) --- drivers/cxl/core/Makefile | 1 drivers/cxl/core/acpi.c | 116 +++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 16 ++++++ 3 files changed, 133 insertions(+) create mode 100644 drivers/cxl/core/acpi.c diff --git a/drivers/cxl/core/Makefile b/drivers/cxl/core/Makefile index 867a8014b462..30d61c8cae22 100644 --- a/drivers/cxl/core/Makefile +++ b/drivers/cxl/core/Makefile @@ -13,5 +13,6 @@ cxl_core-y += mbox.o cxl_core-y += pci.o cxl_core-y += hdm.o cxl_core-y += cdat.o +cxl_core-y += acpi.o cxl_core-$(CONFIG_TRACING) += trace.o cxl_core-$(CONFIG_CXL_REGION) += region.o diff --git a/drivers/cxl/core/acpi.c b/drivers/cxl/core/acpi.c new file mode 100644 index 000000000000..6eda5cad8d59 --- /dev/null +++ b/drivers/cxl/core/acpi.c @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Intel Corporation. All rights reserved. */ +#include +#include +#include +#include +#include +#include +#include "cxlpci.h" +#include "cxl.h" + +const guid_t acpi_cxl_qtg_id_guid = + GUID_INIT(0xF365F9A6, 0xA7DE, 0x4071, + 0xA6, 0x6A, 0xB4, 0x0C, 0x0B, 0x4F, 0x8E, 0x52); + +/** + * cxl_acpi_evaluate_qtg_dsm - Retrieve QTG ids via ACPI _DSM + * @handle: ACPI handle + * @input: bandwidth and latency data + * + * Issue QTG _DSM with accompanied bandwidth and latency data in order to get + * the QTG IDs that falls within the performance data. + */ +struct qtg_dsm_output *cxl_acpi_evaluate_qtg_dsm(acpi_handle handle, + struct qtg_dsm_input *input) +{ + union acpi_object *out_obj, *out_buf, *pkg; + union acpi_object in_buf = { + .buffer = { + .type = ACPI_TYPE_BUFFER, + .pointer = (u8 *)input, + .length = sizeof(u32) * 4, + }, + }; + union acpi_object in_obj = { + .package = { + .type = ACPI_TYPE_PACKAGE, + .count = 1, + .elements = &in_buf + }, + }; + struct qtg_dsm_output *output = NULL; + int len, rc, i; + u16 *max_qtg; + + out_obj = acpi_evaluate_dsm(handle, &acpi_cxl_qtg_id_guid, 1, 1, &in_obj); + if (!out_obj) + return ERR_PTR(-ENXIO); + + if (out_obj->type != ACPI_TYPE_PACKAGE) { + rc = -ENXIO; + goto err; + } + + /* Check Max QTG ID */ + pkg = &out_obj->package.elements[0]; + if (pkg->type != ACPI_TYPE_BUFFER) { + rc = -ENXIO; + goto err; + } + + if (pkg->buffer.length != sizeof(u16)) { + rc = -ENXIO; + goto err; + } + max_qtg = (u16 *)pkg->buffer.pointer; + + /* Retrieve QTG IDs package */ + pkg = &out_obj->package.elements[1]; + if (pkg->type != ACPI_TYPE_PACKAGE) { + rc = -ENXIO; + goto err; + } + + out_buf = &pkg->package.elements[0]; + if (out_buf->type != ACPI_TYPE_BUFFER) { + rc = -ENXIO; + goto err; + } + + len = out_buf->buffer.length; + + /* It's legal to have 0 QTG entries */ + if (len == 0) + goto out; + + /* Malformed package, not multiple of WORD size */ + if (len % sizeof(u16)) { + rc = -ENXIO; + goto out; + } + + output = kmalloc(len + sizeof(*output), GFP_KERNEL); + if (!output) { + rc = -ENOMEM; + goto err; + } + + output->nr = len / sizeof(u16); + memcpy(output->qtg_ids, out_buf->buffer.pointer, len); + + for (i = 0; i < output->nr; i++) { + if (output->qtg_ids[i] > *max_qtg) + pr_warn("QTG ID %u greater than MAX %u\n", + output->qtg_ids[i], *max_qtg); + } + +out: + ACPI_FREE(out_obj); + return output; + +err: + ACPI_FREE(out_obj); + return ERR_PTR(rc); +} +EXPORT_SYMBOL_NS_GPL(cxl_acpi_evaluate_qtg_dsm, CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 50ac74f66cbd..04b8a032bd14 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -791,6 +792,21 @@ static inline struct cxl_dax_region *to_cxl_dax_region(struct device *dev) } #endif +struct qtg_dsm_input { + u32 rd_lat; + u32 wr_lat; + u32 rd_bw; + u32 wr_bw; +}; + +struct qtg_dsm_output { + int nr; + u16 qtg_ids[]; +}; + +struct qtg_dsm_output *cxl_acpi_evaluate_qtg_dsm(acpi_handle handle, + struct qtg_dsm_input *input); + /* * Unit test builds overrides this to __weak, find the 'strong' version * of these symbols in tools/testing/cxl/. From patchwork Fri Apr 7 17:12:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6A74C77B61 for ; Fri, 7 Apr 2023 17:12:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230081AbjDGRMz (ORCPT ); Fri, 7 Apr 2023 13:12:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230085AbjDGRMx (ORCPT ); Fri, 7 Apr 2023 13:12:53 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9210AF3C; Fri, 7 Apr 2023 10:12:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887571; x=1712423571; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8t2nGx+s7JFBFfgM6MI1h6lYgxIWg/nxB+bI0TPO3rg=; b=J8RfdZYttyHnuZllX+od57YJr6gOi0DUaCfm/hjfmCNCeUuzOsHA2xTO 83BKFbc/Ue0OZngeyQj0ioRcDNvDwkt53cTd3WZfzrbTudLWlh+yPuZgb ZmzjIF2OM0B4QR6n25Ey3suvd0eaQmIQ2AeQF4eHIwG/EbkeEmXKCBS32 0wuRjrgWOSeUdzW720jivcrlsmGNfX//yeu3mmtqQXZYxraOOul5+1pJM ImZA6vwa83ZZnn3cRcn2y/yMq7Niy1Uy9aVadwkc09ke+S76NvPqtvGaG IH3yaYtWiKF5sDx5vDGzjmS6hW6wl0cdjO17dL0nUU0l/BrpyIAh+yN+f w==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="331684819" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="331684819" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="681086999" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="681086999" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:50 -0700 Subject: [PATCH 3 09/23] cxl: Add helper function to retrieve ACPI handle of CXL root device From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:50 -0700 Message-ID: <168088757032.1441063.11223896622367813193.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Provide a helper to find the ACPI0017 device in order to issue the _DSM. The helper will take the 'struct device' from a cxl_port and iterate until the root device is reached. The ACPI handle will be returned from the root device. Signed-off-by: Dave Jiang --- v2: - Fix commenting style. (Jonathan) - Fix var declaration aligning. (Jonathan) --- drivers/cxl/core/acpi.c | 34 ++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 1 + 2 files changed, 35 insertions(+) diff --git a/drivers/cxl/core/acpi.c b/drivers/cxl/core/acpi.c index 6eda5cad8d59..191644d0ca6d 100644 --- a/drivers/cxl/core/acpi.c +++ b/drivers/cxl/core/acpi.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include "cxlpci.h" #include "cxl.h" @@ -13,6 +14,39 @@ const guid_t acpi_cxl_qtg_id_guid = GUID_INIT(0xF365F9A6, 0xA7DE, 0x4071, 0xA6, 0x6A, 0xB4, 0x0C, 0x0B, 0x4F, 0x8E, 0x52); +/** + * cxl_acpi_get_rootdev_handle - get the ACPI handle of the CXL root device + * @dev: 'struct device' to start searching from. Should be from cxl_port->dev. + * + * Return: acpi_handle on success, errptr of errno on error. + * + * Looks for the ACPI0017 device and return the ACPI handle + **/ +acpi_handle cxl_acpi_get_rootdev_handle(struct device *dev) +{ + struct device *itr = dev; + struct device *root_dev; + acpi_handle handle; + + if (!dev) + return ERR_PTR(-EINVAL); + + while (itr->parent) { + root_dev = itr; + itr = itr->parent; + } + + if (!dev_is_platform(root_dev)) + return ERR_PTR(-ENODEV); + + handle = ACPI_HANDLE(root_dev); + if (!handle) + return ERR_PTR(-ENODEV); + + return handle; +} +EXPORT_SYMBOL_NS_GPL(cxl_acpi_get_rootdev_handle, CXL); + /** * cxl_acpi_evaluate_qtg_dsm - Retrieve QTG ids via ACPI _DSM * @handle: ACPI handle diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 04b8a032bd14..dc6da641ced0 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -806,6 +806,7 @@ struct qtg_dsm_output { struct qtg_dsm_output *cxl_acpi_evaluate_qtg_dsm(acpi_handle handle, struct qtg_dsm_input *input); +acpi_handle cxl_acpi_get_rootdev_handle(struct device *dev); /* * Unit test builds overrides this to __weak, find the 'strong' version From patchwork Fri Apr 7 17:12:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67786C77B6C for ; Fri, 7 Apr 2023 17:13:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230058AbjDGRNA (ORCPT ); Fri, 7 Apr 2023 13:13:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230091AbjDGRM7 (ORCPT ); Fri, 7 Apr 2023 13:12:59 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4300911C; Fri, 7 Apr 2023 10:12:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887578; x=1712423578; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=U9t8irqFfRaxiNZDrcPZ50oWgg6/WGThyNchyUo1CQ0=; b=hLRrxmVlDZ0KaOnPZorc7y0Pj+MuY/htrkog14JwbRWwHMEqNbU1UeRc qgjJP0J8Q0NKye3QmMXhsDKwZSU+eiO8ZNKnC6JlJB9GWqkw6F6AGS79V gzZyHfcPfAM0kkRZyN37rgFPEHNo/umOHEqjHfsWTvVLAWK1AO7+Luydt IAA/DP5+0IdStCnr0YZkt/XvxryBT9u4v/O9my/hbdQFpCCJwMhVfIDLe YV28wBsxzyvJR4xsJQPBoihDi9HCfyBT+5XK2zJWcR7O81IyocnIdwBb3 mrRoFstg2/mNV+LtffSr00qgEqf+1EUq1BAzYmyp7yNvpYu97ztcUGDAF w==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="344806191" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="344806191" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="637766403" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="637766403" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:12:57 -0700 Subject: [PATCH 3 10/23] cxl: Add helpers to calculate pci latency for the CXL device From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:12:56 -0700 Message-ID: <168088757644.1441063.5632015975342945863.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org The latency is calculated by dividing the flit size over the bandwidth. Add support to retrieve the flit size for the CXL device and calculate the latency of the downstream link. Signed-off-by: Dave Jiang --- v2: - Fix commit log issues. (Jonathan) - Fix var declaration issues. (Jonathan) --- drivers/cxl/core/pci.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxlpci.h | 15 +++++++++++ drivers/cxl/pci.c | 13 --------- 3 files changed, 83 insertions(+), 13 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 1c415b26e866..9d5a0addbef2 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -712,3 +712,71 @@ pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, return PCI_ERS_RESULT_NEED_RESET; } EXPORT_SYMBOL_NS_GPL(cxl_error_detected, CXL); + +static int pcie_speed_to_mbps(enum pci_bus_speed speed) +{ + switch (speed) { + case PCIE_SPEED_2_5GT: + return 2500; + case PCIE_SPEED_5_0GT: + return 5000; + case PCIE_SPEED_8_0GT: + return 8000; + case PCIE_SPEED_16_0GT: + return 16000; + case PCIE_SPEED_32_0GT: + return 32000; + case PCIE_SPEED_64_0GT: + return 64000; + default: + break; + } + + return -EINVAL; +} + +static int cxl_pci_mbits_to_mbytes(struct pci_dev *pdev) +{ + int mbits; + + mbits = pcie_speed_to_mbps(pdev->bus->cur_bus_speed); + if (mbits < 0) + return mbits; + + return mbits >> 3; +} + +static int cxl_flit_size(struct pci_dev *pdev) +{ + if (cxl_pci_flit_256(pdev)) + return 256; + + return 68; +} + +/** + * cxl_pci_get_latency - calculate the link latency for the PCIe link + * @pdev - PCI device + * + * return: calculated latency or -errno + * + * CXL Memory Device SW Guide v1.0 2.11.4 Link latency calculation + * Link latency = LinkPropagationLatency + FlitLatency + RetimerLatency + * LinkProgationLatency is negligible, so 0 will be used + * RetimerLatency is assumed to be negligible and 0 will be used + * FlitLatency = FlitSize / LinkBandwidth + * FlitSize is defined by spec. CXL rev3.0 4.2.1. + * 68B flit is used up to 32GT/s. >32GT/s, 256B flit size is used. + * The FlitLatency is converted to picoseconds. + */ +long cxl_pci_get_latency(struct pci_dev *pdev) +{ + long bw; + + bw = cxl_pci_mbits_to_mbytes(pdev); + if (bw < 0) + return bw; + + return cxl_flit_size(pdev) * 1000000L / bw; +} +EXPORT_SYMBOL_NS_GPL(cxl_pci_get_latency, CXL); diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 1bca1c0e4b40..795eba31fe29 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -167,6 +167,19 @@ struct cdat_sslbis { #define SSLBIS_US_PORT 0x0100 #define SSLBIS_ANY_PORT 0xffff +/* + * CXL v3.0 6.2.3 Table 6-4 + * The table indicates that if PCIe Flit Mode is set, then CXL is in 256B flits + * mode, otherwise it's 68B flits mode. + */ +static inline bool cxl_pci_flit_256(struct pci_dev *pdev) +{ + u16 lnksta2; + + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &lnksta2); + return lnksta2 & PCI_EXP_LNKSTA2_FLIT; +} + int devm_cxl_port_enumerate_dports(struct cxl_port *port); struct cxl_dev_state; int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, @@ -189,4 +202,6 @@ int cxl_##x##_parse_entry(struct cdat_entry_header *header, void *arg) cxl_parse_entry(dsmas); cxl_parse_entry(dslbis); cxl_parse_entry(sslbis); + +long cxl_pci_get_latency(struct pci_dev *pdev); #endif /* __CXL_PCI_H__ */ diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index ea38bd49b0cf..ed39d133b70d 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -365,19 +365,6 @@ static bool is_cxl_restricted(struct pci_dev *pdev) return pci_pcie_type(pdev) == PCI_EXP_TYPE_RC_END; } -/* - * CXL v3.0 6.2.3 Table 6-4 - * The table indicates that if PCIe Flit Mode is set, then CXL is in 256B flits - * mode, otherwise it's 68B flits mode. - */ -static bool cxl_pci_flit_256(struct pci_dev *pdev) -{ - u16 lnksta2; - - pcie_capability_read_word(pdev, PCI_EXP_LNKSTA2, &lnksta2); - return lnksta2 & PCI_EXP_LNKSTA2_FLIT; -} - static int cxl_pci_ras_unmask(struct pci_dev *pdev) { struct pci_host_bridge *host_bridge = pci_find_host_bridge(pdev->bus); From patchwork Fri Apr 7 17:13:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8367BC6FD1D for ; Fri, 7 Apr 2023 17:13:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230089AbjDGRNG (ORCPT ); Fri, 7 Apr 2023 13:13:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230085AbjDGRNF (ORCPT ); Fri, 7 Apr 2023 13:13:05 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E9364C15; Fri, 7 Apr 2023 10:13:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887584; x=1712423584; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xb8LEuAAC0tQp/FGVl8tryrye56REhmRYVJtrAIZn4o=; b=OyWf9aCpO/sDDE39aJwqluudEvM6al5Ullhw5FrB9WGKgRVTiYXnKUvy pgD1vpLQYytqrumxpuqWAfj+iBbo87G1lGRcH6aKvBqc3xQGvn8k0Dy/H eg3XAeU0eQaPwy2+VdhRJqSv5hWSeG46g7N7Mvhg/cQMMkaTlr84I0wRt zADwqw68iEZF18CS/GUJ45RKq0WP2lUqntaHJNC6S2f+/47/xkYH/Xz1v 4o/QwW+1rrguD5vKLuOjeqyZ1E4GfpGc6X1Kdw6AmK4PJMZO/1chd7wLQ tSwTLeET/G2y63FNOitaUkfj2eI8SFv5Wu3fvlhRyq6Ba2B/ZVhnb1odF Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="344806211" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="344806211" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="637766433" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="637766433" Received: from rramanna-mobl1.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:03 -0700 Subject: [PATCH 3 11/23] cxl: Add helper function that calculates QoS values for switches From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:13:02 -0700 Message-ID: <168088758261.1441063.10463189258856481896.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org The CDAT information from the switch, Switch Scoped Latency and Bandwidth Information Strucutre (SSLBIS), is parsed and stored in an xarray under the cxl_port. The QoS data are indexed by the downstream port id. Walk the CXL ports from endpoint to root and retrieve the relevant QoS information (bandwidth and latency) that are from the switch CDAT. If read or write QoS values are not available, then use the access QoS value. Signed-off-by: Dave Jiang --- v3: - Move to use 'struct node_hmem_attrs' --- drivers/cxl/core/port.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 2 + 2 files changed, 83 insertions(+) diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 8febb0a2d116..8f3bff6a3b3b 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -1945,6 +1945,87 @@ bool schedule_cxl_memdev_detach(struct cxl_memdev *cxlmd) } EXPORT_SYMBOL_NS_GPL(schedule_cxl_memdev_detach, CXL); +/** + * cxl_port_get_switch_qos - retrieve QoS data for CXL switches + * @port: endpoint cxl_port + * @rd_bw: writeback value for min read bandwidth + * @rd_lat: writeback value for total read latency + * @wr_bw: writeback value for min write bandwidth + * @wr_lat: writeback value for total write latency + * + * Return: Errno on failure, 0 on success. -ENOENT if no switch device + */ +int cxl_port_get_switch_qos(struct cxl_port *port, u64 *rd_bw, u64 *rd_lat, + u64 *wr_bw, u64 *wr_lat) +{ + u64 min_rd_bw = ULONG_MAX; + u64 min_wr_bw = ULONG_MAX; + struct cxl_dport *dport; + struct cxl_port *nport; + u64 total_rd_lat = 0; + u64 total_wr_lat = 0; + struct device *next; + int switches = 0; + int rc = 0; + + if (!is_cxl_endpoint(port)) + return -EINVAL; + + /* Skip the endpoint */ + next = port->dev.parent; + nport = to_cxl_port(next); + dport = port->parent_dport; + + do { + struct node_hmem_attrs *hmem_attrs; + u64 lat, bw; + + if (!nport->cdat.table) + break; + + if (!dev_is_pci(dport->dport)) + break; + + hmem_attrs = xa_load(&nport->cdat.sslbis_xa, dport->port_id); + if (xa_is_err(hmem_attrs)) + return xa_err(hmem_attrs); + + if (!hmem_attrs) { + hmem_attrs = xa_load(&nport->cdat.sslbis_xa, SSLBIS_ANY_PORT); + if (xa_is_err(hmem_attrs)) + return xa_err(hmem_attrs); + if (!hmem_attrs) + return -ENXIO; + } + + bw = hmem_attrs->write_bandwidth; + lat = hmem_attrs->write_latency; + min_wr_bw = min_t(u64, min_wr_bw, bw); + total_wr_lat += lat; + + bw = hmem_attrs->read_bandwidth; + lat = hmem_attrs->read_latency; + min_rd_bw = min_t(u64, min_rd_bw, bw); + total_rd_lat += lat; + + dport = nport->parent_dport; + next = next->parent; + nport = to_cxl_port(next); + switches++; + } while (next); + + *wr_bw = min_wr_bw; + *wr_lat = total_wr_lat; + *rd_bw = min_rd_bw; + *rd_lat = total_rd_lat; + + if (!switches) + return -ENOENT; + + return rc; +} +EXPORT_SYMBOL_NS_GPL(cxl_port_get_switch_qos, CXL); + /* for user tooling to ensure port disable work has completed */ static ssize_t flush_store(struct bus_type *bus, const char *buf, size_t count) { diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index dc6da641ced0..21e7c1f78f1f 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -807,6 +807,8 @@ struct qtg_dsm_output { struct qtg_dsm_output *cxl_acpi_evaluate_qtg_dsm(acpi_handle handle, struct qtg_dsm_input *input); acpi_handle cxl_acpi_get_rootdev_handle(struct device *dev); +int cxl_port_get_switch_qos(struct cxl_port *port, u64 *rd_bw, u64 *rd_lat, + u64 *wr_bw, u64 *wr_lat); /* * Unit test builds overrides this to __weak, find the 'strong' version From patchwork Fri Apr 7 17:13:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E17C6C77B6C for ; Fri, 7 Apr 2023 17:13:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230085AbjDGRNN (ORCPT ); Fri, 7 Apr 2023 13:13:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230095AbjDGRNL (ORCPT ); Fri, 7 Apr 2023 13:13:11 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3568CAF30; Fri, 7 Apr 2023 10:13:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887590; x=1712423590; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gU0+fjD7n8Oa++jPhoowFHj07aGJ2pAuhA7crY5fq4E=; b=Fv+tgivrqcKj04zucWxO+rLyYdfQNnZOOw9TzRvrXzLhddK/ykJPno2z oXG+xP11MMBRrXrUE+/3Jx2ESw4z9taLyU3m8wdNafFbB0cvk3Xx6D+nA WaqnofM3a3Aj4kjM+UP4FkC/zzJj8t98072lIvqWSuXRtUfGVjxLKfRMa 8an7pk5pPnTnZbr1LxXUCG0gLYTPVZzKtVIeu4nf1HihjRQ2FnD3DvZd0 ZM6mjYpqG+GOet0qQpC5eUGZ8gkZt4mcbxear+GCjX7AagQHS6AUQ+9CJ vb8G+RNYwxQeD73Bi91+Z5G+U7jBlzdplzYo1nI01kNP8MCQ17pfQQTNv Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="340519881" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="340519881" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="690119044" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="690119044" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:09 -0700 Subject: [PATCH 3 12/23] cxl: Add helper function that calculate QoS values for PCI path From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:13:09 -0700 Message-ID: <168088758909.1441063.6718239019369220365.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Calculate the link bandwidth and latency for the PCIe path from the device to the CXL Host Bridge. This does not include the CDAT data from the device or the switch(es) in the path. Signed-off-by: Dave Jiang --- drivers/cxl/core/port.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 2 ++ 2 files changed, 63 insertions(+) diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 8f3bff6a3b3b..9e935d420d00 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -2026,6 +2026,67 @@ int cxl_port_get_switch_qos(struct cxl_port *port, u64 *rd_bw, u64 *rd_lat, } EXPORT_SYMBOL_NS_GPL(cxl_port_get_switch_qos, CXL); +/** + * cxl_port_get_downstream_qos - retrieve QoS data for PCIE downstream path + * @port: endpoint cxl_port + * @bandwidth: writeback value for min bandwidth + * @latency: writeback value for total latency + * + * Return: Errno on failure, 0 on success. + */ +int cxl_port_get_downstream_qos(struct cxl_port *port, u64 *bandwidth, + u64 *latency) +{ + u64 min_bw = ULONG_MAX; + struct pci_dev *pdev; + struct cxl_port *p; + struct device *dev; + u64 total_lat = 0; + int devices = 0; + u64 lat; + + /* Grab the device that is the PCI device for CXL memdev */ + dev = port->uport->parent; + /* Skip if it's not PCI, most likely a cxl_test device */ + if (!dev_is_pci(dev)) + return 0; + + pdev = to_pci_dev(dev); + min_bw = pcie_bandwidth_available(pdev, NULL, NULL, NULL); + if (min_bw == 0) + return -ENXIO; + + /* convert to MB/s from Mb/s */ + min_bw >>= 3; + + p = port; + do { + struct cxl_dport *dport; + + lat = cxl_pci_get_latency(pdev); + if (lat < 0) + return lat; + + total_lat += lat; + devices++; + + dport = p->parent_dport; + if (!dport) + break; + + p = dport->port; + dev = p->uport; + if (!dev_is_pci(dev)) + break; + pdev = to_pci_dev(dev); + } while (1); + + *bandwidth = min_bw; + *latency = total_lat; + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_port_get_downstream_qos, CXL); + /* for user tooling to ensure port disable work has completed */ static ssize_t flush_store(struct bus_type *bus, const char *buf, size_t count) { diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 21e7c1f78f1f..67e844645ef6 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -809,6 +809,8 @@ struct qtg_dsm_output *cxl_acpi_evaluate_qtg_dsm(acpi_handle handle, acpi_handle cxl_acpi_get_rootdev_handle(struct device *dev); int cxl_port_get_switch_qos(struct cxl_port *port, u64 *rd_bw, u64 *rd_lat, u64 *wr_bw, u64 *wr_lat); +int cxl_port_get_downstream_qos(struct cxl_port *port, u64 *bandwidth, + u64 *latency); /* * Unit test builds overrides this to __weak, find the 'strong' version From patchwork Fri Apr 7 17:13:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4144C77B61 for ; Fri, 7 Apr 2023 17:13:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230095AbjDGRNd (ORCPT ); Fri, 7 Apr 2023 13:13:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230091AbjDGRNc (ORCPT ); Fri, 7 Apr 2023 13:13:32 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EAFFF6A57; Fri, 7 Apr 2023 10:13:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887610; x=1712423610; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fQi+31r++DvDJmP4KQ6/yCYwz1hjnYjtCci1BBEzq/o=; b=hRz54xyDQJA+N8vXbolSMkjaUjcS5E/HtD/b6Lz99N2p2yHLKDrmk/1T 7WBbuBvYD8AzsEtsKritA1lf8LAJ0r+TcvRkstBT9+lGD3U5FviQ877RP 7h45XVQbjlWAmA0nkMCjOgtX36ID1m+XKvB9iOoApW/GqUXLkAgRsfUuN NcnZODkxgnuRiX6OJ8yfZC//uK9qu0XEYMBAcTDEzFeBMQa8mpVdIBZWt sINXdZdsgh4UIjjLBK5/isL9IPuW8IKbX6cp0vQYZSD21oWP61wSF2v5N TvwodaBYq3leDJfG2UeY8CUZCooh9NM4ElGB0Gy8juP6Sx/9bsDqJFFQS Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="340519948" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="340519948" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="690119074" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="690119074" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:18 -0700 Subject: [PATCH 3 13/23] ACPI: NUMA: Create enum for memory_target hmem_attrs indexing From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:13:14 -0700 Message-ID: <168088759491.1441063.12824154767581292588.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Create enums to provide named indexing for the hmem_attrs array. This is in preparation for adding generic port support which will add a third member in the array to keep the generic port attributes separate from the memory attributes. Signed-off-by: Dave Jiang --- drivers/acpi/numa/hmat.c | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c index bba268ecd802..4ce56b6f42da 100644 --- a/drivers/acpi/numa/hmat.c +++ b/drivers/acpi/numa/hmat.c @@ -57,12 +57,18 @@ struct target_cache { struct node_cache_attrs cache_attrs; }; +enum { + ACCESS_NODE_MEM = 0, + ACCESS_NODE_CPU, + ACCESS_NODE_MAX, +}; + struct memory_target { struct list_head node; unsigned int memory_pxm; unsigned int processor_pxm; struct resource memregions; - struct node_hmem_attrs hmem_attrs[2]; + struct node_hmem_attrs hmem_attrs[ACCESS_NODE_MAX]; struct list_head caches; struct node_cache_attrs cache_attrs; bool registered; @@ -338,10 +344,12 @@ static __init int hmat_parse_locality(union acpi_subtable_headers *header, if (mem_hier == ACPI_HMAT_MEMORY) { target = find_mem_target(targs[targ]); if (target && target->processor_pxm == inits[init]) { - hmat_update_target_access(target, type, value, 0); + hmat_update_target_access(target, type, value, + ACCESS_NODE_MEM); /* If the node has a CPU, update access 1 */ if (node_state(pxm_to_node(inits[init]), N_CPU)) - hmat_update_target_access(target, type, value, 1); + hmat_update_target_access(target, type, value, + ACCESS_NODE_CPU); } } } @@ -600,10 +608,12 @@ static void hmat_register_target_initiators(struct memory_target *target) */ if (target->processor_pxm != PXM_INVAL) { cpu_nid = pxm_to_node(target->processor_pxm); - register_memory_node_under_compute_node(mem_nid, cpu_nid, 0); + register_memory_node_under_compute_node(mem_nid, cpu_nid, + ACCESS_NODE_MEM); access0done = true; if (node_state(cpu_nid, N_CPU)) { - register_memory_node_under_compute_node(mem_nid, cpu_nid, 1); + register_memory_node_under_compute_node(mem_nid, cpu_nid, + ACCESS_NODE_CPU); return; } } @@ -644,12 +654,13 @@ static void hmat_register_target_initiators(struct memory_target *target) } if (best) hmat_update_target_access(target, loc->hmat_loc->data_type, - best, 0); + best, ACCESS_NODE_MEM); } for_each_set_bit(i, p_nodes, MAX_NUMNODES) { cpu_nid = pxm_to_node(i); - register_memory_node_under_compute_node(mem_nid, cpu_nid, 0); + register_memory_node_under_compute_node(mem_nid, cpu_nid, + ACCESS_NODE_MEM); } } @@ -681,11 +692,13 @@ static void hmat_register_target_initiators(struct memory_target *target) clear_bit(initiator->processor_pxm, p_nodes); } if (best) - hmat_update_target_access(target, loc->hmat_loc->data_type, best, 1); + hmat_update_target_access(target, loc->hmat_loc->data_type, best, + ACCESS_NODE_CPU); } for_each_set_bit(i, p_nodes, MAX_NUMNODES) { cpu_nid = pxm_to_node(i); - register_memory_node_under_compute_node(mem_nid, cpu_nid, 1); + register_memory_node_under_compute_node(mem_nid, cpu_nid, + ACCESS_NODE_CPU); } } @@ -746,8 +759,8 @@ static void hmat_register_target(struct memory_target *target) if (!target->registered) { hmat_register_target_initiators(target); hmat_register_target_cache(target); - hmat_register_target_perf(target, 0); - hmat_register_target_perf(target, 1); + hmat_register_target_perf(target, ACCESS_NODE_MEM); + hmat_register_target_perf(target, ACCESS_NODE_CPU); target->registered = true; } mutex_unlock(&target_lock); From patchwork Fri Apr 7 17:13:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DFA3C77B6C for ; Fri, 7 Apr 2023 17:13:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230113AbjDGRNe (ORCPT ); Fri, 7 Apr 2023 13:13:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230091AbjDGRNd (ORCPT ); Fri, 7 Apr 2023 13:13:33 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A599729F; Fri, 7 Apr 2023 10:13:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887612; x=1712423612; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pqvHrq3JTEcbP4F03/9y6x1mUWiES3JePROGlFjOIsk=; b=JoidxRhuv7yHsSwSrAyQt4c12okMhH0kD4C4g5kIAlV4dJyPgh6twpJt QtY6Maf/7grhe/tqYSrVSdt/a4XmgNPHuT8LqXSRTsSQjiOX+2em1r52m BFA3++oUznfZUR6RV0zOviBecqO8wgDh+BAiFcN/FB4ims8XAbcLV93wt NSelOBuMl4ECtGg3oNUV1HhRC6qk8qdy7Ougeqd1d73khO6Osu6JISOYr FXtZmvqGJF9chnAR5AHrHaXWluo6dysSdsf9RIx5gptkiVzLNK5EwQHD5 F3sXXYQ0p82FNJksInIiAFKXZwbxA1IK3QxGVOqgFpWAygiCKiQeUagIa A==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="340519973" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="340519973" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="690119130" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="690119130" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:24 -0700 Subject: [PATCH 3 14/23] ACPI: NUMA: Add genport target allocation to the HMAT parsing From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:13:24 -0700 Message-ID: <168088760432.1441063.8987031206713273633.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Add SRAT parsing for the HMAT init in order to collect the device handle from the Generic Port Affinity Structure. The devie handle will serve as the key to search for target data. Signed-off-by: Dave Jiang --- v3: - Refactored to share common code --- drivers/acpi/numa/hmat.c | 52 +++++++++++++++++++++++++++++++++++++++++++--- include/acpi/actbl3.h | 2 ++ 2 files changed, 51 insertions(+), 3 deletions(-) diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c index 4ce56b6f42da..5c53aa95d291 100644 --- a/drivers/acpi/numa/hmat.c +++ b/drivers/acpi/numa/hmat.c @@ -71,6 +71,7 @@ struct memory_target { struct node_hmem_attrs hmem_attrs[ACCESS_NODE_MAX]; struct list_head caches; struct node_cache_attrs cache_attrs; + u8 device_handle[ACPI_SRAT_DEVICE_HANDLE_SIZE]; bool registered; }; @@ -125,8 +126,7 @@ static __init void alloc_memory_initiator(unsigned int cpu_pxm) list_add_tail(&initiator->node, &initiators); } -static __init void alloc_memory_target(unsigned int mem_pxm, - resource_size_t start, resource_size_t len) +static __init struct memory_target *alloc_target(unsigned int mem_pxm) { struct memory_target *target; @@ -134,7 +134,7 @@ static __init void alloc_memory_target(unsigned int mem_pxm, if (!target) { target = kzalloc(sizeof(*target), GFP_KERNEL); if (!target) - return; + return NULL; target->memory_pxm = mem_pxm; target->processor_pxm = PXM_INVAL; target->memregions = (struct resource) { @@ -147,6 +147,18 @@ static __init void alloc_memory_target(unsigned int mem_pxm, INIT_LIST_HEAD(&target->caches); } + return target; +} + +static __init void alloc_memory_target(unsigned int mem_pxm, + resource_size_t start, resource_size_t len) +{ + struct memory_target *target; + + target = alloc_target(mem_pxm); + if (!target) + return; + /* * There are potentially multiple ranges per PXM, so record each * in the per-target memregions resource tree. @@ -157,6 +169,17 @@ static __init void alloc_memory_target(unsigned int mem_pxm, start, start + len, mem_pxm); } +static __init void alloc_genport_target(unsigned int mem_pxm, u8 *handle) +{ + struct memory_target *target; + + target = alloc_target(mem_pxm); + if (!target) + return; + + memcpy(target->device_handle, handle, ACPI_SRAT_DEVICE_HANDLE_SIZE); +} + static __init const char *hmat_data_type(u8 type) { switch (type) { @@ -498,6 +521,22 @@ static __init int srat_parse_mem_affinity(union acpi_subtable_headers *header, return 0; } +static __init int srat_parse_genport_affinity(union acpi_subtable_headers *header, + const unsigned long end) +{ + struct acpi_srat_generic_affinity *ga = (void *)header; + + if (!ga) + return -EINVAL; + + if (!(ga->flags & ACPI_SRAT_GENERIC_AFFINITY_ENABLED)) + return 0; + + alloc_genport_target(ga->proximity_domain, (u8 *)ga->device_handle); + + return 0; +} + static u32 hmat_initiator_perf(struct memory_target *target, struct memory_initiator *initiator, struct acpi_hmat_locality *hmat_loc) @@ -848,6 +887,13 @@ static __init int hmat_init(void) ACPI_SRAT_TYPE_MEMORY_AFFINITY, srat_parse_mem_affinity, 0) < 0) goto out_put; + + if (acpi_table_parse_entries(ACPI_SIG_SRAT, + sizeof(struct acpi_table_srat), + ACPI_SRAT_TYPE_GENERIC_PORT_AFFINITY, + srat_parse_genport_affinity, 0) < 0) + goto out_put; + acpi_put_table(tbl); status = acpi_get_table(ACPI_SIG_HMAT, 0, &tbl); diff --git a/include/acpi/actbl3.h b/include/acpi/actbl3.h index 832c6464f063..0daf5a94f08a 100644 --- a/include/acpi/actbl3.h +++ b/include/acpi/actbl3.h @@ -289,6 +289,8 @@ struct acpi_srat_generic_affinity { u32 reserved1; }; +#define ACPI_SRAT_DEVICE_HANDLE_SIZE 16 + /* Flags for struct acpi_srat_generic_affinity */ #define ACPI_SRAT_GENERIC_AFFINITY_ENABLED (1) /* 00: Use affinity structure */ From patchwork Fri Apr 7 17:13:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6BF2C6FD1D for ; Fri, 7 Apr 2023 17:13:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230091AbjDGRNi (ORCPT ); Fri, 7 Apr 2023 13:13:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230094AbjDGRNd (ORCPT ); Fri, 7 Apr 2023 13:13:33 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D07416A57; Fri, 7 Apr 2023 10:13:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887612; x=1712423612; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nb9cXS4op/kjmY5UIvlPea+yMzVt9eWRe9c+uNmvxck=; b=hdKeKpj6uf8ey4dMnHqcxvggZnDC8j14PLyDZzWRjdj5MKQ5zu63NZwO IzKi7Hdr5JYwCQuU4HiKr6RE9dhb4Vq7FIsLcIX5oi9Oj7t9XIrxr+bmn 5EfF9/1Z7g7XifkSP1xU/vu8K0PyPJOKL1oejh9ixdIXfWDXOPe0fa3H7 lYQFVFpWlV2p1pq9W2yBybeit5fRyjvNUG5nZDMgBj+hNRXp+2eDBIx1Y eQcUwzoEJwRAxYrjYGKTCLGts5PU32mmHp+SVdTFGUsJEvCDMlG1DREff 2X+9r1bFXkiHERsaQUHHvFqhrXZ8biGfa2/zycCzy4SC3IQzbJbE2ktjE Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="340519983" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="340519983" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="690119133" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="690119133" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:30 -0700 Subject: [PATCH 3 15/23] ACPI: NUMA: Add setting of generic port locality attributes From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:13:30 -0700 Message-ID: <168088761007.1441063.16227274190441918773.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Add generic port support for the parsing of HMAT locality sub-table. The attributes will be added to the third array member of the hmem_attrs in order to not mix with the existing memory attributes it only provides the locality attributes from initator to the generic port targets and is missing the rest of the data from the actual memory device. The actual memory attributes will be updated when a memory device is attached and the locality information is calculated end to end. Signed-off-by: Dave Jiang --- drivers/acpi/numa/hmat.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c index 5c53aa95d291..2596eb9eb419 100644 --- a/drivers/acpi/numa/hmat.c +++ b/drivers/acpi/numa/hmat.c @@ -60,6 +60,7 @@ struct target_cache { enum { ACCESS_NODE_MEM = 0, ACCESS_NODE_CPU, + ACCESS_NODE_GENPORT, ACCESS_NODE_MAX, }; @@ -367,6 +368,12 @@ static __init int hmat_parse_locality(union acpi_subtable_headers *header, if (mem_hier == ACPI_HMAT_MEMORY) { target = find_mem_target(targs[targ]); if (target && target->processor_pxm == inits[init]) { + if (*(target->device_handle)) { + hmat_update_target_access(target, type, value, + ACCESS_NODE_GENPORT); + continue; + } + hmat_update_target_access(target, type, value, ACCESS_NODE_MEM); /* If the node has a CPU, update access 1 */ From patchwork Fri Apr 7 17:13:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0355EC6FD1D for ; Fri, 7 Apr 2023 17:13:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230137AbjDGRNo (ORCPT ); Fri, 7 Apr 2023 13:13:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230094AbjDGRNm (ORCPT ); Fri, 7 Apr 2023 13:13:42 -0400 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE0C1AD13; Fri, 7 Apr 2023 10:13:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887620; x=1712423620; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5tvzis9vZH6ocAk1I9hJKqp0bsV1H0EBYTkN66XhAdA=; b=PlZaRIDcnmIK0rRUsrMlSfCPhS/VjipYLYF/8tUNk7zVFIpTTErgBg5f VgpJxfE7dOKWKmUdGEz18FlYJlrW7Alf9YY54e08jab14EFBx4OFbGnbp jOELNUgos0Cn+zkwp+qQlE6CzT5ClmMbgu8xyJ8JUuu8jG5Ovq6TMCeRZ +DQA+AY4mHGR55Ff6BGO7y1C0ys87n0B1YoYW+0huCnTyL2eeNW+LbC3e YzCnM8FRU20iPUx8/g3F2g80yxTtXU7mAzu/uyAfBMAd0BIW964lrfXvR ZyZC94SRErIWEH+/WPr2jcK9geiIQg2txqQpdSJt9UWQIA11aR2wlLVBD A==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="340520004" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="340520004" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="690119162" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="690119162" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:36 -0700 Subject: [PATCH 3 16/23] ACPI: NUMA: Add helper function to retrieve the performance attributes From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:13:36 -0700 Message-ID: <168088761615.1441063.15822678926474147168.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Add helper to retrieve the performance attributes based on the device handle. The helper function is exported so the CXL driver can use that to acquire the performance data between the CPU and the CXL host bridge. Signed-off-by: Dave Jiang --- v3: - Fix 0-day report on extra ';' --- drivers/acpi/numa/hmat.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ include/linux/acpi.h | 6 ++++++ 2 files changed, 50 insertions(+) diff --git a/drivers/acpi/numa/hmat.c b/drivers/acpi/numa/hmat.c index 2596eb9eb419..c3598ddeed1c 100644 --- a/drivers/acpi/numa/hmat.c +++ b/drivers/acpi/numa/hmat.c @@ -107,6 +107,50 @@ static struct memory_target *find_mem_target(unsigned int mem_pxm) return NULL; } +static struct memory_target *acpi_find_genport_target(u8 *device_handle) +{ + struct memory_target *target; + + list_for_each_entry(target, &targets, node) { + if (!strncmp(target->device_handle, device_handle, + ACPI_SRAT_DEVICE_HANDLE_SIZE)) + return target; + } + + return NULL; +} + +int acpi_get_genport_attrs(u8 *device_handle, u64 *val, int type) +{ + struct memory_target *target; + + target = acpi_find_genport_target(device_handle); + if (!target) + return -ENOENT; + + switch (type) { + case ACPI_HMAT_ACCESS_LATENCY: + case ACPI_HMAT_READ_LATENCY: + *val = target->hmem_attrs[ACCESS_NODE_GENPORT].read_latency; + break; + case ACPI_HMAT_WRITE_LATENCY: + *val = target->hmem_attrs[ACCESS_NODE_GENPORT].write_latency; + break; + case ACPI_HMAT_ACCESS_BANDWIDTH: + case ACPI_HMAT_READ_BANDWIDTH: + *val = target->hmem_attrs[ACCESS_NODE_GENPORT].read_bandwidth; + break; + case ACPI_HMAT_WRITE_BANDWIDTH: + *val = target->hmem_attrs[ACCESS_NODE_GENPORT].write_bandwidth; + break; + default: + return -EINVAL; + } + + return 0; +} +EXPORT_SYMBOL_GPL(acpi_get_genport_attrs); + static __init void alloc_memory_initiator(unsigned int cpu_pxm) { struct memory_initiator *initiator; diff --git a/include/linux/acpi.h b/include/linux/acpi.h index efff750f326d..2dc49ef3e28a 100644 --- a/include/linux/acpi.h +++ b/include/linux/acpi.h @@ -451,6 +451,7 @@ extern bool acpi_osi_is_win8(void); #ifdef CONFIG_ACPI_NUMA int acpi_map_pxm_to_node(int pxm); int acpi_get_node(acpi_handle handle); +int acpi_get_genport_attrs(u8 *device_handle, u64 *val, int type); /** * pxm_to_online_node - Map proximity ID to online node @@ -485,6 +486,11 @@ static inline int acpi_get_node(acpi_handle handle) { return 0; } + +static inline int acpi_get_genport_attrs(u8 *device_handle, u64 *val, int type) +{ + return -EOPNOTSUPP; +} #endif extern int acpi_paddr_to_node(u64 start_addr, u64 size); From patchwork Fri Apr 7 17:13:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81519C6FD1D for ; Fri, 7 Apr 2023 17:13:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230151AbjDGRNu (ORCPT ); Fri, 7 Apr 2023 13:13:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230144AbjDGRNp (ORCPT ); Fri, 7 Apr 2023 13:13:45 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7CFF07A8F; Fri, 7 Apr 2023 10:13:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887623; x=1712423623; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3D19eVA6StizRDQ3Hc2AQXS3EISK8uXQhS4ITUho+hg=; b=AdV7yd/x+OIh2qvp6S0Ocx7lfiNUY1tPwyKlWb1eTVHGhjGj9y9XMzXo H0Q7azxxpthkFdncmNBLJoMFXp59pn4OqsL6AbdG3VQrGxUov04wrvACn 3U/T+JIYmUZrqXsgVj9VlH3PbDNxhczHXYKP450PFjn61gdXO16T8bnmj Vda/2iwjwTkG1sMMvnegPtUz1Z0aHRwO8KQk6j4uqERspwxDL0ZWO/05g eCmu6Mfk2PCHNM3M9EkrJ6E7fxCthET+XaCLQ9hJN1fXuDKLSPr4fpX0p IETFHYvt5OpIU6bwb2VpLmvXfGzp7ZqB/VgUinZj3+emVyJcgbzpusQ3C g==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="345682042" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="345682042" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="831251741" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="831251741" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:42 -0700 Subject: [PATCH 3 17/23] cxl: Add helper function to retrieve generic port QoS From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:13:41 -0700 Message-ID: <168088762190.1441063.16189131969173431967.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Add CXL helper function that retrieves the bandwidth and latency data of a generic port by calling acpi_get_genport_attrs() function. A device handle is passed in constructed from the ACPI HID and UID of the CXL host bridge (ACPI0016) device. Signed-off-by: Dave Jiang --- drivers/cxl/core/acpi.c | 30 ++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 1 + 2 files changed, 31 insertions(+) diff --git a/drivers/cxl/core/acpi.c b/drivers/cxl/core/acpi.c index 191644d0ca6d..41eeaa8c272e 100644 --- a/drivers/cxl/core/acpi.c +++ b/drivers/cxl/core/acpi.c @@ -148,3 +148,33 @@ struct qtg_dsm_output *cxl_acpi_evaluate_qtg_dsm(acpi_handle handle, return ERR_PTR(rc); } EXPORT_SYMBOL_NS_GPL(cxl_acpi_evaluate_qtg_dsm, CXL); + +/** + * cxl_acpi_get_hb_qos - retrieve QoS data for generic port + * @host: 'struct device' of the CXL host bridge + * @latency: genport latency data + * @bandwidth: genport bandwidth data + * + * Return: Errno on failure, 0 on success. + */ +int cxl_acpi_get_hb_qos(struct device *host, u64 *latency, u64 *bandwidth) +{ + u8 handle[ACPI_SRAT_DEVICE_HANDLE_SIZE] = { 0 }; + struct acpi_device *adev = ACPI_COMPANION(host); + int rc; + + /* ACPI spec 6.5 Table 5.65 */ + memcpy(handle, acpi_device_hid(adev), 8); + memcpy(&handle[8], acpi_device_uid(adev), 4); + + rc = acpi_get_genport_attrs(handle, latency, ACPI_HMAT_ACCESS_LATENCY); + if (rc) + return rc; + + rc = acpi_get_genport_attrs(handle, bandwidth, ACPI_HMAT_ACCESS_BANDWIDTH); + if (rc) + return rc; + + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_acpi_get_hb_qos, CXL); diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 67e844645ef6..56bcf144eede 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -811,6 +811,7 @@ int cxl_port_get_switch_qos(struct cxl_port *port, u64 *rd_bw, u64 *rd_lat, u64 *wr_bw, u64 *wr_lat); int cxl_port_get_downstream_qos(struct cxl_port *port, u64 *bandwidth, u64 *latency); +int cxl_acpi_get_hb_qos(struct device *host, u64 *latency, u64 *bandwidth); /* * Unit test builds overrides this to __weak, find the 'strong' version From patchwork Fri Apr 7 17:13:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05CE4C77B6F for ; Fri, 7 Apr 2023 17:13:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230117AbjDGRNv (ORCPT ); Fri, 7 Apr 2023 13:13:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230114AbjDGRNu (ORCPT ); Fri, 7 Apr 2023 13:13:50 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED002729F; Fri, 7 Apr 2023 10:13:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887629; x=1712423629; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tEKgXBGrCQZJ5AH79Txb90f0fhtVQ6urF776D2C7YiY=; b=GmAL8DC5NXIXNy0b5o9QUAyZusagrJa0KJildkJSShj4O4/zGp8nvn6b jGIOc4RXGWHlcmP2TPqnYzSGhxvclsp7rrUoAWwuk3tA/M+V80e4e1BjQ D9zUPaY0oHRIoLR9M2zsJQ28R8X169Vt26hG4dAOeDZYkGMEdoOVmA9vz dfDh7/EXSQZMXgFNIAu6pSCFKKJLkUBsCfox99F1RSyEIYtVMV0aWGFn8 Z84scGzgDKMoqraMRZJ8eTfLJ29kz6r1QjoV2m+o3DLClqBviIbXBrqno kzcqywxtd3bFP6pE/BmsfyBCliISO9P8psHG9kT+59IJ+d+loYNWVUs0W A==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="345682060" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="345682060" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="831251778" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="831251778" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:48 -0700 Subject: [PATCH 3 18/23] cxl: Add latency and bandwidth calculations for the CXL path From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:13:47 -0700 Message-ID: <168088762770.1441063.9693855327937653263.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org CXL Memory Device SW Guide rev1.0 2.11.2 provides instruction on how to caluclate latency and bandwidth for CXL memory device. Calculate minimum bandwidth and total latency for the path from the CXL device to the root port. The retrieved QTG ID is stored to the cxl_port of the CXL device. For example for a device that is directly attached to a host bus: Total Latency = Device Latency (from CDAT) + Dev to Host Bus (HB) Link Latency + Generic Port Latency Min Bandwidth = Min bandwidth for link bandwidth between HB and CXL device, device CDAT bandwidth, and Generic Port Bandwidth For a device that has a switch in between host bus and CXL device: Total Latency = Device (CDAT) Latency + Dev to Switch Link Latency + Switch (CDAT) Latency + Switch to HB Link Latency + Generic Port Latency Min Bandwidth = Min bandwidth for link bandwidth between CXL device to CXL switch, CXL device CDAT bandwidth, CXL switch CDAT bandwidth, CXL switch to HB bandwidth, and Generic Port Bandwidth. Signed-off-by: Dave Jiang --- drivers/cxl/cxlpci.h | 1 + drivers/cxl/port.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 795eba31fe29..ff4c2d10ca4a 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -120,6 +120,7 @@ struct dsmas_entry { struct range dpa_range; u8 handle; struct node_hmem_attrs hmem_attrs; + u16 qtg_id; }; /* Sub-table 0: Device Scoped Memory Affinity Structure (DSMAS) */ diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 164383ed5b7a..dad5258b4604 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -67,6 +67,63 @@ static void dsmas_list_destroy(struct list_head *dsmas_list) } } +static int cxl_port_qos_calculate(struct cxl_port *port, + struct list_head *dsmas_list) +{ + u64 sw_wr_bw, sw_wr_lat, sw_rd_bw, sw_rd_lat; + u64 min_rd_bw, total_rd_lat, min_wr_bw, total_wr_lat; + struct qtg_dsm_output *output; + struct qtg_dsm_input input; + struct dsmas_entry *dent; + acpi_handle handle; + u64 gp_bw, gp_lat; + u64 ds_bw, ds_lat; + int rc; + + rc = cxl_port_get_downstream_qos(port, &ds_bw, &ds_lat); + if (rc) + return rc; + + rc = cxl_port_get_switch_qos(port, &sw_rd_bw, &sw_rd_lat, + &sw_wr_bw, &sw_wr_lat); + if (rc && rc != -ENOENT) + return rc; + + rc = cxl_acpi_get_hb_qos(port->host_bridge, &gp_lat, &gp_bw); + if (rc) + return rc; + + min_rd_bw = min_t(u64, ds_bw, sw_rd_bw); + min_rd_bw = min_t(u64, gp_bw, min_rd_bw); + total_rd_lat = ds_lat + gp_lat + sw_rd_lat; + + min_wr_bw = min_t(u64, ds_bw, sw_wr_bw); + min_wr_bw = min_t(u64, gp_bw, min_wr_bw); + total_wr_lat = ds_lat + gp_lat + sw_wr_lat; + + handle = cxl_acpi_get_rootdev_handle(&port->dev); + if (IS_ERR(handle)) + return PTR_ERR(handle); + + list_for_each_entry(dent, dsmas_list, list) { + input.rd_lat = dent->hmem_attrs.read_latency + total_rd_lat; + input.wr_lat = dent->hmem_attrs.write_latency + total_wr_lat; + input.rd_bw = min_t(int, min_rd_bw, + dent->hmem_attrs.read_bandwidth); + input.wr_bw = min_t(int, min_wr_bw, + dent->hmem_attrs.write_bandwidth); + + output = cxl_acpi_evaluate_qtg_dsm(handle, &input); + if (IS_ERR(output)) + continue; + + dent->qtg_id = output->qtg_ids[0]; + kfree(output); + } + + return 0; +} + static int cxl_switch_port_probe(struct cxl_port *port) { struct cxl_hdm *cxlhdm; @@ -165,6 +222,10 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) dev_warn(&port->dev, "Failed to parse DSMAS: %d\n", rc); } + rc = cxl_port_qos_calculate(port, &dsmas_list); + if (rc) + dev_dbg(&port->dev, "Failed to do QoS calculations\n"); + dsmas_list_destroy(&dsmas_list); } From patchwork Fri Apr 7 17:13:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6042BC77B61 for ; Fri, 7 Apr 2023 17:14:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230144AbjDGROP (ORCPT ); Fri, 7 Apr 2023 13:14:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230114AbjDGROO (ORCPT ); Fri, 7 Apr 2023 13:14:14 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C344DA5CB; Fri, 7 Apr 2023 10:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887652; x=1712423652; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ljebYg9qvunRUXg2iAM4lfi0gFE5bHRhzKpxaWOS0Kg=; b=jsJ/qoglC+OURq5v35+xXtZFCXTPujm1+XaotFG5tQcn85JbWUSPbA+E +X0eGEehyPo193ukw2cpn7vxTKQa5fD7pt5vQNrI/mwo2PA/G7lSU8DMb qZTMdCQpEYcoMUi1sN9rOXvhdkZktBdOz5GUX9UhmlapEIhK7C7kyEX8d WmJ43M4f7fm8Nx5bPOw3Da+uALsrqICX/vY2Mfm+PGnNMlD7OaE+rZ2DY BIxvNZKT9R1pzd3DflYevXFvU+KgtDNANsDE6Ei2+WBfzDdhk6H24plgS jh63nwymml+tSa0SDA6QeFpqrokKjbK9Qy/wYDEoBMh9G55rKoC9bCBHC A==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="345682108" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="345682108" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:14:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="831251792" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="831251792" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:13:54 -0700 Subject: [PATCH 3 19/23] cxl: Wait Memory_Info_Valid before access memory related info From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:13:53 -0700 Message-ID: <168088763346.1441063.5474477785580754741.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org CXL rev3.0 8.1.3.8.2 Memory_Info_valid field The Memory_Info_Valid bit indicates that the CXL Range Size High and Size Low registers are valid. The bit must be set within 1 second of reset deassertion to the device. Check valid bit before we check the Memory_Active bit when waiting for cxl_await_media_ready() to ensure that the memory info is valid for consumption. Fixes: 2e4ba0ec9783 ("cxl/pci: Move cxl_await_media_ready() to the core") Signed-off-by: Dave Jiang --- v2: - Check both ranges. (Jonathan) --- drivers/cxl/core/pci.c | 83 +++++++++++++++++++++++++++++++++++++++++++----- drivers/cxl/cxlpci.h | 2 + 2 files changed, 77 insertions(+), 8 deletions(-) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 9d5a0addbef2..0c314fb8b42c 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -101,21 +101,55 @@ int devm_cxl_port_enumerate_dports(struct cxl_port *port) } EXPORT_SYMBOL_NS_GPL(devm_cxl_port_enumerate_dports, CXL); -/* - * Wait up to @media_ready_timeout for the device to report memory - * active. - */ -int cxl_await_media_ready(struct cxl_dev_state *cxlds) +static int cxl_dvsec_mem_range_valid(struct cxl_dev_state *cxlds, int id) +{ + struct pci_dev *pdev = to_pci_dev(cxlds->dev); + int d = cxlds->cxl_dvsec; + bool valid = false; + int rc, i; + u32 temp; + + if (id > CXL_DVSEC_RANGE_MAX) + return -EINVAL; + + /* Check MEM INFO VALID bit first, give up after 1s */ + i = 1; + do { + rc = pci_read_config_dword(pdev, + d + CXL_DVSEC_RANGE_SIZE_LOW(id), + &temp); + if (rc) + return rc; + + valid = FIELD_GET(CXL_DVSEC_MEM_INFO_VALID, temp); + if (valid) + break; + msleep(1000); + } while (i--); + + if (!valid) { + dev_err(&pdev->dev, + "Timeout awaiting memory range %d valid after 1s.\n", + id); + return -ETIMEDOUT; + } + + return 0; +} + +static int cxl_dvsec_mem_range_active(struct cxl_dev_state *cxlds, int id) { struct pci_dev *pdev = to_pci_dev(cxlds->dev); int d = cxlds->cxl_dvsec; bool active = false; - u64 md_status; int rc, i; + u32 temp; - for (i = media_ready_timeout; i; i--) { - u32 temp; + if (id > CXL_DVSEC_RANGE_MAX) + return -EINVAL; + /* Check MEM ACTIVE bit, up to 60s timeout by default */ + for (i = media_ready_timeout; i; i--) { rc = pci_read_config_dword( pdev, d + CXL_DVSEC_RANGE_SIZE_LOW(0), &temp); if (rc) @@ -134,6 +168,39 @@ int cxl_await_media_ready(struct cxl_dev_state *cxlds) return -ETIMEDOUT; } + return 0; +} + +/* + * Wait up to @media_ready_timeout for the device to report memory + * active. + */ +int cxl_await_media_ready(struct cxl_dev_state *cxlds) +{ + struct pci_dev *pdev = to_pci_dev(cxlds->dev); + int d = cxlds->cxl_dvsec; + int rc, i, hdm_count; + u64 md_status; + u16 cap; + + rc = pci_read_config_word(pdev, + d + CXL_DVSEC_CAP_OFFSET, &cap); + if (rc) + return rc; + + hdm_count = FIELD_GET(CXL_DVSEC_HDM_COUNT_MASK, cap); + for (i = 0; i < hdm_count; i++) { + rc = cxl_dvsec_mem_range_valid(cxlds, i); + if (rc) + return rc; + } + + for (i = 0; i < hdm_count; i++) { + rc = cxl_dvsec_mem_range_active(cxlds, i); + if (rc) + return rc; + } + md_status = readq(cxlds->regs.memdev + CXLMDEV_STATUS_OFFSET); if (!CXLMDEV_READY(md_status)) return -EIO; diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index ff4c2d10ca4a..cca86a76bd75 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -32,6 +32,8 @@ #define CXL_DVSEC_RANGE_BASE_LOW(i) (0x24 + (i * 0x10)) #define CXL_DVSEC_MEM_BASE_LOW_MASK GENMASK(31, 28) +#define CXL_DVSEC_RANGE_MAX 2 + /* CXL 2.0 8.1.4: Non-CXL Function Map DVSEC */ #define CXL_DVSEC_FUNCTION_MAP 2 From patchwork Fri Apr 7 17:14:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0F4BC77B71 for ; Fri, 7 Apr 2023 17:14:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230173AbjDGROQ (ORCPT ); Fri, 7 Apr 2023 13:14:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229520AbjDGROP (ORCPT ); Fri, 7 Apr 2023 13:14:15 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D9BCAD25; Fri, 7 Apr 2023 10:14:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887654; x=1712423654; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+mZI/k+IwVP7frgAtKYxsQS4FuBNwyZvfyDnmIgFQC8=; b=gtMD3YurP8YZ5eM3UReaPyabC1gaDqZJxbVWGy8rAzY7mEbjaq0IbJN1 VYgjLV2iv028axZxPPSLZRo9ns0Vhcq/TOy2mjznIQPHHlp0+lilFtEXg btPMg230UxB4d7W08WkD+oF2Ym9akNMii+wo5GY/QxhB5cSwowfFc+Vf6 k5O5Y9ii2bmmtlWBgAdrIbDddgcLiNDr37OrcEnKSmYZ9+J+oXjPUaTfJ 14dTc5OYaQy3r98xxrCfflpieLRLBrAGqBZBzAV7royv68q5xS1Lcxuia YjzO1DnuHI6Yc4n4eIFwX6q3FoSa1HjZRHrqhHtooJA1kbYULi/Ywchk8 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="345682112" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="345682112" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:14:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="831251806" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="831251806" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:14:00 -0700 Subject: [PATCH 3 20/23] cxl: Move identify and partition query from pci probe to port probe From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:14:00 -0700 Message-ID: <168088764000.1441063.12595330462471499795.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Move the enumeration of device capacity to cxl_port_probe() from cxl_pci_probe(). The size and capacity information should be read after cxl_await_media_ready() so the data is valid. Fixes: 5e2411ae8071 ("cxl/memdev: Change cxl_mem to a more descriptive name") Signed-off-by: Dave Jiang --- drivers/cxl/pci.c | 8 -------- drivers/cxl/port.c | 8 ++++++++ 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index ed39d133b70d..06324266eae8 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -707,14 +707,6 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (rc) return rc; - rc = cxl_dev_state_identify(cxlds); - if (rc) - return rc; - - rc = cxl_mem_create_range_info(cxlds); - if (rc) - return rc; - rc = cxl_alloc_irq_vectors(pdev); if (rc) return rc; diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index dad5258b4604..70189309c1c0 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -187,6 +187,14 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) return rc; } + rc = cxl_dev_state_identify(cxlds); + if (rc) + return rc; + + rc = cxl_mem_create_range_info(cxlds); + if (rc) + return rc; + rc = devm_cxl_enumerate_decoders(cxlhdm, &info); if (rc) return rc; From patchwork Fri Apr 7 17:14:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A005C77B73 for ; Fri, 7 Apr 2023 17:14:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229520AbjDGROR (ORCPT ); Fri, 7 Apr 2023 13:14:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230165AbjDGROP (ORCPT ); Fri, 7 Apr 2023 13:14:15 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D0A9AF34; Fri, 7 Apr 2023 10:14:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887654; x=1712423654; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ZWsYJfw0gbUl/xTVOGIsEJag6zroc9ti2yLD7YmUSuc=; b=VqzdoxFCuJQVt2AkvYbXRhMG1+Gh3cEurvjqIVJzbKQ/cBnGWaV39YW6 AnLy0aTyI6I7S60WrCa7/FSQ49kooAJidcl5a++mH6ebXLPTuxm6P6axG BAPW+P9uRq0yX/3ous0luJZKR7hnBRj3rDq2aNwgYcOl+dy4EzWKCxI5U wawPj+PeoHrABWWPWalv2R5b2CvwJC4Mlj1/d3jqaqH+1VdFg4P9qJj2f mRuaoCmHUpGixHS/zIDdWBcDUFuu55rwarWWfo/iffx7g2Wlmy+PtFMOO Vs1TZlLcgBwX3QJpOTMEFzQvideeb1T3f1xg8PeUtwODHyAUCJ4f4E9C1 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="345682119" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="345682119" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:14:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="831251826" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="831251826" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:14:06 -0700 Subject: [PATCH 3 21/23] cxl: Store QTG IDs and related info to the CXL memory device context From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:14:05 -0700 Message-ID: <168088764576.1441063.4073780159473151210.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Once the QTG ID _DSM is executed successfully, the QTG ID is retrieved from the return package. Create a list of entries in the cxl_memdev context and store the QTG ID and the associated DPA range. This information can be exposed to user space via sysfs in order to help region setup for hot-plugged CXL memory devices. Signed-off-by: Dave Jiang --- v3: - Move back to QTG ID per partition --- drivers/cxl/core/mbox.c | 3 +++ drivers/cxl/cxlmem.h | 20 ++++++++++++++++++++ drivers/cxl/port.c | 35 +++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+) diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index f2addb457172..a535a1b78b0e 100644 --- a/drivers/cxl/core/mbox.c +++ b/drivers/cxl/core/mbox.c @@ -1120,6 +1120,9 @@ struct cxl_dev_state *cxl_dev_state_create(struct device *dev) mutex_init(&cxlds->mbox_mutex); mutex_init(&cxlds->event.log_lock); cxlds->dev = dev; + INIT_LIST_HEAD(&cxlds->qos_list); + cxlds->ram_qtg_id = -1; + cxlds->pmem_qtg_id = -1; return cxlds; } diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index 001dabf0231b..7107ef594555 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -5,6 +5,7 @@ #include #include #include +#include #include "cxl.h" /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */ @@ -40,6 +41,7 @@ * @cxl_nvd: optional bridge to an nvdimm if the device supports pmem * @id: id number of this memdev instance. * @depth: endpoint port depth + * @qos_list: QTG ID related list of entries */ struct cxl_memdev { struct device dev; @@ -50,6 +52,7 @@ struct cxl_memdev { struct cxl_nvdimm *cxl_nvd; int id; int depth; + struct list_head qos_list; }; static inline struct cxl_memdev *to_cxl_memdev(struct device *dev) @@ -215,6 +218,19 @@ struct cxl_event_state { struct mutex log_lock; }; +/** + * struct qos_prop - QoS property entry + * @list - list entry + * @dpa_range - range for DPA address + * @qtg_id - QoS Throttling Group ID + */ +struct qos_prop_entry { + struct list_head list; + struct range dpa_range; + u16 qtg_id; + struct node_hmem_attrs hmem_attrs; +}; + /** * struct cxl_dev_state - The driver device state * @@ -283,6 +299,10 @@ struct cxl_dev_state { u64 next_volatile_bytes; u64 next_persistent_bytes; + int ram_qtg_id; + int pmem_qtg_id; + struct list_head qos_list; + resource_size_t component_reg_phys; u64 serial; diff --git a/drivers/cxl/port.c b/drivers/cxl/port.c index 70189309c1c0..facecb5c8b00 100644 --- a/drivers/cxl/port.c +++ b/drivers/cxl/port.c @@ -124,6 +124,40 @@ static int cxl_port_qos_calculate(struct cxl_port *port, return 0; } +static void cxl_memdev_set_qtg(struct cxl_dev_state *cxlds, struct list_head *dsmas_list) +{ + struct range pmem_range = { + .start = cxlds->pmem_res.start, + .end = cxlds->pmem_res.end, + }; + struct range ram_range = { + .start = cxlds->ram_res.start, + .end = cxlds->ram_res.end, + }; + struct qos_prop_entry *qos; + struct dsmas_entry *dent; + + list_for_each_entry(dent, dsmas_list, list) { + qos = devm_kzalloc(cxlds->dev, sizeof(*qos), GFP_KERNEL); + if (!qos) + return; + + qos->dpa_range = dent->dpa_range; + qos->qtg_id = dent->qtg_id; + qos->hmem_attrs = dent->hmem_attrs; + list_add_tail(&qos->list, &cxlds->qos_list); + + if (resource_size(&cxlds->ram_res) && + range_contains(&ram_range, &dent->dpa_range) && + cxlds->ram_qtg_id == -1) + cxlds->ram_qtg_id = dent->qtg_id; + else if (resource_size(&cxlds->pmem_res) && + range_contains(&pmem_range, &dent->dpa_range) && + cxlds->pmem_qtg_id == -1) + cxlds->pmem_qtg_id = dent->qtg_id; + } +} + static int cxl_switch_port_probe(struct cxl_port *port) { struct cxl_hdm *cxlhdm; @@ -234,6 +268,7 @@ static int cxl_endpoint_port_probe(struct cxl_port *port) if (rc) dev_dbg(&port->dev, "Failed to do QoS calculations\n"); + cxl_memdev_set_qtg(cxlds, &dsmas_list); dsmas_list_destroy(&dsmas_list); } From patchwork Fri Apr 7 17:14:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25942C77B70 for ; Fri, 7 Apr 2023 17:14:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230171AbjDGROP (ORCPT ); Fri, 7 Apr 2023 13:14:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230141AbjDGROO (ORCPT ); Fri, 7 Apr 2023 13:14:14 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCD4AAF30; Fri, 7 Apr 2023 10:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887652; x=1712423652; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9R7HNSKmW8NWE/s1t9PEK+QWbqYRyKyuD7hN/TDoF3A=; b=NpbTfu291TDEn0y3DJNacLTiI/VaCtIz3/REUrNfw8JxrXTvPntjqjgl SdEZ6IbiNui5nkONKxfidHCiNfYoqBqXQ8LfA2MRfIOMTKHu6I5eYvZxX 7T8sxc5EI3DQxiY0ikYGimTiOoGIO229Io+O8WernK6zO4/BEBHYqQvZ8 1eSIscFbAgmOi3ydh4DTjF4GYyK5eLGCxkG3UyUGiYDZp2kuv3Q3k3Crx ayJykMrH+0Z/VF1ucIaU6F3o8AI4dh+l47piYIcs4pi9PlH7SWMg/1EUZ VubwZTo3eyiiYdunCEfIyNygffvscxpPmDVcLirNMBbLYd5Kew3s1VHzR A==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="343037437" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="343037437" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:14:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="933648780" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="933648780" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:14:11 -0700 Subject: [PATCH 3 22/23] cxl: Export sysfs attributes for memory device QTG ID From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: Dan Williams , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:14:11 -0700 Message-ID: <168088765152.1441063.11040973621342294619.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Export qtg_id sysfs attributes for the CXL memory device. The QTG ID should show up as /sys/bus/cxl/devices/memX/ram/qtg_id for the volatile partition and /sys/bus/cxl/devices/memX/pmem/qtg_id for the persistent partition. The QTG ID is retrieved via _DSM after supplying the calculated bandwidth and latency for the entire CXL path from device to the CPU. This ID is used to match up to the root decoder QTG ID to determine which CFMWS the memory range of a hotplugged CXL mem device should be assigned under. While there may be multiple DSMAS exported by the device CDAT, the driver will only expose the first QTG ID per partition in sysfs for now. In the future when multiple QTG IDs are necessary, they can be exposed. [1] [1]: https://lore.kernel.org/linux-cxl/167571650007.587790.10040913293130712882.stgit@djiang5-mobl3.local/T/#md2a47b1ead3e1ba08f50eab29a4af1aed1d215ab Suggested-by: Dan Williams Signed-off-by: Dave Jiang --- v3: - Expand description of qtg_id. (Alison) --- Documentation/ABI/testing/sysfs-bus-cxl | 22 ++++++++++++++++++++++ drivers/cxl/core/memdev.c | 26 ++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl index 471ac9a37078..1640b6ff5429 100644 --- a/Documentation/ABI/testing/sysfs-bus-cxl +++ b/Documentation/ABI/testing/sysfs-bus-cxl @@ -28,6 +28,17 @@ Description: Payload in the CXL-2.0 specification. +What: /sys/bus/cxl/devices/memX/ram/qtg_id +Date: January, 2023 +KernelVersion: v6.4 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) Shows calculated QoS Throttling Group ID for the + "Volatile Only Capacity" DPA range. When creating regions, + the qtg_id for the memory range should match the root + decoder's qtg_id to have optimal performance. + + What: /sys/bus/cxl/devices/memX/pmem/size Date: December, 2020 KernelVersion: v5.12 @@ -38,6 +49,17 @@ Description: Payload in the CXL-2.0 specification. +What: /sys/bus/cxl/devices/memX/pmem/qtg_id +Date: January, 2023 +KernelVersion: v6.4 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) Shows calculated QoS Throttling Group ID for the + "Persistent Only Capacity" DPA range. When creating regions, + the qtg_id for the memory range should match the root + decoder's qtg_id to have optimal performance. + + What: /sys/bus/cxl/devices/memX/serial Date: January, 2022 KernelVersion: v5.18 diff --git a/drivers/cxl/core/memdev.c b/drivers/cxl/core/memdev.c index 28a05f2fe32d..04058ec5fcff 100644 --- a/drivers/cxl/core/memdev.c +++ b/drivers/cxl/core/memdev.c @@ -76,6 +76,18 @@ static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr, static struct device_attribute dev_attr_ram_size = __ATTR(size, 0444, ram_size_show, NULL); +static ssize_t ram_qtg_id_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); + struct cxl_dev_state *cxlds = cxlmd->cxlds; + + return sysfs_emit(buf, "%d\n", cxlds->ram_qtg_id); +} + +static struct device_attribute dev_attr_ram_qtg_id = + __ATTR(qtg_id, 0444, ram_qtg_id_show, NULL); + static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -89,6 +101,18 @@ static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, static struct device_attribute dev_attr_pmem_size = __ATTR(size, 0444, pmem_size_show, NULL); +static ssize_t pmem_qtg_id_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); + struct cxl_dev_state *cxlds = cxlmd->cxlds; + + return sysfs_emit(buf, "%d\n", cxlds->pmem_qtg_id); +} + +static struct device_attribute dev_attr_pmem_qtg_id = + __ATTR(qtg_id, 0444, pmem_qtg_id_show, NULL); + static ssize_t serial_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -117,11 +141,13 @@ static struct attribute *cxl_memdev_attributes[] = { static struct attribute *cxl_memdev_pmem_attributes[] = { &dev_attr_pmem_size.attr, + &dev_attr_pmem_qtg_id.attr, NULL, }; static struct attribute *cxl_memdev_ram_attributes[] = { &dev_attr_ram_size.attr, + &dev_attr_ram_qtg_id.attr, NULL, }; From patchwork Fri Apr 7 17:14:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Jiang X-Patchwork-Id: 671032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A38CC77B6C for ; Fri, 7 Apr 2023 17:14:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229587AbjDGROZ (ORCPT ); Fri, 7 Apr 2023 13:14:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230189AbjDGROU (ORCPT ); Fri, 7 Apr 2023 13:14:20 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87BFAB752; Fri, 7 Apr 2023 10:14:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680887658; x=1712423658; h=subject:from:to:cc:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=s+EXZVJ1JkQIsvOgC4hoYVQTXB71iThB2DX8P+RuUtI=; b=VpgOWcKKEtXIOMCRMAyijZEpOWJHgbGCwG6brKz1z8KikGvEBSjfdFCF WGE0SkX8vzprryHlxcPHIma8hFm5r9x+URCvkAk4lyS5J/BI3Aw16JDzJ VMK5SAOOh8QBd4XX2QqTKLQPFOr7UhG79RAQ/PLB92+r/rrNHGlnXIEKs ReZ5fDx3OQiTzfNa3D928SbEynYj0IVO078xus7t2Zn+C0eVs8xM9kbxv /VDaT/kuEnWg1WwSyo4Jl96fgb7EBdnq4o83f5AeZI3R6UOGV92oowYZL ALkiJAwoFHmnwgVkj2w/mKDdtOHYuhB6Wdgm4Y++bHXzndMnOO6zBoq/C w==; X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="343037457" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="343037457" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:14:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10673"; a="933648814" X-IronPort-AV: E=Sophos;i="5.98,327,1673942400"; d="scan'208";a="933648814" Received: from djiang5-mobl3.amr.corp.intel.com (HELO [192.168.1.177]) ([10.209.160.172]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2023 10:14:17 -0700 Subject: [PATCH 3 23/23] cxl/mem: Add debugfs output for QTG related data From: Dave Jiang To: linux-cxl@vger.kernel.org, linux-acpi@vger.kernel.org Cc: Dan Williams , dan.j.williams@intel.com, ira.weiny@intel.com, vishal.l.verma@intel.com, alison.schofield@intel.com, rafael@kernel.org, lukas@wunner.de, Jonathan.Cameron@huawei.com Date: Fri, 07 Apr 2023 10:14:17 -0700 Message-ID: <168088765758.1441063.3077950661602680130.stgit@djiang5-mobl3> In-Reply-To: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> References: <168088732996.1441063.10107817505475386072.stgit@djiang5-mobl3> User-Agent: StGit/1.5 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org Add debugfs output to /sys/kernel/debug/cxl/memX/qtgmap The debugfs attribute will dump out all the DSMAS ranges and the associated QTG ID exported by the CXL device CDAT. Suggested-by: Dan Williams Signed-off-by: Dave Jiang --- Documentation/ABI/testing/debugfs-cxl | 11 +++++++++++ drivers/cxl/mem.c | 16 ++++++++++++++++ 2 files changed, 27 insertions(+) create mode 100644 Documentation/ABI/testing/debugfs-cxl diff --git a/Documentation/ABI/testing/debugfs-cxl b/Documentation/ABI/testing/debugfs-cxl new file mode 100644 index 000000000000..0f36eeb7e59b --- /dev/null +++ b/Documentation/ABI/testing/debugfs-cxl @@ -0,0 +1,11 @@ +What: /sys/kernel/debug/cxl/memX/qtg_map +Date: Mar, 2023 +KernelVersion: v6.4 +Contact: linux-cxl@vger.kernel.org +Description: + (RO) Entries of all Device Physical Address (DPA) ranges + provided by the device Coherent Device Attributes Table (CDAT) + Device Scoped Memory Affinity Structure (DSMAS) entries with + the matching QoS Throttling Group (QTG) id calculated from the + latency and bandwidth of the CXL path from the memory device + to the CPU. diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c index 39c4b54f0715..bf2cb5a54a7f 100644 --- a/drivers/cxl/mem.c +++ b/drivers/cxl/mem.c @@ -45,6 +45,21 @@ static int cxl_mem_dpa_show(struct seq_file *file, void *data) return 0; } +static int cxl_mem_qtg_show(struct seq_file *file, void *data) +{ + struct device *dev = file->private; + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); + struct qos_prop_entry *qos; + + list_for_each_entry(qos, &cxlmd->qos_list, list) { + seq_printf(file, "%08llx-%08llx : QTG ID %u\n", + qos->dpa_range.start, qos->dpa_range.end, + qos->qtg_id); + } + + return 0; +} + static int devm_cxl_add_endpoint(struct device *host, struct cxl_memdev *cxlmd, struct cxl_dport *parent_dport) { @@ -117,6 +132,7 @@ static int cxl_mem_probe(struct device *dev) dentry = cxl_debugfs_create_dir(dev_name(dev)); debugfs_create_devm_seqfile(dev, "dpamem", dentry, cxl_mem_dpa_show); + debugfs_create_devm_seqfile(dev, "qtgmap", dentry, cxl_mem_qtg_show); rc = devm_add_action_or_reset(dev, remove_debugfs, dentry); if (rc) return rc;