From patchwork Tue Mar 3 13:28:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 194382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F350C3F2D7 for ; Tue, 3 Mar 2020 13:30:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EAA8320848 for ; Tue, 3 Mar 2020 13:30:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="HQBCxKOH" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729059AbgCCNa2 (ORCPT ); Tue, 3 Mar 2020 08:30:28 -0500 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:21163 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728068AbgCCNa2 (ORCPT ); Tue, 3 Mar 2020 08:30:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583242227; x=1614778227; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Gz6pfLMfGNZpaFxwlrBNkkqcP/vdNODzuYge/T20qto=; b=HQBCxKOHQt8BFgjPadIT56uojQmu9TgLHXc+f6XvVhqYcTnhr86m2Cm4 nQp4syemLJjggSq940AbPPli52Y6yA7wDOkHbNlBjCvJZlaAE3TN/ZQLv CK0Ov0hJ2wTN7CPn/bbbm5XrXoU8VDlAkpZFnCVbari7I3ed7s6apYbXc o=; IronPort-SDR: FHWoXQAaMNLcqoTOhhHlOg33oQbd0JRg7pw5S6KyEKhcilo89yKsSNHnrucJPkxcNe0PNN2cuY EATC/f881HJA== X-IronPort-AV: E=Sophos;i="5.70,511,1574121600"; d="scan'208";a="19279361" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-22cc717f.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 03 Mar 2020 13:30:01 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-22cc717f.us-west-2.amazon.com (Postfix) with ESMTPS id 88C55A2785; Tue, 3 Mar 2020 13:30:00 +0000 (UTC) Received: from EX13D12EUA004.ant.amazon.com (10.43.165.162) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Tue, 3 Mar 2020 13:29:49 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D12EUA004.ant.amazon.com (10.43.165.162) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 3 Mar 2020 13:29:48 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 3 Mar 2020 13:29:46 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nherr?= , "Jonathan Corbet" , Ashok Raj , Alex Williamson , Sinan Kaya , Rajat Jain , kbuild test robot Subject: [PATCH v3 03/17] PCI: Use pci_bridge_wait_for_secondary_bus after SBR Date: Tue, 3 Mar 2020 14:28:38 +0100 Message-ID: <20200303132852.13184-4-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200303132852.13184-1-stanspas@amazon.com> References: <20200303132852.13184-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org From: Stanislav Spassov So far, pci_bridge_wait_for_secondary_bus() was only invoked by PM code after (runtime) resume of devices, but it naturally makes sense for handling post-SBR waiting as well. It uses the PCI_PM_D3COLD_WAIT value (100ms), potentially overridden on a per-device basis to a lower-value, as the basis for determining how long to wait, and handles special cases such as legacy PCI devices (requiring Trhfa), and the different starting points for the waiting time depending on PCIe port speed. On PCI Express, there will be cases where the new code sleeps far less than the 1s being replaced by this patch. This should be okay, because PCI Express Base Specification Revision 5.0 Version 1.0 (May 22, 2019) in Section 6.6.1 "Conventional Reset" only notes 100ms as the minimum waiting time. After this time, the OS is permitted to issue Configuration Requests, but it is possible that the device responds with Configuration Request Retry Status (CRS) Completions, rather than Successful Completion. Returning CRS can go on for up to 1 second after a Conventional Reset (such as SBR) before the OS can consider the device broken. This additional wait is handled by pci_dev_wait. Currently, the only callchain that lands in the function modified by this patch starts at pci_bridge_secondary_bus_reset which invokes one out of two versions of pcibios_reset_secondary_bus that both end with a call to pci_reset_secondary_bus. Afterwards, pci_bridge_secondary_bus_reset always invokes pci_dev_wait. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index ac8504d75c32..c1a866f733e9 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4800,14 +4800,7 @@ void pci_reset_secondary_bus(struct pci_dev *dev) ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET; pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); - /* - * Trhfa for conventional PCI is 2^25 clock cycles. - * Assuming a minimum 33MHz clock this results in a 1s - * delay before we can consider subordinate devices to - * be re-initialized. PCIe has some ways to shorten this, - * but we don't make use of them yet. - */ - ssleep(1); + pci_bridge_wait_for_secondary_bus(dev); } void __weak pcibios_reset_secondary_bus(struct pci_dev *dev) From patchwork Tue Mar 3 13:28:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 194384 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC634C3F2C6 for ; Tue, 3 Mar 2020 13:30:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8EA4520838 for ; Tue, 3 Mar 2020 13:30:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="onAKr0+t" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728866AbgCCNaJ (ORCPT ); Tue, 3 Mar 2020 08:30:09 -0500 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:61587 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728068AbgCCNaJ (ORCPT ); Tue, 3 Mar 2020 08:30:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583242208; x=1614778208; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7WM9yX1OJkKXJJaN0Bul5znh9nhHO2FbDkre7Td3gO0=; b=onAKr0+ts16dbD2+VkV/undi3EfSN346NAQHhV56k2/qSY6xi9yzm4MI O46QCb5LmCSPhEryy4ZGPDhsQ9yh/cs9CGI322jSEpjnVW+mqft7E4GFk g5HpKEzUICUU5KKRfqO0UxSoNqc1a+m4FTtebd9p3fbwjtnezOPBN61mF I=; IronPort-SDR: mdqZXfd6/iJ+37fvCK9FrLPkkeAsXdUvL1PNWQ85fyeOBxPFqFSXt6UG5vhZxLyjggZVvqa3Ft 4x9mXvfNyDFA== X-IronPort-AV: E=Sophos;i="5.70,511,1574121600"; d="scan'208";a="30249617" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2c-397e131e.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 03 Mar 2020 13:30:06 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-397e131e.us-west-2.amazon.com (Postfix) with ESMTPS id E118AA34C5; Tue, 3 Mar 2020 13:30:04 +0000 (UTC) Received: from EX13D12EUA003.ant.amazon.com (10.43.165.147) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Tue, 3 Mar 2020 13:29:53 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D12EUA003.ant.amazon.com (10.43.165.147) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 3 Mar 2020 13:29:52 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 3 Mar 2020 13:29:49 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nherr?= , "Jonathan Corbet" , Ashok Raj , Alex Williamson , Sinan Kaya , Rajat Jain , kbuild test robot Subject: [PATCH v3 04/17] PCI: Do not override delay for D0->D3hot transition Date: Tue, 3 Mar 2020 14:28:39 +0100 Message-ID: <20200303132852.13184-5-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200303132852.13184-1-stanspas@amazon.com> References: <20200303132852.13184-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org From: Stanislav Spassov Both specifications that document mechanisms for overriding the D3hot->D0 waiting time only speak of this specific direction. Nothing is mentioned about the opposite (D*->D3hot) except for the default value of 10ms in PCI Express Base Specification r5.0 (May 22, 2019), Section 5.9 "State Transition Recovery Time Requirements". Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index c1a866f733e9..03103bb15b42 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4589,7 +4589,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe) csr &= ~PCI_PM_CTRL_STATE_MASK; csr |= PCI_D3hot; pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr); - pci_dev_d3_sleep(dev); + msleep(PCI_PM_D3_WAIT); csr &= ~PCI_PM_CTRL_STATE_MASK; csr |= PCI_D0; From patchwork Tue Mar 3 13:28:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 194383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 277B4C3F2C6 for ; Tue, 3 Mar 2020 13:30:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DF91420848 for ; Tue, 3 Mar 2020 13:30:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="FyjDdLQw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728330AbgCCNaY (ORCPT ); Tue, 3 Mar 2020 08:30:24 -0500 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:64941 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728068AbgCCNaY (ORCPT ); Tue, 3 Mar 2020 08:30:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583242224; x=1614778224; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hin5pnYLRZeeL4R19g+s7EtcXGJQv8ECPkAJzZhQii0=; b=FyjDdLQwrbKhAYzXBiPaKbEuHt2n/4UiIMxGFQUwWOOy/XA6XQFG4DmO myHbOnZRbA1Gxh7O/2nEitWh9Jpj0jP/xv0bL3a/1eE5rqqOmfSDaVTX1 ZNdRHtNl49Yar/PjfqNDjpF9EbIfivTldInIrSJDA/n0fLj/Bkf8mR+bR U=; IronPort-SDR: TABliuHYPdbFXhfkdhfm4OBVi+lLJ7DEGMqGJ/ndPVTp+aGvZNcXX1kmwiS9mFTOYiBy21KFw/ VXukI0vJUF9g== X-IronPort-AV: E=Sophos;i="5.70,511,1574121600"; d="scan'208";a="19806525" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2c-397e131e.us-west-2.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 03 Mar 2020 13:30:10 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-397e131e.us-west-2.amazon.com (Postfix) with ESMTPS id ADD57A34C9; Tue, 3 Mar 2020 13:30:08 +0000 (UTC) Received: from EX13D04EUB001.ant.amazon.com (10.43.166.190) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Tue, 3 Mar 2020 13:29:57 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D04EUB001.ant.amazon.com (10.43.166.190) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 3 Mar 2020 13:29:55 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 3 Mar 2020 13:29:53 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nherr?= , "Jonathan Corbet" , Ashok Raj , Alex Williamson , Sinan Kaya , Rajat Jain , kbuild test robot Subject: [PATCH v3 05/17] PCI: Fix handling of _DSM 8 (avoiding reset delays) Date: Tue, 3 Mar 2020 14:28:40 +0100 Message-ID: <20200303132852.13184-6-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200303132852.13184-1-stanspas@amazon.com> References: <20200303132852.13184-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org From: Stanislav Spassov On Sx Resume (boot from ACPI S5, resume from ACPI S4 or S3), platform firmware may enforce that sufficient time has passed to cover the mandatory (per PCI Express Specification) power-on reset delays of all PCI devices before control is handed over to the operating system. To avoid duplicated waiting by the OS, the firmware uses _DSM 8 in the ACPI scope of a PCI host bus to inform the OS that for the whole PCI subsystem underneath that host bridge, power-on reset delays are being covered by firmware. Previously, the kernel used _DSM 8 to override the d3cold_delay value stored in struct pci_dev to 0. However, the assumption that the delay is covered by firmware when _DSM 8 returns 1 only holds for the firmware-initiated resets as part of Sx Resume flows. If an OS has the ability to apply Conventional Reset to devices without system firmware involvement, then it still needs to adhere to the usual reset delays. Runtime Device Power Management triggering a D3cold->D0 transition is an existing example of the kernel not involving system firmware. Another example is the OS triggering Secondary Bus Reset by setting the SBR bit in the Bridge Control register of any Port device. In preparation for future work to improve post-reset delays, this patch preserves the value of d3cold_delay even when _DSM 8 returns 1. The decision not to wait after resume is instead taken at a higher level in pci_bridge_wait_for_secondary_bus based on a new parameter that indicates whether the waiting is due to an Sx Resume event. Signed-off-by: Stanislav Spassov --- drivers/pci/pci-acpi.c | 65 ++++++++++++++++++++++++++++++---------- drivers/pci/pci-driver.c | 4 +-- drivers/pci/pci.c | 11 +++++-- drivers/pci/pci.h | 2 +- include/linux/pci-acpi.h | 8 ++--- include/linux/pci.h | 4 ++- 6 files changed, 67 insertions(+), 27 deletions(-) diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c index 0c02d500158f..a8fa13d6089d 100644 --- a/drivers/pci/pci-acpi.c +++ b/drivers/pci/pci-acpi.c @@ -1120,21 +1120,35 @@ void acpi_pci_add_bus(struct pci_bus *bus) acpi_pci_slot_enumerate(bus); acpiphp_enumerate_slots(bus); - /* - * For a host bridge, check its _DSM for function 8 and if - * that is available, mark it in pci_host_bridge. + /* For a host bridge, check _DSM function 8 and if that returns 1, + * mark it in the pci_host_bridge. + * + * Function 8, "Avoid Power-On Reset Delay Duplication on Sx Resume" + * applies to the entire hierarchy below a PCI host bridge. + * If it returns one, the OS may assume that all devices in the + * hierarchy have already completed power-on reset delays + * before FW handed over control to the OS on Sx Resume (such as boot + * from ACPI S5, or resume from ACPI S4 or S3 states). + * This _DSM is applicable whether reductions in device readiness + * timing, via Readiness Notification or _DSM function 9, are available + * or not. + * + * This _DSM function is defined by the PCI Firmware Specification + * Revision 3.2 (January 26, 2015), after originally introduced by a + * draft ECN of January 28, 2014, titled "ACPI additions for FW latency + * optimizations." */ if (!pci_is_root_bus(bus)) return; obj = acpi_evaluate_dsm(ACPI_HANDLE(bus->bridge), &pci_acpi_dsm_guid, 3, - RESET_DELAY_DSM, NULL); + IGNORE_RESET_DELAY_ON_SX_RESUME_DSM, NULL); if (!obj) return; if (obj->type == ACPI_TYPE_INTEGER && obj->integer.value == 1) { bridge = pci_find_host_bridge(bus); - bridge->ignore_reset_delay = 1; + bridge->ignore_reset_delay_on_sx_resume = 1; } ACPI_FREE(obj); } @@ -1168,19 +1182,38 @@ static struct acpi_device *acpi_pci_find_companion(struct device *dev) * @handle: ACPI handle of this device * * Update the d3_delay and d3cold_delay of a PCI device from the ACPI _DSM - * control method of either the device itself or the PCI host bridge. - * - * Function 8, "Reset Delay," applies to the entire hierarchy below a PCI - * host bridge. If it returns one, the OS may assume that all devices in - * the hierarchy have already completed power-on reset delays. + * Function 9 of the device, and cache the parent host bridge's flag for + * ignoring reset delay upon Sx Resume (the flag is originally set in + * acpi_pci_add_bus through _DSM Function 8). * * Function 9, "Device Readiness Durations," applies only to the object * where it is located. It returns delay durations required after various - * events if the device requires less time than the spec requires. Delays - * from this function take precedence over the Reset Delay function. + * events if the device requires less time than the spec requires. + * Values provided by this function can only be used to lower (reduce) the + * latency required by specification or values discovered from device. + * + * This _DSM function is defined by the PCI Firmware Specification Rev 3.2 + * (January 26, 2015), after originally introduced by a draft ECN of + * January 28, 2014, titled "ACPI additions for FW latency optimizations." * - * These _DSM functions are defined by the draft ECN of January 28, 2014, - * titled "ACPI additions for FW latency optimizations." + * XXX The PCI Firmware Specification contradicts itself by stating, in addition + * to the above "can only be used to lower (reduce)", that also: + * Values must be interpreted as overriding any Configuration Ready indicator + * from hardware, whether increasing or decreasing required delays. This + * includes ignoring FRS and DRS notifications where overridden by this + * _DSM function, as well as ignoring values specified in the Readiness Time + * Reporting Extended Capability, if present. + * Meanwhile, the PCI Express Base Specification Revision 5.0 Version 1.0 + * (22 May 2019) states in section 7.9.17 Readiness Time Reporting Extended + * Capability: + * Software is permitted to issue requests upon the earliest of: + * - Receiving a Readiness Notification messages + * - Waiting the appropriate time as per relevant specifications + * - Waiting the time indicated in the associated field of this capability + * - Waiting the time defined by system software or firmware + * The kernel does not yet support Readiness Notifications, and does not yet + * use a Readiness Time Reporting capability if present, so we do not need to + * worry about the prioritization for now. */ static void pci_acpi_optimize_delay(struct pci_dev *pdev, acpi_handle handle) @@ -1189,8 +1222,8 @@ static void pci_acpi_optimize_delay(struct pci_dev *pdev, int value; union acpi_object *obj, *elements; - if (bridge->ignore_reset_delay) - pdev->d3cold_delay = 0; + pdev->ignore_reset_delay_on_sx_resume = + bridge->ignore_reset_delay_on_sx_resume; obj = acpi_evaluate_dsm(handle, &pci_acpi_dsm_guid, 3, FUNCTION_DELAY_DSM, NULL); diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index 0454ca0e4e3f..7e8ca8115c4f 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -917,7 +917,7 @@ static int pci_pm_resume_noirq(struct device *dev) pcie_pme_root_status_cleanup(pci_dev); if (!skip_bus_pm && prev_state == PCI_D3cold) - pci_bridge_wait_for_secondary_bus(pci_dev); + pci_bridge_wait_for_secondary_bus(pci_dev, true); if (pci_has_legacy_pm_support(pci_dev)) return 0; @@ -1321,7 +1321,7 @@ static int pci_pm_runtime_resume(struct device *dev) pci_pm_default_resume(pci_dev); if (prev_state == PCI_D3cold) - pci_bridge_wait_for_secondary_bus(pci_dev); + pci_bridge_wait_for_secondary_bus(pci_dev, false); if (pm && pm->runtime_resume) error = pm->runtime_resume(dev); diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 03103bb15b42..4899b12b5a38 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -2843,6 +2843,7 @@ void pci_pm_init(struct pci_dev *dev) } dev->pm_cap = pm; + dev->ignore_reset_delay_on_sx_resume = 0; dev->d3_delay = PCI_PM_D3_WAIT; dev->d3cold_delay = PCI_PM_D3COLD_WAIT; dev->bridge_d3 = pci_bridge_d3_possible(dev); @@ -4698,7 +4699,7 @@ static int pci_bus_max_d3cold_delay(const struct pci_bus *bus) * conventional PCI it means Tpvrh + Trhfa specified in PCI 3.0 section * 4.3.2. */ -void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev) +void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, bool sx_resume) { struct pci_dev *child; int delay; @@ -4723,7 +4724,11 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev) } /* Take d3cold_delay requirements into account */ - delay = pci_bus_max_d3cold_delay(dev->subordinate); + if (sx_resume && dev->ignore_reset_delay_on_sx_resume) + delay = 0; + else + delay = pci_bus_max_d3cold_delay(dev->subordinate); + if (!delay) { up_read(&pci_bus_sem); return; @@ -4800,7 +4805,7 @@ void pci_reset_secondary_bus(struct pci_dev *dev) ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET; pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); - pci_bridge_wait_for_secondary_bus(dev); + pci_bridge_wait_for_secondary_bus(dev, false); } void __weak pcibios_reset_secondary_bus(struct pci_dev *dev) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 659ab3f9042a..c4c3ba926f45 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -107,7 +107,7 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev); void pci_free_cap_save_buffers(struct pci_dev *dev); bool pci_bridge_d3_possible(struct pci_dev *dev); void pci_bridge_d3_update(struct pci_dev *dev); -void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev); +void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, bool sx_resume); static inline void pci_wakeup_event(struct pci_dev *dev) { diff --git a/include/linux/pci-acpi.h b/include/linux/pci-acpi.h index 62b7fdcc661c..99865773d65e 100644 --- a/include/linux/pci-acpi.h +++ b/include/linux/pci-acpi.h @@ -107,10 +107,10 @@ static inline void acpiphp_check_host_bridge(struct acpi_device *adev) { } #endif extern const guid_t pci_acpi_dsm_guid; -#define IGNORE_PCI_BOOT_CONFIG_DSM 0x05 -#define DEVICE_LABEL_DSM 0x07 -#define RESET_DELAY_DSM 0x08 -#define FUNCTION_DELAY_DSM 0x09 +#define IGNORE_PCI_BOOT_CONFIG_DSM 0x05 +#define DEVICE_LABEL_DSM 0x07 +#define IGNORE_RESET_DELAY_ON_SX_RESUME_DSM 0x08 +#define FUNCTION_DELAY_DSM 0x09 #else /* CONFIG_ACPI */ static inline void acpi_pci_add_bus(struct pci_bus *bus) { } diff --git a/include/linux/pci.h b/include/linux/pci.h index 3840a541a9de..22a5b7164c32 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -354,6 +354,8 @@ struct pci_dev { user sysfs */ unsigned int clear_retrain_link:1; /* Need to clear Retrain Link bit manually */ + unsigned int ignore_reset_delay_on_sx_resume:1; /* Cached value from + pci_host_bridge */ unsigned int d3_delay; /* D3->D0 transition time in ms */ unsigned int d3cold_delay; /* D3cold->D0 transition time in ms */ @@ -503,7 +505,7 @@ struct pci_host_bridge { void (*release_fn)(struct pci_host_bridge *); void *release_data; struct msi_controller *msi; - unsigned int ignore_reset_delay:1; /* For entire hierarchy */ + unsigned int ignore_reset_delay_on_sx_resume:1; /* For entire hierarchy */ unsigned int no_ext_tags:1; /* No Extended Tags */ unsigned int native_aer:1; /* OS may use PCIe AER */ unsigned int native_pcie_hotplug:1; /* OS may use PCIe hotplug */ From patchwork Tue Mar 3 13:28:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 194381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B049C3F2CD for ; Tue, 3 Mar 2020 13:30:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6226B20842 for ; Tue, 3 Mar 2020 13:30:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="piCrJYkF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729255AbgCCNax (ORCPT ); Tue, 3 Mar 2020 08:30:53 -0500 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:65027 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727862AbgCCNaw (ORCPT ); Tue, 3 Mar 2020 08:30:52 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583242253; x=1614778253; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=f1H7XOSMeRzetd59oqKV+/fQ2SI/DghKGu3MCc/evPI=; b=piCrJYkF1uj0162E7+rA/GbdoWZ+9YIVvnk+QZ+9e3qxgFFmHgwoMW77 Zc0EhzzCXjnLp0RTM1faCVly9VS4TGCqCssFeSGT18iuQ0aXvA6aEiem1 IfGCagLrwluHItXfTJ3mZaw9K56UH65lsAX9mhdD8LS19KZDHj9tomJDC 4=; IronPort-SDR: qTro6rTEC6aDeS82mdhYKYFPgA8o4/Wt6mrb58PHZGqytmTgqsY5uTldji1Y+1mfSzRMN+uX4k E2ge8zYH4DSw== X-IronPort-AV: E=Sophos;i="5.70,511,1574121600"; d="scan'208";a="19806614" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2b-4e24fd92.us-west-2.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 03 Mar 2020 13:30:51 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-4e24fd92.us-west-2.amazon.com (Postfix) with ESMTPS id 99206A2E11; Tue, 3 Mar 2020 13:30:49 +0000 (UTC) Received: from EX13D12EUC001.ant.amazon.com (10.43.164.45) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Tue, 3 Mar 2020 13:30:21 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D12EUC001.ant.amazon.com (10.43.164.45) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 3 Mar 2020 13:30:20 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 3 Mar 2020 13:30:18 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nherr?= , "Jonathan Corbet" , Ashok Raj , Alex Williamson , Sinan Kaya , Rajat Jain , kbuild test robot Subject: [PATCH v3 09/17] PCI: Generalize pci_bus_max_d3cold_delay to pci_bus_max_delay Date: Tue, 3 Mar 2020 14:28:44 +0100 Message-ID: <20200303132852.13184-10-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200303132852.13184-1-stanspas@amazon.com> References: <20200303132852.13184-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org From: Stanislav Spassov This allows determining the maximum of any of the several delay values stored in struct pci_dev. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index ba54164652cc..e4840dbf2d1c 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4669,21 +4669,26 @@ bool pcie_wait_for_link(struct pci_dev *pdev, bool active) } /* - * Find maximum D3cold delay required by all the devices on the bus. The - * spec says 100 ms, but firmware can lower it and we allow drivers to - * increase it as well. + * Find maximum delay required by all the devices on the bus after the + * given initialization event. * * Called with @pci_bus_sem locked for reading. + * + * XXX: It is not clear if this should descend down across bridges (if any) */ -static int pci_bus_max_d3cold_delay(const struct pci_bus *bus) +static int pci_bus_max_delay(const struct pci_bus *bus, + enum pci_init_event event, int default_delay) { const struct pci_dev *pdev; - int min_delay = 100; + int min_delay = default_delay; int max_delay = 0; int delay; + if (event >= PCI_INIT_EVENT_COUNT) + return default_delay; + list_for_each_entry(pdev, &bus->devices, bus_list) { - delay = pdev->delay[PCI_INIT_EVENT_RESET]; + delay = pdev->delay[event]; if (delay < min_delay) min_delay = delay; if (delay > max_delay) @@ -4728,11 +4733,13 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, bool sx_resume) return; } - /* Take d3cold_delay requirements into account */ + /* Take delay requirements into account */ if (sx_resume && dev->ignore_reset_delay_on_sx_resume) delay = 0; else - delay = pci_bus_max_d3cold_delay(dev->subordinate); + delay = pci_bus_max_delay(dev->subordinate, + PCI_INIT_EVENT_RESET, + PCI_RESET_DELAY); if (!delay) { up_read(&pci_bus_sem); From patchwork Tue Mar 3 13:28:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 194380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B8F5C3F2D8 for ; Tue, 3 Mar 2020 13:30:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 135D020838 for ; Tue, 3 Mar 2020 13:30:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="XRE/d/ls" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729423AbgCCNa4 (ORCPT ); Tue, 3 Mar 2020 08:30:56 -0500 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:16894 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727862AbgCCNaz (ORCPT ); Tue, 3 Mar 2020 08:30:55 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583242255; x=1614778255; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9xqMD1N8mF24q9FbAWh4uDveX6H7dYjjMS/baY775kQ=; b=XRE/d/lsA6moDiV/m/euLztL02OMMVvGfZc2Hb0neyX4JmJkc37G8kfY lQaYc+4+3FWkYY917sznS2UCb5Afm02tqwzOuvsO16iv++w1KN3dRF/nT TIGK8KMKMFOEz28yzdjB6m5Au3t7HxGn48oLPZLZLwb423iG8UpSFyktu A=; IronPort-SDR: QXgtJ4opWTT+69jwBRvSBdm/7QPp6001D3PSirDHQy1f7YmKzHwZO/d4FuxLVC6B0ggAAf9Y6V yxKBhzp8ipAw== X-IronPort-AV: E=Sophos;i="5.70,511,1574121600"; d="scan'208";a="19549048" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 03 Mar 2020 13:30:53 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com (Postfix) with ESMTPS id C1C4DA2FF8; Tue, 3 Mar 2020 13:30:51 +0000 (UTC) Received: from EX13D04EUA002.ant.amazon.com (10.43.165.75) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Tue, 3 Mar 2020 13:30:23 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D04EUA002.ant.amazon.com (10.43.165.75) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 3 Mar 2020 13:30:23 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 3 Mar 2020 13:30:20 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nherr?= , "Jonathan Corbet" , Ashok Raj , Alex Williamson , Sinan Kaya , Rajat Jain , kbuild test robot Subject: [PATCH v3 10/17] PCI: Use correct delay in pci_bridge_wait_for_secondary_bus Date: Tue, 3 Mar 2020 14:28:45 +0100 Message-ID: <20200303132852.13184-11-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200303132852.13184-1-stanspas@amazon.com> References: <20200303132852.13184-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org From: Stanislav Spassov PCI Express Base Specification r5.0 (May 22, 2019) details the rules for device reset in Section 6.6. For a Downstream Port that does not support Link speeds greater than 5.0 GT/s, the minimum waiting period before software is permitted to send a Configuration Request after a Conventional Reset is 100ms (PCI_RESET_DELAY). For a Downstream Port that supports Link speeds greater than 5.0 GT/s (such ports are required to be Data Link Layer Link Active Reporting capable), the period is again 100ms but measured after the link has become active (PCI_DL_UP_DELAY). The delays for both cases above can be overridden independently, and pci_bridge_wait_for_secondary_bus should use the appropriate one. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index e4840dbf2d1c..7e08c5f38190 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4736,6 +4736,12 @@ void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, bool sx_resume) /* Take delay requirements into account */ if (sx_resume && dev->ignore_reset_delay_on_sx_resume) delay = 0; + else if (pcie_downstream_port(dev) && + pcie_get_speed_cap(dev) > PCIE_SPEED_5_0GT && + dev->link_active_reporting) + delay = pci_bus_max_delay(dev->subordinate, + PCI_INIT_EVENT_DL_UP, + PCI_DL_UP_DELAY); else delay = pci_bus_max_delay(dev->subordinate, PCI_INIT_EVENT_RESET, From patchwork Tue Mar 3 13:28:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 194379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D213C3F2CD for ; Tue, 3 Mar 2020 13:31:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 01F9B206E2 for ; Tue, 3 Mar 2020 13:31:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="iJItasEo" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729090AbgCCNbB (ORCPT ); Tue, 3 Mar 2020 08:31:01 -0500 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]:35902 "EHLO smtp-fw-6001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729248AbgCCNbB (ORCPT ); Tue, 3 Mar 2020 08:31:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583242260; x=1614778260; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zo+xgKja6kmPxHfkNHFXpC45vKnXxIgJgU/su5JdReI=; b=iJItasEo8mWdwMJOJTxEE+qFmXdKSxn+m7IKVQwixjyN4QVh8i7wNV8e Onag4eCv3Op8khUPMFiu8T2HO61bXAlcAWmPmIq7xsKsHFY60HABLgdxJ sjzmV2Ggu3nlHeebNL58k86TkO6h14qIdoXQo4tS7284X+NymKuJdLur3 Y=; IronPort-SDR: 20iqOC4jwvUhe9AChWG7nY6tjhxgZd/f5FFOXXqf2Nm+sVXTWaytkmG3c72mP0SOTqHiXu9N3F MX+BuA8zLgrw== X-IronPort-AV: E=Sophos;i="5.70,511,1574121600"; d="scan'208";a="20718008" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-81e76b79.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 03 Mar 2020 13:30:58 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-81e76b79.us-west-2.amazon.com (Postfix) with ESMTPS id 6BB0FA1FAE; Tue, 3 Mar 2020 13:30:57 +0000 (UTC) Received: from EX13D04EUA003.ant.amazon.com (10.43.165.148) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Tue, 3 Mar 2020 13:30:32 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D04EUA003.ant.amazon.com (10.43.165.148) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 3 Mar 2020 13:30:31 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 3 Mar 2020 13:30:28 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nherr?= , "Jonathan Corbet" , Ashok Raj , Alex Williamson , Sinan Kaya , Rajat Jain , kbuild test robot Subject: [PATCH v3 13/17] PCI: Cache CRS Software Visibiliy in struct pci_dev Date: Tue, 3 Mar 2020 14:28:48 +0100 Message-ID: <20200303132852.13184-14-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200303132852.13184-1-stanspas@amazon.com> References: <20200303132852.13184-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org From: Stanislav Spassov Arguably, since CRS SV is a capability of Root Ports and determines how Root Ports will handle incoming CRS Completions, it makes more sense to store this flag in the struct pci bus that represents the port's secondary bus, and to cache it in any buses further down the hierarchy. However, storing the flag in struct pci_dev allows individual devices to be marked as not supporting CRS properly even when CRS SV is enabled on their parent Root Port. This way, future code that relies on the new flag will not be misled that it is safe to probe a device by relying on CRS SV to not cause platform timeouts (See the note on CRS SV on p. 553 of PCI Express Base Specification r5.0 from May 22, 2019). Note: Endpoints integrated into the Root Complex (RCiEP) may also be capable of using CRS. In that case, the software visibility is controlled using a Root Complex Register Block (RCRB). This is currently not supported by the kernel. The code introduced here would simply not set the newly introduced flag for RCiEP as for those bus->self is NULL. Signed-off-by: Stanislav Spassov --- drivers/pci/probe.c | 8 +++++++- include/linux/pci.h | 3 +++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 512cb4312ddd..eeff8a07f330 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -1080,9 +1080,11 @@ static void pci_enable_crs(struct pci_dev *pdev) /* Enable CRS Software Visibility if supported */ pcie_capability_read_word(pdev, PCI_EXP_RTCAP, &root_cap); - if (root_cap & PCI_EXP_RTCAP_CRSVIS) + if (root_cap & PCI_EXP_RTCAP_CRSVIS) { pcie_capability_set_word(pdev, PCI_EXP_RTCTL, PCI_EXP_RTCTL_CRSSVE); + pdev->crssv_enabled = true; + } } static unsigned int pci_scan_child_bus_extend(struct pci_bus *bus, @@ -2414,6 +2416,10 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus) list_add_tail(&dev->bus_list, &bus->devices); up_write(&pci_bus_sem); + /* Propagate CRS Software Visibility bit from the parent bridge */ + if (bus->self) + dev->crssv_enabled = bus->self->crssv_enabled; + ret = pcibios_add_device(dev); WARN_ON(ret < 0); diff --git a/include/linux/pci.h b/include/linux/pci.h index 16dbfff2092e..1763e98625b9 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -386,6 +386,9 @@ struct pci_dev { bit manually */ unsigned int ignore_reset_delay_on_sx_resume:1; /* Cached value from pci_host_bridge */ + unsigned int crssv_enabled:1; /* Configuration Request Retry + Status Software Visibility + enabled on (parent) bridge */ unsigned int delay[PCI_INIT_EVENT_COUNT]; /* minimum waiting time after various events in ms */ From patchwork Tue Mar 3 13:28:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 194378 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21B13C3F2D8 for ; Tue, 3 Mar 2020 13:31:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E95FE206E2 for ; Tue, 3 Mar 2020 13:31:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="Cyg+/LWT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729431AbgCCNbG (ORCPT ); Tue, 3 Mar 2020 08:31:06 -0500 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:31533 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729248AbgCCNbG (ORCPT ); Tue, 3 Mar 2020 08:31:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583242266; x=1614778266; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KTszniR1B8GtjTWJyYnBLjmuCmS7g4iBL/oem8tvWpM=; b=Cyg+/LWTlyj5KQi3zGnSJhDavgEYf9XZd/zoAKBHJIVooL2m5JEm90e8 Inqt6ihKWRxjsRnu8EOpE7CVmBUxRKtN4SCiY5EPEO9FHq2vLxhXBnS3h 16xZvQ3K/HMTNJjBP1H3HvZxQ1PfLM+FdmlnoUIlw9dFSmQYmQa9QHRU1 4=; IronPort-SDR: iS4ibjMdT1pjGJAt6zmzGDbztuJJpl2ESA6Q2gfiT559hxsCjQ56JwpiDh/gzYpd5gISU6U6BY /zATVOTrRXNA== X-IronPort-AV: E=Sophos;i="5.70,511,1574121600"; d="scan'208";a="20496350" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-81e76b79.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 03 Mar 2020 13:30:59 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-81e76b79.us-west-2.amazon.com (Postfix) with ESMTPS id 1D8C1A17D1; Tue, 3 Mar 2020 13:30:58 +0000 (UTC) Received: from EX13D12EUA001.ant.amazon.com (10.43.165.48) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Tue, 3 Mar 2020 13:30:34 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D12EUA001.ant.amazon.com (10.43.165.48) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 3 Mar 2020 13:30:33 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 3 Mar 2020 13:30:31 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nherr?= , "Jonathan Corbet" , Ashok Raj , Alex Williamson , Sinan Kaya , Rajat Jain , kbuild test robot Subject: [PATCH v3 14/17] PCI: Introduce per-device reset_ready_poll override Date: Tue, 3 Mar 2020 14:28:49 +0100 Message-ID: <20200303132852.13184-15-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200303132852.13184-1-stanspas@amazon.com> References: <20200303132852.13184-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org From: Stanislav Spassov A broken device may never become responsive after reset, hence the need for a timeout. However, waiting for too long can have unintended side effects such as triggering hung task timeouts for processes waiting on a lock held during the reset. Locks that are shared across multiple devices, such as VFIO's per-bus reflck, are especially problematic, because a single broken VF can cause hangs for processes working with other VFs on the same bus. To allow lowering the global default post-reset timeout, while still accommodating devices that require more time, this patch introduces a per-device override that can be configured via a quirk. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 5 +---- drivers/pci/pci.h | 3 +++ drivers/pci/probe.c | 2 ++ include/linux/pci.h | 3 +++ 4 files changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 5d62d4841d68..e81fd3b53bd0 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -157,9 +157,6 @@ static int __init pcie_port_pm_setup(char *str) } __setup("pcie_port_pm=", pcie_port_pm_setup); -/* Time to wait after a reset for device to become responsive */ -#define PCIE_RESET_READY_POLL_MS 60000 - /** * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children * @bus: pointer to PCI bus structure to search @@ -1033,7 +1030,7 @@ void pci_wakeup_bus(struct pci_bus *bus) static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) { const char *event_name = pci_init_event_name(event); - int timeout = PCIE_RESET_READY_POLL_MS; + int timeout = dev->reset_ready_poll_ms; int delay = 1; u32 id; diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 9b5dd6ea2f52..d8043d4dbe2f 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -113,6 +113,9 @@ int pci_bus_error_reset(struct pci_dev *dev); /* D0/D1->D2 and D2->D0 delay */ #define PCI_PM_D2_DELAY 200 +/* Time to wait after a reset for device to become responsive */ +#define PCIE_RESET_READY_POLL_MS 60000 + /** * struct pci_platform_pm_ops - Firmware PM callbacks * diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index eeff8a07f330..50b7219406ed 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2168,6 +2168,8 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus) if (!dev) return NULL; + dev->reset_ready_poll_ms = PCIE_RESET_READY_POLL_MS; + INIT_LIST_HEAD(&dev->bus_list); dev->dev.type = &pci_dev_type; dev->bus = pci_bus_get(bus); diff --git a/include/linux/pci.h b/include/linux/pci.h index 1763e98625b9..978ede89741e 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -392,6 +392,9 @@ struct pci_dev { unsigned int delay[PCI_INIT_EVENT_COUNT]; /* minimum waiting time after various events in ms */ + unsigned int reset_ready_poll_ms; /* Timeout for polling after + reset before the device is + deemed broken. */ #ifdef CONFIG_PCIEASPM struct pcie_link_state *link_state; /* ASPM link state */ From patchwork Tue Mar 3 13:28:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Spassov X-Patchwork-Id: 194377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC895C3F2CD for ; Tue, 3 Mar 2020 13:31:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8227F20842 for ; Tue, 3 Mar 2020 13:31:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="Zd7OCKNg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729433AbgCCNbH (ORCPT ); Tue, 3 Mar 2020 08:31:07 -0500 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:31561 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728976AbgCCNbG (ORCPT ); Tue, 3 Mar 2020 08:31:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583242266; x=1614778266; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j4m+zq3gvOi1+9w1LAeWgb10VdG3HMJ9oTkeRMpYQqM=; b=Zd7OCKNgGyFMB7kRyrlfKSZfVSZAsEInkHgsYUgCpo62tYZj6o0q/XzJ mlJFG9AyyL7yD78I4W6J09za5WAogQbmZbWJAieVYXC8UGVwfB7mzYqDp mrwwGZt81VI59opay6ETaSyrYepEe/fkbjGpKqmnR1CEoHQqSgFPT2SH5 E=; IronPort-SDR: Dc1bk1pyjzKQGu6X0PdY6Fk7iS1GxR3Pu664xP/NBSQFYBZjfN9oJx2FfeMV0EgRZL0SFn6HHr Wt+WE3UiseYw== X-IronPort-AV: E=Sophos;i="5.70,511,1574121600"; d="scan'208";a="20496357" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-81e76b79.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 03 Mar 2020 13:31:01 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-81e76b79.us-west-2.amazon.com (Postfix) with ESMTPS id EAAAAA18EA; Tue, 3 Mar 2020 13:30:59 +0000 (UTC) Received: from EX13D12EUC001.ant.amazon.com (10.43.164.45) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Tue, 3 Mar 2020 13:30:37 +0000 Received: from EX13MTAUEE002.ant.amazon.com (10.43.62.24) by EX13D12EUC001.ant.amazon.com (10.43.164.45) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 3 Mar 2020 13:30:35 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.62.224) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 3 Mar 2020 13:30:33 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?utf-8?q?Jan_H_=2E_Sch=C3=B6nherr?= , "Jonathan Corbet" , Ashok Raj , Alex Williamson , Sinan Kaya , Rajat Jain , kbuild test robot Subject: [PATCH v3 15/17] PCI: Refactor polling loop out of pci_dev_wait Date: Tue, 3 Mar 2020 14:28:50 +0100 Message-ID: <20200303132852.13184-16-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200303132852.13184-1-stanspas@amazon.com> References: <20200303132852.13184-1-stanspas@amazon.com> MIME-Version: 1.0 Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org From: Stanislav Spassov This patch does not (intentionally) introduce any observable difference in runtime behavior. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 71 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 50 insertions(+), 21 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index e81fd3b53bd0..f1ba931b0ead 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1027,27 +1027,28 @@ void pci_wakeup_bus(struct pci_bus *bus) pci_walk_bus(bus, pci_wakeup, NULL); } -static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) +/* + * Performs DWORD Configuration Reads at a specific offset until the value read + * (with mask applied) is not equal to bad_value. + */ +static inline int pci_dev_poll_until_not_equal(struct pci_dev *dev, int where, + u32 mask, u32 bad_value, + const char *event_name, + int timeout, int *waited, + u32 *final_value) { - const char *event_name = pci_init_event_name(event); - int timeout = dev->reset_ready_poll_ms; int delay = 1; - u32 id; + u32 value; - /* - * After reset, the device should not silently discard config - * requests, but it may still indicate that it needs more time by - * responding to them with CRS completions. The Root Port will - * generally synthesize ~0 data to complete the read (except when - * CRS SV is enabled and the read was for the Vendor ID; in that - * case it synthesizes 0x0001 data). - * - * Wait for the device to return a non-CRS completion. Read the - * Command register instead of Vendor ID so we don't have to - * contend with the CRS SV value. - */ - pci_read_config_dword(dev, PCI_COMMAND, &id); - while (id == ~0) { + if (!event_name) + event_name = ""; + + if (waited) + delay = *waited + 1; + + pci_read_config_dword(dev, where, &value); + + while ((value & mask) == bad_value) { if (delay > timeout) { pci_warn(dev, "not ready %dms after %s; giving up\n", delay - 1, event_name); @@ -1060,16 +1061,44 @@ static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) msleep(delay); delay *= 2; - pci_read_config_dword(dev, PCI_COMMAND, &id); + + pci_read_config_dword(dev, where, &value); } if (delay > 1000) - pci_info(dev, "ready %dms after %s\n", delay - 1, - event_name); + pci_info(dev, "ready %dms after %s\n", delay - 1, event_name); + + if (waited) + *waited = delay - 1; + + if (final_value) + *final_value = value; return 0; } +static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) +{ + const char *event_name = pci_init_event_name(event); + int timeout = dev->reset_ready_poll_ms; + + /* + * After reset, the device should not silently discard config + * requests, but it may still indicate that it needs more time by + * responding to them with CRS completions. The Root Port will + * generally synthesize ~0 data to complete the read (except when + * CRS SV is enabled and the read was for the Vendor ID; in that + * case it synthesizes 0x0001 data). + * + * Wait for the device to return a non-CRS completion. Read the + * Command register instead of Vendor ID so we don't have to + * contend with the CRS SV value. + */ + return pci_dev_poll_until_not_equal(dev, PCI_COMMAND, ~0, ~0, + event_name, timeout, NULL, + NULL); +} + /** * pci_power_up - Put the given device into D0 * @dev: PCI device to power up