| Message ID | 20210702045120.22855-1-aaron.ma@canonical.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/2] igc: don't rd/wr iomem when PCI is removed | expand |
On 7/2/2021 07:51, Aaron Ma wrote: > Such as dock hot plug event when runtime, for hardware implementation, > the MAC copy takes less than one second when BIOS enabled MAC passthrough. > After test on Lenovo TBT4 dock, 600ms is enough to update the > MAC address. > Otherwise ethernet fails to work. > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > --- > drivers/net/ethernet/intel/igc/igc_main.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index 606b72cb6193..c8bc5f089255 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -5468,6 +5468,9 @@ static int igc_probe(struct pci_dev *pdev, > memcpy(&hw->mac.ops, ei->mac_ops, sizeof(hw->mac.ops)); > memcpy(&hw->phy.ops, ei->phy_ops, sizeof(hw->phy.ops)); > > + if (pci_is_thunderbolt_attached(pdev) > + msleep(600); I believe it is a bit fragile. I would recommend here look for another indication instead of delay. Can we poll for a 'pci_channel_io_normal' state? (igc->pdev->error_state == pci_channel_io_normal) > + > /* Initialize skew-specific constants */ > err = ei->get_invariants(hw); > if (err) > Thanks Aaron, sasha
+ Bjorn, Krzysztof and linux-pci On Friday 02 July 2021 12:51:19 Aaron Ma wrote: > Check PCI state when rd/wr iomem. > Implement wr32 function as rd32 too. > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > Trace: > BUG: unable to handle page fault for address: 000000000000b604 > Oops: 0000 [#1] SMP NOPTI > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > Call Trace: > igc_ptp_suspend+0x6c/0xa0 [igc] > igc_ptp_stop+0x12/0x50 [igc] > igc_remove+0x7f/0x1c0 [igc] > pci_device_remove+0x3e/0xb0 > __device_release_driver+0x181/0x240 > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > --- > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > 2 files changed, 18 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index f1adf154ec4a..606b72cb6193 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) Hello! This code pattern and commit message looks like that we could use pci_dev_is_disconnected() helper function for checking if device is still connected or was disconnected. Apparently pci_dev_is_disconnected() is defined only in private header file drivers/pci/pci.h and not in public include/linux/pci.h. Aaron: can you check if pci_dev_is_disconnected() is really something which should be used and it helps you? Bjorn, Krzysztof: what do you think about lifting helper function pci_dev_is_disconnected() to be available to all drivers and not only in PCI subsystem? I think that such helper function makes driver code more readable and can be useful also for other drivers which are checking if return value is all F's. > + return 0; Aaron: should not you return all F's on error? Because few lines below in this function is returned value with all F's when PCIe link lost. > + > value = readl(&hw_addr[reg]); Anyway, this code looks to be racy. When pci_channel_io_perm_failure is set (e.g. by hotplug interrupt) after checking for pdev->error_state and prior executing above readl() then mentioned fatal error still occurs. > > /* reads should not return all F's */ > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > return value; > } > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > +{ > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > + > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return; > + > + writel((val), &hw_addr[(reg)]); > +} > + > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > { > struct igc_mac_info *mac = &adapter->hw.mac; > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > index cc174853554b..eb4be87d0e8b 100644 > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > @@ -260,13 +260,10 @@ struct igc_hw; > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > /* write operations, indexed using DWORDS */ > -#define wr32(reg, val) \ > -do { \ > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > - writel((val), &hw_addr[(reg)]); \ > -} while (0) > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > #define rd32(reg) (igc_rd32(hw, reg)) > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > #define wrfl() ((void)rd32(IGC_STATUS)) > > -- > 2.30.2 >
On 7/4/21 10:28 PM, Pali Rohár wrote: > + Bjorn, Krzysztof and linux-pci > > On Friday 02 July 2021 12:51:19 Aaron Ma wrote: >> Check PCI state when rd/wr iomem. >> Implement wr32 function as rd32 too. >> >> When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: >> Trace: >> BUG: unable to handle page fault for address: 000000000000b604 >> Oops: 0000 [#1] SMP NOPTI >> RIP: 0010:igc_rd32+0x1c/0x90 [igc] >> Call Trace: >> igc_ptp_suspend+0x6c/0xa0 [igc] >> igc_ptp_stop+0x12/0x50 [igc] >> igc_remove+0x7f/0x1c0 [igc] >> pci_device_remove+0x3e/0xb0 >> __device_release_driver+0x181/0x240 >> >> Signed-off-by: Aaron Ma <aaron.ma@canonical.com> >> --- >> drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ >> drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- >> 2 files changed, 18 insertions(+), 5 deletions(-) >> >> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c >> index f1adf154ec4a..606b72cb6193 100644 >> --- a/drivers/net/ethernet/intel/igc/igc_main.c >> +++ b/drivers/net/ethernet/intel/igc/igc_main.c >> @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) >> u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); >> u32 value = 0; >> >> + if (igc->pdev && >> + igc->pdev->error_state == pci_channel_io_perm_failure) > > Hello! This code pattern and commit message looks like that we could use > pci_dev_is_disconnected() helper function for checking if device is > still connected or was disconnected. > > Apparently pci_dev_is_disconnected() is defined only in private header > file drivers/pci/pci.h and not in public include/linux/pci.h. > > Aaron: can you check if pci_dev_is_disconnected() is really something > which should be used and it helps you? > Hi Pali, How about using pci_channel_offline instead? It's ready and also safe for frozen state, and verified on hw. > Bjorn, Krzysztof: what do you think about lifting helper function > pci_dev_is_disconnected() to be available to all drivers and not only in > PCI subsystem? > > I think that such helper function makes driver code more readable and > can be useful also for other drivers which are checking if return value > is all F's. > >> + return 0; > > Aaron: should not you return all F's on error? Because few lines below > in this function is returned value with all F's when PCIe link lost. > If you agree with the above change, I can fix it to "return -1" in v2. Thanks for your comments, Aaron >> + >> value = readl(&hw_addr[reg]); > > Anyway, this code looks to be racy. When pci_channel_io_perm_failure is > set (e.g. by hotplug interrupt) after checking for pdev->error_state and > prior executing above readl() then mentioned fatal error still occurs. > >> >> /* reads should not return all F's */ >> @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) >> return value; >> } >> >> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) >> +{ >> + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); >> + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); >> + >> + if (igc->pdev && >> + igc->pdev->error_state == pci_channel_io_perm_failure) >> + return; >> + >> + writel((val), &hw_addr[(reg)]); >> +} >> + >> int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) >> { >> struct igc_mac_info *mac = &adapter->hw.mac; >> diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h >> index cc174853554b..eb4be87d0e8b 100644 >> --- a/drivers/net/ethernet/intel/igc/igc_regs.h >> +++ b/drivers/net/ethernet/intel/igc/igc_regs.h >> @@ -260,13 +260,10 @@ struct igc_hw; >> u32 igc_rd32(struct igc_hw *hw, u32 reg); >> >> /* write operations, indexed using DWORDS */ >> -#define wr32(reg, val) \ >> -do { \ >> - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ >> - writel((val), &hw_addr[(reg)]); \ >> -} while (0) >> +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); >> >> #define rd32(reg) (igc_rd32(hw, reg)) >> +#define wr32(reg, val) (igc_wr32(hw, reg, val)) >> >> #define wrfl() ((void)rd32(IGC_STATUS)) >> >> -- >> 2.30.2 >>
On 7/4/21 1:36 PM, Neftin, Sasha wrote: > On 7/2/2021 07:51, Aaron Ma wrote: >> Such as dock hot plug event when runtime, for hardware implementation, >> the MAC copy takes less than one second when BIOS enabled MAC passthrough. >> After test on Lenovo TBT4 dock, 600ms is enough to update the >> MAC address. >> Otherwise ethernet fails to work. >> >> Signed-off-by: Aaron Ma <aaron.ma@canonical.com> >> --- >> drivers/net/ethernet/intel/igc/igc_main.c | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c >> index 606b72cb6193..c8bc5f089255 100644 >> --- a/drivers/net/ethernet/intel/igc/igc_main.c >> +++ b/drivers/net/ethernet/intel/igc/igc_main.c >> @@ -5468,6 +5468,9 @@ static int igc_probe(struct pci_dev *pdev, >> memcpy(&hw->mac.ops, ei->mac_ops, sizeof(hw->mac.ops)); >> memcpy(&hw->phy.ops, ei->phy_ops, sizeof(hw->phy.ops)); >> + if (pci_is_thunderbolt_attached(pdev) > + msleep(600); > I believe it is a bit fragile. I would recommend here look for another indication instead of delay. Can we poll for a 'pci_channel_io_normal' state? (igc->pdev->error_state == pci_channel_io_normal) Hi sasha, In this situation, the error_state is always pci_channel_io_normal. The delay is necessary. Refer to "627239-Intel® Ethernet Controller I225-MAC-Address-Passthrough-rev1.2" section "3.5 Timing Considerations": "For hardware implementation, when the operating system is already running, the MAC copy must happen not more than one second after TBT link is established. the I225 Windows driver prevents the operating system from detecting the I225 for one second. This allows enough time for hardware to update the MAC address." Thanks sasha, Aaron >> + >> /* Initialize skew-specific constants */ >> err = ei->get_invariants(hw); >> if (err) >> > Thanks Aaron, > sasha
On Fri, 2 Jul 2021 at 14:53, Aaron Ma <aaron.ma@canonical.com> wrote: > > Check PCI state when rd/wr iomem. > Implement wr32 function as rd32 too. > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > Trace: > BUG: unable to handle page fault for address: 000000000000b604 > Oops: 0000 [#1] SMP NOPTI > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > Call Trace: > igc_ptp_suspend+0x6c/0xa0 [igc] > igc_ptp_stop+0x12/0x50 [igc] > igc_remove+0x7f/0x1c0 [igc] > pci_device_remove+0x3e/0xb0 > __device_release_driver+0x181/0x240 > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > --- Drive-by, but won't this add a lot of overhead on every register access? has this been benchmarked with lots of small network transfers or anything? Dave. > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > 2 files changed, 18 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index f1adf154ec4a..606b72cb6193 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return 0; > + > value = readl(&hw_addr[reg]); > > /* reads should not return all F's */ > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > return value; > } > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > +{ > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > + > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return; > + > + writel((val), &hw_addr[(reg)]); > +} > + > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > { > struct igc_mac_info *mac = &adapter->hw.mac; > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > index cc174853554b..eb4be87d0e8b 100644 > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > @@ -260,13 +260,10 @@ struct igc_hw; > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > /* write operations, indexed using DWORDS */ > -#define wr32(reg, val) \ > -do { \ > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > - writel((val), &hw_addr[(reg)]); \ > -} while (0) > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > #define rd32(reg) (igc_rd32(hw, reg)) > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > #define wrfl() ((void)rd32(IGC_STATUS)) > > -- > 2.30.2 >
On 7/5/2021 10:38, Aaron Ma wrote: > > > On 7/4/21 1:36 PM, Neftin, Sasha wrote: >> On 7/2/2021 07:51, Aaron Ma wrote: >>> Such as dock hot plug event when runtime, for hardware implementation, >>> the MAC copy takes less than one second when BIOS enabled MAC >>> passthrough. >>> After test on Lenovo TBT4 dock, 600ms is enough to update the >>> MAC address. >>> Otherwise ethernet fails to work. >>> >>> Signed-off-by: Aaron Ma <aaron.ma@canonical.com> >>> --- >>> drivers/net/ethernet/intel/igc/igc_main.c | 3 +++ >>> 1 file changed, 3 insertions(+) >>> >>> diff --git a/drivers/net/ethernet/intel/igc/igc_main.c >>> b/drivers/net/ethernet/intel/igc/igc_main.c >>> index 606b72cb6193..c8bc5f089255 100644 >>> --- a/drivers/net/ethernet/intel/igc/igc_main.c >>> +++ b/drivers/net/ethernet/intel/igc/igc_main.c >>> @@ -5468,6 +5468,9 @@ static int igc_probe(struct pci_dev *pdev, >>> memcpy(&hw->mac.ops, ei->mac_ops, sizeof(hw->mac.ops)); >>> memcpy(&hw->phy.ops, ei->phy_ops, sizeof(hw->phy.ops)); >>> + if (pci_is_thunderbolt_attached(pdev) > + msleep(600); >> I believe it is a bit fragile. I would recommend here look for another >> indication instead of delay. Can we poll for a 'pci_channel_io_normal' >> state? (igc->pdev->error_state == pci_channel_io_normal) > > Hi sasha, > In this situation, the error_state is always pci_channel_io_normal. Ok. > The delay is necessary. > > Refer to "627239-Intel® Ethernet Controller > I225-MAC-Address-Passthrough-rev1.2" > section "3.5 > Timing Considerations": Hello Aaron, Thanks to point me on this document. I see... This is recommendation for Windows driver. Anyway, "delay" approach is error-prone. We need rather ask for MNG FW confirmation (message) that MAC address is copied. Can we call (in case we know that MNG FW copied MAC address): igc_rar_set (method from igc_mac.c), update the mac.addr and then perform": memcpy(netdev->dev_addr, hw->mac.addr, netdev->addr_len);? > "For hardware implementation, > > when the operating system is already running, the MAC copy must happen > not more than one > > second after TBT link is established. > the I225 Windows driver prevents the operating > > system from detecting the I225 for one second. This allows enough time > for hardware to update the > > MAC address." > > Thanks sasha, > Aaron > >>> + >>> /* Initialize skew-specific constants */ >>> err = ei->get_invariants(hw); >>> if (err) >>> >> Thanks Aaron, >> sasha Sasha
Hi Pali, [...] > Aaron: can you check if pci_dev_is_disconnected() is really something > which should be used and it helps you? While having a closer look, I've noticed that quite a few of the network drivers handle this somewhat, as I see that a lot of them have some sort of I/O error handles set where a check for "pci_channel_io_perm_failure" seem to be having place. This is also true for this driver looking at the igc_io_error_detected(). Is this not working for the igc driver? Or is this for something completely different? Having said all that, I am not an expert in network drivers, so pardon me if I am asking about something completely different, and I apologise if that is the case. > Bjorn, Krzysztof: what do you think about lifting helper function > pci_dev_is_disconnected() to be available to all drivers and not only in > PCI subsystem? No objections from me, if we believe it's useful and that it might encourage people to use a common API. Currently, I can see at least five potential users of this helper. Krzysztof
On 7/5/21 3:47 PM, Dave Airlie wrote: > Drive-by, but won't this add a lot of overhead on every register > access? has this been benchmarked with lots of small network transfers > or anything? > iperf3 is tested, the result is the same as before. Due to the registers are rd/wr even after error_handler and remove. Didn't find better fix. Please let me know if you have any idea. Thanks, Aaron > Dave.
On 7/5/21 7:54 PM, Neftin, Sasha wrote: > Hello Aaron, Thanks to point me on this document. I see... This is recommendation for Windows driver. Anyway, "delay" approach is error-prone. We need rather ask for MNG FW confirmation (message) that MAC address is copied. > Can we call (in case we know that MNG FW copied MAC address): > igc_rar_set (method from igc_mac.c), update the mac.addr and then perform": memcpy(netdev->dev_addr, hw->mac.addr, netdev->addr_len);? Without delay, after igc_rar_set, the MAC address is all 0. The MAC addr is the from dock instead of MAC passthrough with the original driver. Thanks, Aaron
On Tuesday 06 July 2021 01:02:12 Krzysztof Wilczyński wrote: > Hi Pali, > > [...] > > Aaron: can you check if pci_dev_is_disconnected() is really something > > which should be used and it helps you? > > While having a closer look, I've noticed that quite a few of the network > drivers handle this somewhat, as I see that a lot of them have some sort > of I/O error handles set where a check for "pci_channel_io_perm_failure" > seem to be having place. This is also true for this driver looking at > the igc_io_error_detected(). > > Is this not working for the igc driver? Or is this for something > completely different? I guess that this callback is called when Bridge receive some kind of fatal error. Non-AER-aware bridges probably do not have to inform system that error happened and kernel would not call this callback. So I guess it depends on to which "machine" you need this network adapter. So in my opinion this callback is there for PCI subsystem to inform driver that error happened and let driver to do any hw specific recovery if it is possible. But I think problem described here can be slightly different. It is needed to check if device is still alive or was disconnected. > Having said all that, I am not an expert in network drivers, so pardon > me if I am asking about something completely different, and I apologise > if that is the case. > > > Bjorn, Krzysztof: what do you think about lifting helper function > > pci_dev_is_disconnected() to be available to all drivers and not only in > > PCI subsystem? > > No objections from me, if we believe it's useful and that it might > encourage people to use a common API. Currently, I can see at least > five potential users of this helper. > > Krzysztof
On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote: > Check PCI state when rd/wr iomem. > Implement wr32 function as rd32 too. > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > Trace: > BUG: unable to handle page fault for address: 000000000000b604 > Oops: 0000 [#1] SMP NOPTI > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > Call Trace: > igc_ptp_suspend+0x6c/0xa0 [igc] > igc_ptp_stop+0x12/0x50 [igc] > igc_remove+0x7f/0x1c0 [igc] > pci_device_remove+0x3e/0xb0 > __device_release_driver+0x181/0x240 > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > --- > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > 2 files changed, 18 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index f1adf154ec4a..606b72cb6193 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return 0; I don't think this solves the problem. - Driver calls igc_rd32(). - "if (pci_channel_io_perm_failure)" evaluates to false (error_state does not indicate an error). - Device is unplugged. - igc_rd32() calls readl(), which performs MMIO read, which fails because the device is no longer present. readl() returns ~0 on most platforms. - Same page fault occurs. The only way is to check *after* the MMIO read to see whether an error occurred. On most platforms that means checking for ~0 data. If you see that, a PCI error *may* have occurred. If you know that ~0 can never be valid, e.g., if you're reading a register where ~0 is not a valid value, you know for sure that an error has occurred. If ~0 might be a valid value, e.g., if you're reading a buffer that contains arbitrary data, you have to look harder. You might read a register than cannot contain ~0, and see if you get the data you expect. Or you might read the Vendor ID or something from config space. > value = readl(&hw_addr[reg]); > > /* reads should not return all F's */ > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > return value; > } > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > +{ > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > + > + if (igc->pdev && > + igc->pdev->error_state == pci_channel_io_perm_failure) > + return; > + > + writel((val), &hw_addr[(reg)]); > +} > + > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > { > struct igc_mac_info *mac = &adapter->hw.mac; > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > index cc174853554b..eb4be87d0e8b 100644 > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > @@ -260,13 +260,10 @@ struct igc_hw; > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > /* write operations, indexed using DWORDS */ > -#define wr32(reg, val) \ > -do { \ > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > - writel((val), &hw_addr[(reg)]); \ > -} while (0) > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > #define rd32(reg) (igc_rd32(hw, reg)) > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > #define wrfl() ((void)rd32(IGC_STATUS)) > > -- > 2.30.2 >
On Tuesday 06 July 2021 15:12:41 Bjorn Helgaas wrote: > On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote: > > Check PCI state when rd/wr iomem. > > Implement wr32 function as rd32 too. > > > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > > Trace: > > BUG: unable to handle page fault for address: 000000000000b604 > > Oops: 0000 [#1] SMP NOPTI > > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > > Call Trace: > > igc_ptp_suspend+0x6c/0xa0 [igc] > > igc_ptp_stop+0x12/0x50 [igc] > > igc_remove+0x7f/0x1c0 [igc] > > pci_device_remove+0x3e/0xb0 > > __device_release_driver+0x181/0x240 > > > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > > --- > > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > > 2 files changed, 18 insertions(+), 5 deletions(-) > > > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > > index f1adf154ec4a..606b72cb6193 100644 > > --- a/drivers/net/ethernet/intel/igc/igc_main.c > > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > u32 value = 0; > > > > + if (igc->pdev && > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > + return 0; > > I don't think this solves the problem. > > - Driver calls igc_rd32(). > > - "if (pci_channel_io_perm_failure)" evaluates to false (error_state > does not indicate an error). > > - Device is unplugged. > > - igc_rd32() calls readl(), which performs MMIO read, which fails > because the device is no longer present. readl() returns ~0 on > most platforms. > > - Same page fault occurs. Hi Bjorn! I think that backtrace show that this error happens when PCIe hotplug get interrupt that device was unplugged and PCIe hotplug code calls remove/unbind procedure to stop unplugged driver. And in this case really does not make sense to try issuing MMIO read, device is already unplugged. I looked that PCIe hotplug driver calls pci_dev_set_disconnected() when this unplug interrupt happens and pci_dev_set_disconnected() just sets pci_channel_io_perm_failure flag. drivers/pci/pci.h provides function pci_dev_is_disconnected() which checks if that flag pci_channel_io_perm_failure is set. So I think that pci_dev_is_disconnected() is useful and could be exported also to drivers (like this one) so they can check if pci_dev_set_disconnected() was called in past and PCI driver is now in unbind/cleanup/remove state because PCIe device is already disconnected and not accessible anymore. But maybe this check should be on other place in driver unbound procedure and not in general MMIO read function? > The only way is to check *after* the MMIO read to see whether an error > occurred. On most platforms that means checking for ~0 data. If you > see that, a PCI error *may* have occurred. > > If you know that ~0 can never be valid, e.g., if you're reading a > register where ~0 is not a valid value, you know for sure that an > error has occurred. > > If ~0 might be a valid value, e.g., if you're reading a buffer that > contains arbitrary data, you have to look harder. You might read a > register than cannot contain ~0, and see if you get the data you > expect. Or you might read the Vendor ID or something from config > space. > > > value = readl(&hw_addr[reg]); > > > > /* reads should not return all F's */ > > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > return value; > > } > > > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > > +{ > > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > + > > + if (igc->pdev && > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > + return; > > + > > + writel((val), &hw_addr[(reg)]); > > +} > > + > > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > > { > > struct igc_mac_info *mac = &adapter->hw.mac; > > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > > index cc174853554b..eb4be87d0e8b 100644 > > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > > @@ -260,13 +260,10 @@ struct igc_hw; > > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > > > /* write operations, indexed using DWORDS */ > > -#define wr32(reg, val) \ > > -do { \ > > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > > - writel((val), &hw_addr[(reg)]); \ > > -} while (0) > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > > > #define rd32(reg) (igc_rd32(hw, reg)) > > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > > > #define wrfl() ((void)rd32(IGC_STATUS)) > > > > -- > > 2.30.2 > >
On Wed, Jul 07, 2021 at 11:53:37PM +0200, Pali Rohár wrote: > On Tuesday 06 July 2021 15:12:41 Bjorn Helgaas wrote: > > On Fri, Jul 02, 2021 at 12:51:19PM +0800, Aaron Ma wrote: > > > Check PCI state when rd/wr iomem. > > > Implement wr32 function as rd32 too. > > > > > > When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: > > > Trace: > > > BUG: unable to handle page fault for address: 000000000000b604 > > > Oops: 0000 [#1] SMP NOPTI > > > RIP: 0010:igc_rd32+0x1c/0x90 [igc] > > > Call Trace: > > > igc_ptp_suspend+0x6c/0xa0 [igc] > > > igc_ptp_stop+0x12/0x50 [igc] > > > igc_remove+0x7f/0x1c0 [igc] > > > pci_device_remove+0x3e/0xb0 > > > __device_release_driver+0x181/0x240 > > > > > > Signed-off-by: Aaron Ma <aaron.ma@canonical.com> > > > --- > > > drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ > > > drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- > > > 2 files changed, 18 insertions(+), 5 deletions(-) > > > > > > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > > > index f1adf154ec4a..606b72cb6193 100644 > > > --- a/drivers/net/ethernet/intel/igc/igc_main.c > > > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > > > @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > > u32 value = 0; > > > > > > + if (igc->pdev && > > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > > + return 0; > > > > I don't think this solves the problem. > > > > - Driver calls igc_rd32(). > > > > - "if (pci_channel_io_perm_failure)" evaluates to false (error_state > > does not indicate an error). > > > > - Device is unplugged. > > > > - igc_rd32() calls readl(), which performs MMIO read, which fails > > because the device is no longer present. readl() returns ~0 on > > most platforms. > > > > - Same page fault occurs. > > Hi Bjorn! I think that backtrace show that this error happens when PCIe > hotplug get interrupt that device was unplugged and PCIe hotplug code > calls remove/unbind procedure to stop unplugged driver. > > And in this case really does not make sense to try issuing MMIO read, > device is already unplugged. > > I looked that PCIe hotplug driver calls pci_dev_set_disconnected() when > this unplug interrupt happens and pci_dev_set_disconnected() just sets > pci_channel_io_perm_failure flag. > > drivers/pci/pci.h provides function pci_dev_is_disconnected() which > checks if that flag pci_channel_io_perm_failure is set. > > So I think that pci_dev_is_disconnected() is useful and could be > exported also to drivers (like this one) so they can check if > pci_dev_set_disconnected() was called in past and PCI driver is now in > unbind/cleanup/remove state because PCIe device is already disconnected > and not accessible anymore. > > But maybe this check should be on other place in driver unbound > procedure and not in general MMIO read function? If we add the check as proposed in this patch, I think people will read it and think this is the correct way to avoid MMIO errors. It does happen to avoid some MMIO errors, but it cannot avoid them all, so it's not a complete solution and it gives a false sense of security. A complete solution requires a test *after* the MMIO read. If you have the test after the read, you don't really need one before. Sure, testing before means you can avoid one MMIO read failure in some cases. But avoiding that failure costs quite a lot in code clutter. > > The only way is to check *after* the MMIO read to see whether an error > > occurred. On most platforms that means checking for ~0 data. If you > > see that, a PCI error *may* have occurred. > > > > If you know that ~0 can never be valid, e.g., if you're reading a > > register where ~0 is not a valid value, you know for sure that an > > error has occurred. > > > > If ~0 might be a valid value, e.g., if you're reading a buffer that > > contains arbitrary data, you have to look harder. You might read a > > register than cannot contain ~0, and see if you get the data you > > expect. Or you might read the Vendor ID or something from config > > space. > > > > > value = readl(&hw_addr[reg]); > > > > > > /* reads should not return all F's */ > > > @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) > > > return value; > > > } > > > > > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) > > > +{ > > > + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > > > + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > > > + > > > + if (igc->pdev && > > > + igc->pdev->error_state == pci_channel_io_perm_failure) > > > + return; > > > + > > > + writel((val), &hw_addr[(reg)]); > > > +} > > > + > > > int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) > > > { > > > struct igc_mac_info *mac = &adapter->hw.mac; > > > diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h > > > index cc174853554b..eb4be87d0e8b 100644 > > > --- a/drivers/net/ethernet/intel/igc/igc_regs.h > > > +++ b/drivers/net/ethernet/intel/igc/igc_regs.h > > > @@ -260,13 +260,10 @@ struct igc_hw; > > > u32 igc_rd32(struct igc_hw *hw, u32 reg); > > > > > > /* write operations, indexed using DWORDS */ > > > -#define wr32(reg, val) \ > > > -do { \ > > > - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ > > > - writel((val), &hw_addr[(reg)]); \ > > > -} while (0) > > > +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); > > > > > > #define rd32(reg) (igc_rd32(hw, reg)) > > > +#define wr32(reg, val) (igc_wr32(hw, reg, val)) > > > > > > #define wrfl() ((void)rd32(IGC_STATUS)) > > > > > > -- > > > 2.30.2 > > >
On Thu, Jul 8, 2021 at 8:40 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > If we add the check as proposed in this patch, I think people will > read it and think this is the correct way to avoid MMIO errors. It > does happen to avoid some MMIO errors, but it cannot avoid them all, > so it's not a complete solution and it gives a false sense of > security. I think it's helpful to classify MMIO errors as either benign or poisonous with the poison MMIOs causing some kind of crash. Most of the discussions about pci_dev_is_disconnected(), including this one, seem to stem from people trying to use it to avoid the poison case. I agree that using pci_dev_is_disconnected() that way is hacky and doesn't really fix the problem, but considering poison MMIOs usually stem from broken hardware or firmware maybe we should allow it anyway. We can't do anything better and it's an improvement compared to crashing. > A complete solution requires a test *after* the MMIO read. If you > have the test after the read, you don't really need one before. Sure, > testing before means you can avoid one MMIO read failure in some > cases. But avoiding that failure costs quite a lot in code clutter. It's not that much clutter if the checks are buried in the MMIO helpers which most drivers define. Speaking of which: > u32 igc_rd32(struct igc_hw *hw, u32 reg) > { > struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); > u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); > u32 value = 0; > > value = readl(&hw_addr[reg]); > > /* reads should not return all F's */ > if (!(~value) && (!reg || !(~readl(hw_addr)))) { > struct net_device *netdev = igc->netdev; > > hw->hw_addr = NULL; > netif_device_detach(netdev); > netdev_err(netdev, "PCIe link lost, device now detached\n"); > WARN(pci_device_is_present(igc->pdev), > "igc: Failed to read reg 0x%x!\n", reg); > } > > return value; > } I think I found where that page fault is coming from. I wonder if we should provide drivers some way of invoking the error recovery mechanisms manually or even just flagging itself as broken. Right now even if the driver bothers with synchronous error detection the driver can't really do anything other than parking itself and hoping AER/EEH recovery kicks in. Oliver
On 7/6/2021 09:46, Aaron Ma wrote: > > On 7/5/21 7:54 PM, Neftin, Sasha wrote: >> Hello Aaron, Thanks to point me on this document. I see... This is >> recommendation for Windows driver. Anyway, "delay" approach is >> error-prone. We need rather ask for MNG FW confirmation (message) that >> MAC address is copied. >> Can we call (in case we know that MNG FW copied MAC address): >> igc_rar_set (method from igc_mac.c), update the mac.addr and then >> perform": memcpy(netdev->dev_addr, hw->mac.addr, netdev->addr_len);? > > Without delay, after igc_rar_set, the MAC address is all 0. > The MAC addr is the from dock instead of MAC passthrough with the > original driver. I would to like suggest checking the following direction: 1. principal question: can we update the netdev device address after it is already set during probe? I meant perform another: memcpy(netdev->dev_addr, hw->mac.addr, netdev->addr_len) up to demand 2. We need to work with Intel's firmware engineer/group and define the message/event: MAC addressis changed and should be updated. As I know MNG FW updates shadow registers. Since shadow registers are different from RAL/RAH registers - it could be a notification that the MAC address changed. Let's check it. > > Thanks, > Aaron
On Thu, Jul 08, 2021 at 12:04:02PM +1000, Oliver O'Halloran wrote: > On Thu, Jul 8, 2021 at 8:40 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > If we add the check as proposed in this patch, I think people will > > read it and think this is the correct way to avoid MMIO errors. It > > does happen to avoid some MMIO errors, but it cannot avoid them all, > > so it's not a complete solution and it gives a false sense of > > security. > > I think it's helpful to classify MMIO errors as either benign or > poisonous with the poison MMIOs causing some kind of crash. Most of > the discussions about pci_dev_is_disconnected(), including this one, > seem to stem from people trying to use it to avoid the poison case. I > agree that using pci_dev_is_disconnected() that way is hacky and > doesn't really fix the problem, but considering poison MMIOs usually > stem from broken hardware or firmware maybe we should allow it > anyway. We can't do anything better and it's an improvement compared > to crashing. Apologies for rehashing what's probably obvious to everybody but me. I'm trying to get a better handle on benign vs poisonous errors. MMIO means CPU reads or writes to the device. In PCI, writes are posted and don't receive a response, so a driver will never see writel() return an error (although an error may be reported asynchronously via AER or similar). So I think we're mostly talking about CPU reads here. We expect a PCI response containing the data. Sometimes there's no response or an error response. The behavior of the host bridge in these error cases is not defined by PCI, so what the CPU sees is not consistent across platforms. In some cases, the bridge handles this as a catastrophic error that forces a system restart. But in most cases, at least on x86, the bridge logs an error and fabricates ~0 data so the CPU read can complete. Then it's up to software to recognize that an error occurred and decide what to do about it. Is this a benign or a poisonous error? I'd say this is a benign error. It certainly can't be ignored, but as long as the driver recognizes the error, it should be able to deal with it without crashing the whole system and forcing a restart. Bjorn
On 7/8/21 12:24 PM, Neftin, Sasha wrote: > I would to like suggest checking the following direction: > 1. principal question: can we update the netdev device address after it is already set during probe? I meant perform another: > memcpy(netdev->dev_addr, hw->mac.addr, netdev->addr_len) up to demand Updating MAC addr may work. Even at the end of probe, it still got the wrong MAC address, delay is still needed. Aaron > 2. We need to work with Intel's firmware engineer/group and define the message/event: MAC addressis changed and should be updated. > As I know MNG FW updates shadow registers. Since shadow registers are different from RAL/RAH registers - it could be a notification that the MAC address changed. Let's check it.
Hello, Aaron, Sasha, On 13/07/2021 16:45, Aaron Ma wrote: > > On 7/8/21 12:24 PM, Neftin, Sasha wrote: >> I would to like suggest checking the following direction: >> 1. principal question: can we update the netdev device address after >> it is already set during probe? I meant perform another: >> memcpy(netdev->dev_addr, hw->mac.addr, netdev->addr_len) up to demand > > Updating MAC addr may work. > Even at the end of probe, it still got the wrong MAC address, delay is > still needed. > > Aaron > >> 2. We need to work with Intel's firmware engineer/group and define the >> message/event: MAC addressis changed and should be updated. >> As I know MNG FW updates shadow registers. Since shadow registers are >> different from RAL/RAH registers - it could be a notification that the >> MAC address changed. Let's check it. There is an interrupt which the FW can issue to the driver to indicate that MAC address has been changed. At that point the driver can update the MAC in its internal structures. The important question is - is there away to update the OS structures at that point so that the MAC address change propagates through all the network stack. Some network stacks do not support such an update, except during device initialization (probe), so in such environments a delay is the only workaround, and it is a problematic one as we know. If we find a mechanism by which the device driver can tell the Linux network stack - "My MAC address has changed; please update it", we can implement it differently, and not need this delay. Who can help us with this inquiry? Thanks, --Dima --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
On Fri, Jul 9, 2021 at 1:45 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > *snip* > > Apologies for rehashing what's probably obvious to everybody but me. > I'm trying to get a better handle on benign vs poisonous errors. > > MMIO means CPU reads or writes to the device. In PCI, writes are > posted and don't receive a response, so a driver will never see > writel() return an error (although an error may be reported > asynchronously via AER or similar). > > So I think we're mostly talking about CPU reads here. We expect a PCI > response containing the data. Sometimes there's no response or an > error response. The behavior of the host bridge in these error cases > is not defined by PCI, so what the CPU sees is not consistent across > platforms. In some cases, the bridge handles this as a catastrophic > error that forces a system restart. > > But in most cases, at least on x86, the bridge logs an error and > fabricates ~0 data so the CPU read can complete. Then it's up to > software to recognize that an error occurred and decide what to do > about it. Is this a benign or a poisonous error? > > I'd say this is a benign error. It certainly can't be ignored, but as > long as the driver recognizes the error, it should be able to deal > with it without crashing the whole system and forcing a restart. I was thinking more in terms of what the driver author sees rather than what's happening on the CPU side. The crash seen in the OP appears to be because the code is "doing an MMIO." However, the reasons for the crash have nothing to do with the actual mechanics of the operation (which should be benign). The point I was making is that the pattern of: if (is_disconnected()) return failure; return do_mmio_read(addr); does have some utility as a last-ditch attempt to prevent crashes in the face of obnoxious bridges or bad hardware. Granted, that should be a platform concern rather than something that should ever appear in driver code, but considering drivers open-code readl()/writel() calls there's not really any place to put that sort of workaround. That all said, the case in the OP is due to an entirely avoidable driver bug and that sort of hack is absolutely the wrong thing to do. Oliver
On Monday 19 July 2021 02:31:10 Oliver O'Halloran wrote: > On Fri, Jul 9, 2021 at 1:45 AM Bjorn Helgaas <helgaas@kernel.org> wrote: > > > > *snip* > > > > Apologies for rehashing what's probably obvious to everybody but me. > > I'm trying to get a better handle on benign vs poisonous errors. > > > > MMIO means CPU reads or writes to the device. In PCI, writes are > > posted and don't receive a response, so a driver will never see > > writel() return an error (although an error may be reported > > asynchronously via AER or similar). > > > > So I think we're mostly talking about CPU reads here. We expect a PCI > > response containing the data. Sometimes there's no response or an > > error response. The behavior of the host bridge in these error cases > > is not defined by PCI, so what the CPU sees is not consistent across > > platforms. In some cases, the bridge handles this as a catastrophic > > error that forces a system restart. > > > > But in most cases, at least on x86, the bridge logs an error and > > fabricates ~0 data so the CPU read can complete. Then it's up to > > software to recognize that an error occurred and decide what to do > > about it. Is this a benign or a poisonous error? > > > > I'd say this is a benign error. It certainly can't be ignored, but as > > long as the driver recognizes the error, it should be able to deal > > with it without crashing the whole system and forcing a restart. > > I was thinking more in terms of what the driver author sees rather > than what's happening on the CPU side. The crash seen in the OP > appears to be because the code is "doing an MMIO." However, the > reasons for the crash have nothing to do with the actual mechanics of > the operation (which should be benign). The point I was making is that > the pattern of: > > if (is_disconnected()) > return failure; > return do_mmio_read(addr); > > does have some utility as a last-ditch attempt to prevent crashes in > the face of obnoxious bridges or bad hardware. Granted, that should be > a platform concern rather than something that should ever appear in > driver code, but considering drivers open-code readl()/writel() calls > there's not really any place to put that sort of workaround. > > That all said, the case in the OP is due to an entirely avoidable > driver bug and that sort of hack is absolutely the wrong thing to do. > > Oliver And do we have some solution for this kind of issue? There are more PCIe controllers / platforms which do not like MMIO read/write operation when card / link is not connected. If we do not provide a way how to solve these problems then we can expect that people would just hack ethernet / wifi / ... device drivers which are currently crashing by patches like in this thread. Maybe PCI subsystem could provide wrapper function which implements above pattern and which can be used by device drivers?
On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote: > > And do we have some solution for this kind of issue? There are more PCIe > controllers / platforms which do not like MMIO read/write operation when > card / link is not connected. Do you have some actual examples? The few times I've seen those crashes were due to broken firmware-first error handling. The AER notifications would be escalated into some kind of ACPI error which the kernel didn't have a good way of dealing with so it panicked instead. Assuming it is a real problem then as Bjorn pointed out this sort of hack doesn't really fix the issue because hotplug and AER notifications are fundamentally asynchronous. If the driver is actively using the device when the error / removal happens then the pci_dev_is_disconnected() check will pass and the MMIO will go through. If the MMIO is poisonous because of dumb hardware then this sort of hack will only paper over the issue. > If we do not provide a way how to solve these problems then we can > expect that people would just hack ethernet / wifi / ... device drivers > which are currently crashing by patches like in this thread. > > Maybe PCI subsystem could provide wrapper function which implements > above pattern and which can be used by device drivers? We could do that and I think there was a proposal to add some pci_readl(pdev, <addr>) style wrappers at one point. On powerpc there's hooks in the arch provided MMIO functions to detect error responses and kick off the error handling machinery when a problem is detected. Those hooks are mainly there to help the platform detect errors though and they don't make life much easier for drivers. Due to locking concerns the driver's .error_detected() callback cannot be called in the MMIO hook so even when the platform detects errors synchronously the driver notifications must happen asynchronously. In the meanwhile the driver still needs to handle the 0xFFs response safely and there's not much we can do from the platform side to help there. Oliver
On Monday 19 July 2021 12:49:18 Oliver O'Halloran wrote: > On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote: > > > > And do we have some solution for this kind of issue? There are more PCIe > > controllers / platforms which do not like MMIO read/write operation when > > card / link is not connected. > > Do you have some actual examples? The few times I've seen those > crashes were due to broken firmware-first error handling. The AER > notifications would be escalated into some kind of ACPI error which > the kernel didn't have a good way of dealing with so it panicked > instead. I have experience and examples with pci aardvark controller. When card is disconnected it sends synchronous abort to CPU when doing MMIO read operation. One example is in this linux-usb thread: https://lore.kernel.org/linux-usb/20210505120117.4wpmo6fhvzznf3wv@pali/t/#u I can trigger this issue at least for xhci, nvme and ath drivers. > Assuming it is a real problem then as Bjorn pointed out this sort of > hack doesn't really fix the issue because hotplug and AER > notifications are fundamentally asynchronous. In case of pci aardvark it is not AER notification. And for MMIO read it is synchronous abort. Anyway, hotplug events are really asynchronous, but there is main issue that this hotplug disconnect event instruct device driver to "unbind" and e.g. these ethernet or usb controllers try to do MMIO operations in their cleanup / remove / unbind phase, even when card is already "disconnected" in PCI subsystem. > If the driver is > actively using the device when the error / removal happens then the > pci_dev_is_disconnected() check will pass and the MMIO will go > through. If the MMIO is poisonous because of dumb hardware then this > sort of hack will only paper over the issue. > > > If we do not provide a way how to solve these problems then we can > > expect that people would just hack ethernet / wifi / ... device drivers > > which are currently crashing by patches like in this thread. > > > > Maybe PCI subsystem could provide wrapper function which implements > > above pattern and which can be used by device drivers? > > We could do that and I think there was a proposal to add some > pci_readl(pdev, <addr>) style wrappers at one point. On powerpc > there's hooks in the arch provided MMIO functions to detect error > responses and kick off the error handling machinery when a problem is > detected. Those hooks are mainly there to help the platform detect > errors though and they don't make life much easier for drivers. Due to > locking concerns the driver's .error_detected() callback cannot be > called in the MMIO hook so even when the platform detects errors > synchronously the driver notifications must happen asynchronously. In > the meanwhile the driver still needs to handle the 0xFFs response > safely and there's not much we can do from the platform side to help > there. > > Oliver
On Mon, Jul 19, 2021 at 12:49:18PM +1000, Oliver O'Halloran wrote: > On Mon, Jul 19, 2021 at 8:51 AM Pali Rohár <pali@kernel.org> wrote: > > > > And do we have some solution for this kind of issue? There are more PCIe > > controllers / platforms which do not like MMIO read/write operation when > > card / link is not connected. > > Do you have some actual examples? The few times I've seen those > crashes were due to broken firmware-first error handling. The AER > notifications would be escalated into some kind of ACPI error which > the kernel didn't have a good way of dealing with so it panicked > instead. > > Assuming it is a real problem then as Bjorn pointed out this sort of > hack doesn't really fix the issue because hotplug and AER > notifications are fundamentally asynchronous. If the driver is > actively using the device when the error / removal happens then the > pci_dev_is_disconnected() check will pass and the MMIO will go > through. If the MMIO is poisonous because of dumb hardware then this > sort of hack will only paper over the issue. > > > If we do not provide a way how to solve these problems then we can > > expect that people would just hack ethernet / wifi / ... device drivers > > which are currently crashing by patches like in this thread. > > > > Maybe PCI subsystem could provide wrapper function which implements > > above pattern and which can be used by device drivers? > > We could do that and I think there was a proposal to add some > pci_readl(pdev, <addr>) style wrappers at one point. Obviously this wouldn't help user-space mmaps, but in the kernel, Documentation/driver-api/device-io.rst [1] does say that drivers are supposed to use readl() et al even though on most arches it "works" to just dereference the result of ioremap(), so maybe we could make a useful wrapper. Seems like we should do *something*, even if it's just a generic #define and some examples. I took a stab at this [2] a couple years ago, but it was only for the PCI core, and it didn't go anywhere. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/driver-api/device-io.rst?id=v5.13#n160 [2] https://lore.kernel.org/linux-pci/20190822200551.129039-1-helgaas@kernel.org/ > On powerpc > there's hooks in the arch provided MMIO functions to detect error > responses and kick off the error handling machinery when a problem is > detected. Those hooks are mainly there to help the platform detect > errors though and they don't make life much easier for drivers. Due to > locking concerns the driver's .error_detected() callback cannot be > called in the MMIO hook so even when the platform detects errors > synchronously the driver notifications must happen asynchronously. In > the meanwhile the driver still needs to handle the 0xFFs response > safely and there's not much we can do from the platform side to help > there. > > Oliver
diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index f1adf154ec4a..606b72cb6193 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -5292,6 +5292,10 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); u32 value = 0; + if (igc->pdev && + igc->pdev->error_state == pci_channel_io_perm_failure) + return 0; + value = readl(&hw_addr[reg]); /* reads should not return all F's */ @@ -5308,6 +5312,18 @@ u32 igc_rd32(struct igc_hw *hw, u32 reg) return value; } +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val) +{ + struct igc_adapter *igc = container_of(hw, struct igc_adapter, hw); + u8 __iomem *hw_addr = READ_ONCE(hw->hw_addr); + + if (igc->pdev && + igc->pdev->error_state == pci_channel_io_perm_failure) + return; + + writel((val), &hw_addr[(reg)]); +} + int igc_set_spd_dplx(struct igc_adapter *adapter, u32 spd, u8 dplx) { struct igc_mac_info *mac = &adapter->hw.mac; diff --git a/drivers/net/ethernet/intel/igc/igc_regs.h b/drivers/net/ethernet/intel/igc/igc_regs.h index cc174853554b..eb4be87d0e8b 100644 --- a/drivers/net/ethernet/intel/igc/igc_regs.h +++ b/drivers/net/ethernet/intel/igc/igc_regs.h @@ -260,13 +260,10 @@ struct igc_hw; u32 igc_rd32(struct igc_hw *hw, u32 reg); /* write operations, indexed using DWORDS */ -#define wr32(reg, val) \ -do { \ - u8 __iomem *hw_addr = READ_ONCE((hw)->hw_addr); \ - writel((val), &hw_addr[(reg)]); \ -} while (0) +void igc_wr32(struct igc_hw *hw, u32 reg, u32 val); #define rd32(reg) (igc_rd32(hw, reg)) +#define wr32(reg, val) (igc_wr32(hw, reg, val)) #define wrfl() ((void)rd32(IGC_STATUS))
Check PCI state when rd/wr iomem. Implement wr32 function as rd32 too. When unplug TBT dock with i225, rd/wr PCI iomem will cause error log: Trace: BUG: unable to handle page fault for address: 000000000000b604 Oops: 0000 [#1] SMP NOPTI RIP: 0010:igc_rd32+0x1c/0x90 [igc] Call Trace: igc_ptp_suspend+0x6c/0xa0 [igc] igc_ptp_stop+0x12/0x50 [igc] igc_remove+0x7f/0x1c0 [igc] pci_device_remove+0x3e/0xb0 __device_release_driver+0x181/0x240 Signed-off-by: Aaron Ma <aaron.ma@canonical.com> --- drivers/net/ethernet/intel/igc/igc_main.c | 16 ++++++++++++++++ drivers/net/ethernet/intel/igc/igc_regs.h | 7 ++----- 2 files changed, 18 insertions(+), 5 deletions(-)