| Message ID | 20210817145245.3555077-1-vladimir.oltean@nxp.com |
|---|---|
| State | New |
| Headers | show |
| Series | [net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse | expand |
Hi Vladimir, On 8/17/21 4:52 PM, Vladimir Oltean wrote: > It seems that of_find_compatible_node has a weird calling convention in > which it calls of_node_put() on the "from" node argument, instead of > leaving that up to the caller. This comes from the fact that > of_find_compatible_node with a non-NULL "from" argument it only supposed > to be used as the iterator function of for_each_compatible_node(). OF > iterator functions call of_node_get on the next OF node and of_node_put() > on the previous one. > > When of_find_compatible_node calls of_node_put, it actually never > expects the refcount to drop to zero, because the call is done under the > atomic devtree_lock context, and when the refcount drops to zero it > triggers a kobject and a sysfs file deletion, which assume blocking > context. > > So any driver call to of_find_compatible_node is probably buggy because > an unexpected of_node_put() takes place. > > What should be done is to use the of_get_compatible_child() function. I have an observation that's slightly out of the scope of your patch, but I'll post here on the off chance that you find it relevant. Apologies if it's out of place. Do these integrated NXP PHYs use a specific PHY driver, or do they just use the Generic PHY driver? If the former is the case, do you experience that the PHY driver fails to get probed during mdiobus registration if the kernel uses fw_devlink=on? In my case I am writing a new subdriver for realtek-smi, a DSA driver which registers an internal MDIO bus analogously to sja1105, which is why I'm asking. I noticed a deferred probe of the PHY driver because the supplier (ethernet-switch) is not ready - presumably because all of this is happening in the probe of the switch driver. See below: [ 83.653213] device_add:3270: device: 'SMI-0': device_add [ 83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0 [ 83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add [ 83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch [ 83.654291] libphy: SMI slave MII: probed ... [ 83.659809] device_add:3270: device: 'SMI-0:00': device_add [ 83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00 [ 83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00 [ 83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add [ 83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list [ 83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list [ 83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch [ 83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet [ 83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready [ 83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list It's not necessarily fatal because phy_attach_direct will just use the Generic PHY driver as a fallback, but it's obviously not the intended behaviour. Perhaps this affects your driver too? Due to lack of hardware I am not in a position to test, but a static code analysis suggests it may be if you are expecting anything but Generic PHY. Kind regards, Alvin > > Fixes: 5a8f09748ee7 ("net: dsa: sja1105: register the MDIO buses for 100base-T1 and 100base-TX") > Link: https://lore.kernel.org/netdev/20210814010139.kzryimmp4rizlznt@skbuf/ > Suggested-by: Frank Rowand <frowand.list@gmail.com> > Suggested-by: Rob Herring <robh+dt@kernel.org> > Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com> > --- > drivers/net/dsa/sja1105/sja1105_mdio.c | 6 ++---- > 1 file changed, 2 insertions(+), 4 deletions(-) > > diff --git a/drivers/net/dsa/sja1105/sja1105_mdio.c b/drivers/net/dsa/sja1105/sja1105_mdio.c > index 19aea8fb76f6..705d3900e43a 100644 > --- a/drivers/net/dsa/sja1105/sja1105_mdio.c > +++ b/drivers/net/dsa/sja1105/sja1105_mdio.c > @@ -284,8 +284,7 @@ static int sja1105_mdiobus_base_tx_register(struct sja1105_private *priv, > struct mii_bus *bus; > int rc = 0; > > - np = of_find_compatible_node(mdio_node, NULL, > - "nxp,sja1110-base-tx-mdio"); > + np = of_get_compatible_child(mdio_node, "nxp,sja1110-base-tx-mdio"); > if (!np) > return 0; > > @@ -339,8 +338,7 @@ static int sja1105_mdiobus_base_t1_register(struct sja1105_private *priv, > struct mii_bus *bus; > int rc = 0; > > - np = of_find_compatible_node(mdio_node, NULL, > - "nxp,sja1110-base-t1-mdio"); > + np = of_get_compatible_child(mdio_node, "nxp,sja1110-base-t1-mdio"); > if (!np) > return 0; > >
> Do these integrated NXP PHYs use a specific PHY driver, or do they just > use the Generic PHY driver? If the former is the case, do you experience > that the PHY driver fails to get probed during mdiobus registration if > the kernel uses fw_devlink=on? The Marvell mv88e6xxx driver registers upto two MDIO busses, and the PHYs on those busses make use of the marvell PHY driver. I've not tested specifically with fw_devlink=on, but in general, the Marvell PHY driver does get bound to these devices. Andrew
On Wed, Aug 18, 2021 at 01:31:01AM +0300, Vladimir Oltean wrote: > So it's a circular dependency? Switch cannot finish probing because it > cannot connect to PHY, which cannot probe because switch has not > finished probing, which.... The missing piece of the puzzle might be that MAC drivers can either connect to the PHY at probe time, or at .ndo_open time. DSA chooses to call phylink_of_phy_connect() inside dsa_port_phylink_register(), so at probe time. Maybe this is a reason why it is a problem for some drivers but not for others.
On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote: > > Hi Alvin, > > On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote: > > I have an observation that's slightly out of the scope of your patch, > > but I'll post here on the off chance that you find it relevant. > > Apologies if it's out of place. > > > > Do these integrated NXP PHYs use a specific PHY driver, or do they just > > use the Generic PHY driver? > > They refuse to probe at all with the Generic PHY driver. I have been > caught off guard a few times now when I had a kernel built with > CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case. > > > If the former is the case, do you experience that the PHY driver fails > > to get probed during mdiobus registration if the kernel uses > > fw_devlink=on? > > I don't test with "fw_devlink=on" in /proc/cmdline, this is the first > time I do it. It behaves exactly as you say. > > > > > In my case I am writing a new subdriver for realtek-smi, a DSA driver > > which registers an internal MDIO bus analogously to sja1105, which is > > why I'm asking. I noticed a deferred probe of the PHY driver because the > > supplier (ethernet-switch) is not ready - presumably because all of this > > is happening in the probe of the switch driver. See below: > > > > [ 83.653213] device_add:3270: device: 'SMI-0': device_add > > [ 83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0 > > [ 83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add > > [ 83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch > > [ 83.654291] libphy: SMI slave MII: probed > > ... > > [ 83.659809] device_add:3270: device: 'SMI-0:00': device_add > > [ 83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00 > > [ 83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00 > > [ 83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add > > [ 83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list > > [ 83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list > > [ 83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch > > [ 83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet > > [ 83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready > > [ 83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list > > So it's a circular dependency? Switch cannot finish probing because it > cannot connect to PHY, which cannot probe because switch has not > finished probing, which.... Hi Vladimir/Alvin, If there's a cyclic dependency between two devices, then fw_devlink=on is smart enough to notice that. Once it notices a cycle, it knows that it can't tell which one is the real dependency and which one is the false dependency and so stops enforcing ordering between the devices in the cycle. But fw_devlink doesn't understand all the properties yet. Just most of them and I'm always trying to add more. So when it only understands the property that's causing the false dependency but not the property that causes the real dependency, it can cause issues like this where fw_devlink=on enforces the false dependency and the driver/code enforces the real dependency. These are generally easy to fix -- you just need to teach fw_devlink how to parse more properties. This is just a preliminary analysis since I don't have all the info yet -- so I could be wrong. With that said, I happened to be working on adding fw_devlink support for phy-handle property and I think it should fix your issue with fw_devlink=on. Can you give [1] a shot? If it doesn't fix it, can one of you please point me to an upstream dts (not dtsi) file for a platform in which you see this issue? And ideally also the DT nodes and their drivers that are involved in this cycle? With that info, I should be able to root cause this if the patch above doesn't already fix it. > > So how is it supposed to be solved then? Intuitively the 'mdio_bus SMI-0:00' > device should not be added to the deferred list, it should have everything > it needs right now (after all, it works without fw_devlink). No? > > It might be the late hour over here too, but right now I just don't > know. Let me add Saravana to the discussion too, he made an impressive > analysis recently on a PHY probing issue with mdio-mux, Lol, thanks for the kind words. > so the PHY > library probing dependencies should still be fresh in his mind, maybe he > has an idea what's wrong. [1] - https://lore.kernel.org/lkml/20210818021717.3268255-1-saravanak@google.com/T/#u Thanks, Saravana
Hello: This patch was applied to netdev/net.git (refs/heads/master): On Tue, 17 Aug 2021 17:52:45 +0300 you wrote: > It seems that of_find_compatible_node has a weird calling convention in > which it calls of_node_put() on the "from" node argument, instead of > leaving that up to the caller. This comes from the fact that > of_find_compatible_node with a non-NULL "from" argument it only supposed > to be used as the iterator function of for_each_compatible_node(). OF > iterator functions call of_node_get on the next OF node and of_node_put() > on the previous one. > > [...] Here is the summary with links: - [net] net: dsa: sja1105: fix use-after-free after calling of_find_compatible_node, or worse https://git.kernel.org/netdev/net/c/ed5d2937a6a8 You are awesome, thank you! -- Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Hi Saravana, On 8/18/21 4:46 AM, Saravana Kannan wrote: > On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote: >> >> Hi Alvin, >> >> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote: >>> I have an observation that's slightly out of the scope of your patch, >>> but I'll post here on the off chance that you find it relevant. >>> Apologies if it's out of place. >>> >>> Do these integrated NXP PHYs use a specific PHY driver, or do they just >>> use the Generic PHY driver? >> >> They refuse to probe at all with the Generic PHY driver. I have been >> caught off guard a few times now when I had a kernel built with >> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case. >> >>> If the former is the case, do you experience that the PHY driver fails >>> to get probed during mdiobus registration if the kernel uses >>> fw_devlink=on? >> >> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first >> time I do it. It behaves exactly as you say. >> >>> >>> In my case I am writing a new subdriver for realtek-smi, a DSA driver >>> which registers an internal MDIO bus analogously to sja1105, which is >>> why I'm asking. I noticed a deferred probe of the PHY driver because the >>> supplier (ethernet-switch) is not ready - presumably because all of this >>> is happening in the probe of the switch driver. See below: >>> >>> [ 83.653213] device_add:3270: device: 'SMI-0': device_add >>> [ 83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0 >>> [ 83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add >>> [ 83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch >>> [ 83.654291] libphy: SMI slave MII: probed >>> ... >>> [ 83.659809] device_add:3270: device: 'SMI-0:00': device_add >>> [ 83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00 >>> [ 83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00 >>> [ 83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add >>> [ 83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list >>> [ 83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list >>> [ 83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch >>> [ 83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet >>> [ 83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready >>> [ 83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list >> >> So it's a circular dependency? Switch cannot finish probing because it >> cannot connect to PHY, which cannot probe because switch has not >> finished probing, which.... > > Hi Vladimir/Alvin, > > If there's a cyclic dependency between two devices, then fw_devlink=on > is smart enough to notice that. Once it notices a cycle, it knows that > it can't tell which one is the real dependency and which one is the > false dependency and so stops enforcing ordering between the devices > in the cycle. > > But fw_devlink doesn't understand all the properties yet. Just most of > them and I'm always trying to add more. So when it only understands > the property that's causing the false dependency but not the property > that causes the real dependency, it can cause issues like this where > fw_devlink=on enforces the false dependency and the driver/code > enforces the real dependency. These are generally easy to fix -- you > just need to teach fw_devlink how to parse more properties. > > This is just a preliminary analysis since I don't have all the info > yet -- so I could be wrong. With that said, I happened to be working > on adding fw_devlink support for phy-handle property and I think it > should fix your issue with fw_devlink=on. Can you give [1] a shot? I tried [1] but it did not seem to have any effect. > > If it doesn't fix it, can one of you please point me to an upstream > dts (not dtsi) file for a platform in which you see this issue? And > ideally also the DT nodes and their drivers that are involved in this > cycle? With that info, I should be able to root cause this if the > patch above doesn't already fix it. I'm working with a non-upstream dts - maybe Vladimir is using an upstream one? The pattern among the drivers/dts is common between our two cases. But for the sake of this discussion, my dts is pretty much the same as what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The nodes of interest from that dts file are below, and the driver is in drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part of the mdiobus registration, but that never happens. See my previous reply for a debug log. / { switch { compatible = "realtek,rtl8366rb"; mdc-gpios = <&gpio0 21 GPIO_ACTIVE_HIGH>; mdio-gpios = <&gpio0 22 GPIO_ACTIVE_HIGH>; reset-gpios = <&gpio0 14 GPIO_ACTIVE_LOW>; realtek,disable-leds; switch_intc: interrupt-controller { /* GPIO 15 provides the interrupt */ interrupt-parent = <&gpio0>; interrupts = <15 IRQ_TYPE_LEVEL_LOW>; interrupt-controller; #address-cells = <0>; #interrupt-cells = <1>; }; ports { /* snip */ }; mdio { compatible = "realtek,smi-mdio"; #address-cells = <1>; #size-cells = <0>; phy0: phy@0 { reg = <0>; interrupt-parent = <&switch_intc>; interrupts = <0>; }; phy1: phy@1 { reg = <1>; interrupt-parent = <&switch_intc>; interrupts = <1>; }; phy2: phy@2 { reg = <2>; interrupt-parent = <&switch_intc>; interrupts = <2>; }; phy3: phy@3 { reg = <3>; interrupt-parent = <&switch_intc>; interrupts = <3>; }; phy4: phy@4 { reg = <4>; interrupt-parent = <&switch_intc>; interrupts = <12>; }; }; }; }; Thanks for looking into this. Let me know if you need any other info or want something tested. Kind regards, Alvin > >> >> So how is it supposed to be solved then? Intuitively the 'mdio_bus SMI-0:00' >> device should not be added to the deferred list, it should have everything >> it needs right now (after all, it works without fw_devlink). No? >> >> It might be the late hour over here too, but right now I just don't >> know. Let me add Saravana to the discussion too, he made an impressive >> analysis recently on a PHY probing issue with mdio-mux, > > Lol, thanks for the kind words. > >> so the PHY >> library probing dependencies should still be fresh in his mind, maybe he >> has an idea what's wrong. > > [1] - https://lore.kernel.org/lkml/20210818021717.3268255-1-saravanak@google.com/T/#u>> > Thanks, > Saravana >
On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > Hi Saravana, > > On 8/18/21 4:46 AM, Saravana Kannan wrote: > > On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote: > >> > >> Hi Alvin, > >> > >> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote: > >>> I have an observation that's slightly out of the scope of your patch, > >>> but I'll post here on the off chance that you find it relevant. > >>> Apologies if it's out of place. > >>> > >>> Do these integrated NXP PHYs use a specific PHY driver, or do they just > >>> use the Generic PHY driver? > >> > >> They refuse to probe at all with the Generic PHY driver. I have been > >> caught off guard a few times now when I had a kernel built with > >> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case. > >> > >>> If the former is the case, do you experience that the PHY driver fails > >>> to get probed during mdiobus registration if the kernel uses > >>> fw_devlink=on? > >> > >> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first > >> time I do it. It behaves exactly as you say. > >> > >>> > >>> In my case I am writing a new subdriver for realtek-smi, a DSA driver > >>> which registers an internal MDIO bus analogously to sja1105, which is > >>> why I'm asking. I noticed a deferred probe of the PHY driver because the > >>> supplier (ethernet-switch) is not ready - presumably because all of this > >>> is happening in the probe of the switch driver. See below: > >>> > >>> [ 83.653213] device_add:3270: device: 'SMI-0': device_add > >>> [ 83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0 > >>> [ 83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add > >>> [ 83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch > >>> [ 83.654291] libphy: SMI slave MII: probed > >>> ... > >>> [ 83.659809] device_add:3270: device: 'SMI-0:00': device_add > >>> [ 83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00 > >>> [ 83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00 > >>> [ 83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add > >>> [ 83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list > >>> [ 83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list > >>> [ 83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch > >>> [ 83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet > >>> [ 83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready > >>> [ 83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list > >> > >> So it's a circular dependency? Switch cannot finish probing because it > >> cannot connect to PHY, which cannot probe because switch has not > >> finished probing, which.... > > > > Hi Vladimir/Alvin, > > > > If there's a cyclic dependency between two devices, then fw_devlink=on > > is smart enough to notice that. Once it notices a cycle, it knows that > > it can't tell which one is the real dependency and which one is the > > false dependency and so stops enforcing ordering between the devices > > in the cycle. > > > > But fw_devlink doesn't understand all the properties yet. Just most of > > them and I'm always trying to add more. So when it only understands > > the property that's causing the false dependency but not the property > > that causes the real dependency, it can cause issues like this where > > fw_devlink=on enforces the false dependency and the driver/code > > enforces the real dependency. These are generally easy to fix -- you > > just need to teach fw_devlink how to parse more properties. > > > > This is just a preliminary analysis since I don't have all the info > > yet -- so I could be wrong. With that said, I happened to be working > > on adding fw_devlink support for phy-handle property and I think it > > should fix your issue with fw_devlink=on. Can you give [1] a shot? > > I tried [1] but it did not seem to have any effect. > > > > > If it doesn't fix it, can one of you please point me to an upstream > > dts (not dtsi) file for a platform in which you see this issue? And > > ideally also the DT nodes and their drivers that are involved in this > > cycle? With that info, I should be able to root cause this if the > > patch above doesn't already fix it. > > I'm working with a non-upstream dts - maybe Vladimir is using an > upstream one? The pattern among the drivers/dts is common between our > two cases. Ideally, I can get a fully upstream example where this issue is happening so that I can look at the actual code that's hitting this issue and be sure my analysis is right. > > But for the sake of this discussion, my dts is pretty much the same as > what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The > nodes of interest from that dts file are below, and the driver is in > drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the > Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part > of the mdiobus registration, but that never happens. See my previous > reply for a debug log. Your DTS might be similar to this, but the driver code also matters for me to be sure. Anyway, I took a look at this, but my analysis below is going to be sketchy because I'm not looking at the actual code that's reproducing this issue. Assuming this issue actually happens with the example you pointed to (I don't know this yet), here's what is happening: The main problem is that the parent device switch seems to be assuming it's child/grandchild devices (mdiobus/PHYs) will have probed successfully as soon as they are added. This assumption is not true and can be broken for multiple reasons such as: 1. The driver for the child devices (PHYs in this case) could be loaded as a module after the parent (switch) is probed. So when the devices are added, the PHYs would not be probed. 2. The child devices could defer probe because one of their suppliers isn't ready yet. Either because of fw_devlink=on or the framework itself returning -EPROBE_DEFER. 3. The child devices could be getting probed asynchronously. So the device_add() would kick off a thread to probe the child devices in a separate thread. (2) is what is happening in this case. fw_devlink=on sees that "switch" implements the "switch_intc" and "switch" hasn't finished probing yet. So it has no way of knowing that switch_intc is actually ready. And even if switch_intc was registered as part of switch's probe() by the time the PHYs are added, switch_intc could get deregistered if the probe fails at a later point. So until probe() returns 0, fw_devlink can't be fully sure the supplier (switch_intc) is ready. Which is good in general because you won't have to forcefully unbind (if that is even handled correctly in the first place) the consumers of a device if it fails probe() half way through registering a few services. I don't fully understand the networking frameworks, but I think Vladimir might have a point in his earlier reply [1]. If you can make the switch driver not assume its child PHYs are ready during the switch's probe and instead have the switch check if the PHYs are ready when the switch is "opened" that'd be better. We can come up with hacks that'll delete the dependency that fw_devlink=on is trying to enforce, but IMHO the proper fix is to have parent drivers not assume child devices will be probed as soon as device_add(child) returns. That's not guaranteed at all. Btw, I do know why things work when you do the module load/unload thing you mention in [2]. That has to do with some forced deletion of dependencies that happens when device_bind_driver() is called when the Generic PHY driver is used. The reason for why that's done is kind of unrelated to the issue at hand, but the comment for device_links_force_bind() should tell you why. Hope that helps. [1] - https://lore.kernel.org/netdev/20210817224008.pzdomrjaw5ewmpdg@skbuf/ [2] - https://lore.kernel.org/netdev/0c3e8814-acce-5836-3b1a-6804c21e9bf0@bang-olufsen.dk/ -Saravana > > / { > switch { > compatible = "realtek,rtl8366rb"; > mdc-gpios = <&gpio0 21 GPIO_ACTIVE_HIGH>; > mdio-gpios = <&gpio0 22 GPIO_ACTIVE_HIGH>; > reset-gpios = <&gpio0 14 GPIO_ACTIVE_LOW>; > realtek,disable-leds; > > switch_intc: interrupt-controller { > /* GPIO 15 provides the interrupt */ > interrupt-parent = <&gpio0>; > interrupts = <15 IRQ_TYPE_LEVEL_LOW>; > interrupt-controller; > #address-cells = <0>; > #interrupt-cells = <1>; > }; > > ports { > /* snip */ > }; > > mdio { > compatible = "realtek,smi-mdio"; > #address-cells = <1>; > #size-cells = <0>; > > phy0: phy@0 { > reg = <0>; > interrupt-parent = <&switch_intc>; > interrupts = <0>; > }; > phy1: phy@1 { > reg = <1>; > interrupt-parent = <&switch_intc>; > interrupts = <1>; > }; > phy2: phy@2 { > reg = <2>; > interrupt-parent = <&switch_intc>; > interrupts = <2>; > }; > phy3: phy@3 { > reg = <3>; > interrupt-parent = <&switch_intc>; > interrupts = <3>; > }; > phy4: phy@4 { > reg = <4>; > interrupt-parent = <&switch_intc>; > interrupts = <12>; > }; > }; > }; > }; > > Thanks for looking into this. Let me know if you need any other info or > want something tested. > > Kind regards, > Alvin > > > > >> > >> So how is it supposed to be solved then? Intuitively the 'mdio_bus SMI-0:00' > >> device should not be added to the deferred list, it should have everything > >> it needs right now (after all, it works without fw_devlink). No? > >> > >> It might be the late hour over here too, but right now I just don't > >> know. Let me add Saravana to the discussion too, he made an impressive > >> analysis recently on a PHY probing issue with mdio-mux, > > > > Lol, thanks for the kind words. > > > >> so the PHY > >> library probing dependencies should still be fresh in his mind, maybe he > >> has an idea what's wrong. > > > > [1] - https://lore.kernel.org/lkml/20210818021717.3268255-1-saravanak@google.com/T/#u>> > > Thanks, > > Saravana > >
On Wed, Aug 18, 2021 at 08:28:44PM -0700, Saravana Kannan wrote: > On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > > > Hi Saravana, > > > > On 8/18/21 4:46 AM, Saravana Kannan wrote: > > > On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote: > > >> > > >> Hi Alvin, > > >> > > >> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote: > > >>> I have an observation that's slightly out of the scope of your patch, > > >>> but I'll post here on the off chance that you find it relevant. > > >>> Apologies if it's out of place. > > >>> > > >>> Do these integrated NXP PHYs use a specific PHY driver, or do they just > > >>> use the Generic PHY driver? > > >> > > >> They refuse to probe at all with the Generic PHY driver. I have been > > >> caught off guard a few times now when I had a kernel built with > > >> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case. > > >> > > >>> If the former is the case, do you experience that the PHY driver fails > > >>> to get probed during mdiobus registration if the kernel uses > > >>> fw_devlink=on? > > >> > > >> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first > > >> time I do it. It behaves exactly as you say. Sorry for the delay, I wanted to reconfirm what I said (hint, I was wrong). In my case, whatever I do, I cannot get the driver core enforce a device link between the ethernet-switch and the PHY. So I cannot actually see the same issue. What I was seeing was in fact stupid testing on my part (it was working with the PHY driver as built-in, it was working, then I made it a module, it broke, I forgot to switch it back to module, then I thought it's broken while the PHY is built-in). > > >> > > >>> > > >>> In my case I am writing a new subdriver for realtek-smi, a DSA driver > > >>> which registers an internal MDIO bus analogously to sja1105, which is > > >>> why I'm asking. I noticed a deferred probe of the PHY driver because the > > >>> supplier (ethernet-switch) is not ready - presumably because all of this > > >>> is happening in the probe of the switch driver. See below: > > >>> > > >>> [ 83.653213] device_add:3270: device: 'SMI-0': device_add > > >>> [ 83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0 > > >>> [ 83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add > > >>> [ 83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch > > >>> [ 83.654291] libphy: SMI slave MII: probed > > >>> ... > > >>> [ 83.659809] device_add:3270: device: 'SMI-0:00': device_add > > >>> [ 83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00 > > >>> [ 83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00 > > >>> [ 83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add > > >>> [ 83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list > > >>> [ 83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list > > >>> [ 83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch > > >>> [ 83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet > > >>> [ 83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready > > >>> [ 83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list > > >> > > >> So it's a circular dependency? Switch cannot finish probing because it > > >> cannot connect to PHY, which cannot probe because switch has not > > >> finished probing, which.... > > > > > > Hi Vladimir/Alvin, > > > > > > If there's a cyclic dependency between two devices, then fw_devlink=on > > > is smart enough to notice that. Once it notices a cycle, it knows that > > > it can't tell which one is the real dependency and which one is the > > > false dependency and so stops enforcing ordering between the devices > > > in the cycle. > > > > > > But fw_devlink doesn't understand all the properties yet. Just most of > > > them and I'm always trying to add more. So when it only understands > > > the property that's causing the false dependency but not the property > > > that causes the real dependency, it can cause issues like this where > > > fw_devlink=on enforces the false dependency and the driver/code > > > enforces the real dependency. These are generally easy to fix -- you > > > just need to teach fw_devlink how to parse more properties. > > > > > > This is just a preliminary analysis since I don't have all the info > > > yet -- so I could be wrong. With that said, I happened to be working > > > on adding fw_devlink support for phy-handle property and I think it > > > should fix your issue with fw_devlink=on. Can you give [1] a shot? > > > > I tried [1] but it did not seem to have any effect. > > I applied the phy-handle patch, and here are my boot logs. [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083] [ 0.000000] Linux version 5.14.0-rc5+ (tigrisor@skbuf) (aarch64-linux-gnu-gcc (Linaro GCC 7.5-2019.12) 7.5.0, GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706) #440 SMP Thu Aug 19 13:48:11 EEST 2021 [ 0.000000] Machine model: NXP Layerscape LX2160ABLUEBOX3 [ 0.000000] fw_devlink_setup: fw_devlink_flags = 0x120 [ 0.000000] efi: UEFI not found. [ 0.000000] [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader! [ 0.000000] NUMA: No NUMA configuration found [ 0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000277fffffff] [ 0.000000] NUMA: NODE_DATA [mem 0x277c1c49c0-0x277c1c6fff] [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000080000000-0x00000000ffffffff] [ 0.000000] DMA32 empty [ 0.000000] Normal [mem 0x0000000100000000-0x000000277fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000080000000-0x00000000fbdfffff] [ 0.000000] node 0: [mem 0x0000002080000000-0x000000277fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000277fffffff] [ 0.000000] On node 0, zone Normal: 16896 pages in unavailable ranges [ 0.000000] cma: Reserved 32 MiB at 0x00000000f9c00000 [ 0.000000] psci: probing for conduit method from DT. [ 0.000000] psci: PSCIv1.1 detected in firmware. [ 0.000000] psci: Using standard PSCI v0.2 function IDs [ 0.000000] psci: MIGRATE_INFO_TYPE not supported. [ 0.000000] psci: SMC Calling Convention v1.2 [ 0.000000] percpu: Embedded 482 pages/cpu s1933592 r8192 d32488 u1974272 [ 0.000000] Detected PIPT I-cache on CPU0 [ 0.000000] CPU features: detected: GIC system register CPU interface [ 0.000000] CPU features: detected: Spectre-v2 [ 0.000000] CPU features: detected: Spectre-v3a [ 0.000000] CPU features: kernel page table isolation forced ON by KASLR [ 0.000000] CPU features: detected: Kernel page table isolation (KPTI) [ 0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923 [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 7724544 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: fw_devlink=on root=/dev/mmcblk0p2 rw console=ttyAMA0,115200 [ 0.000000] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear) [ 0.000000] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear) [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.000000] software IO TLB: mapped [mem 0x00000000f5c00000-0x00000000f9c00000] (64MB) [ 0.000000] Memory: 30595480K/31389696K available (23040K kernel code, 4562K rwdata, 10700K rodata, 9536K init, 11765K bss, 761448K reserved, 32768K cma-reserved) [ 0.000000] trace event string verifier disabled [ 0.000000] Running RCU self tests [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU lockdep checking is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=16. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=16 [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode [ 0.000000] GICv3: 256 SPIs implemented [ 0.000000] GICv3: 0 Extended SPIs implemented [ 0.000000] GICv3: Distributor has no Range Selector support [ 0.000000] Root IRQ handler: gic_handle_irq [ 0.000000] GICv3: 16 PPIs implemented [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006200000 [ 0.000000] ITS [mem 0x06020000-0x0603ffff] [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices @2080100000 (flat, esz 8, psz 64K, shr 0) [ 0.000000] ITS: using cache flushing for cmd queue [ 0.000000] GICv3: Using preallocated redistributor tables [ 0.000000] GICv3: using LPI property table @0x000000277fe00000 [ 0.000000] GICv3: CPU0: using reserved LPI pending table @0x000000277fe10000 [ 0.000000] random: get_random_bytes called from start_kernel+0x544/0x748 with crng_init=0 [ 0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys). [ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c409fb33, max_idle_ns: 440795203156 ns [ 0.000000] sched_clock: 56 bits at 25MHz, resolution 39ns, wraps every 4398046511103ns [ 0.001138] Console: colour dummy device 80x25 [ 0.001160] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar [ 0.001167] ... MAX_LOCKDEP_SUBCLASSES: 8 [ 0.001174] ... MAX_LOCK_DEPTH: 48 [ 0.001181] ... MAX_LOCKDEP_KEYS: 8192 [ 0.001187] ... CLASSHASH_SIZE: 4096 [ 0.001194] ... MAX_LOCKDEP_ENTRIES: 32768 [ 0.001201] ... MAX_LOCKDEP_CHAINS: 65536 [ 0.001207] ... CHAINHASH_SIZE: 32768 [ 0.001214] memory used by lock dependency info: 6877 kB [ 0.001220] memory used for stack traces: 4224 kB [ 0.001227] per task-struct memory footprint: 2688 bytes [ 0.001308] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000) [ 0.001322] pid_max: default: 32768 minimum: 301 [ 0.002396] LSM: Security Framework initializing [ 0.002583] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, linear) [ 0.002672] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, linear) [ 0.005656] Running RCU-tasks wait API self tests [ 0.005972] rcu: Hierarchical SRCU implementation. [ 0.006409] dyndbg: Ignore empty _ddebug table in a CONFIG_DYNAMIC_DEBUG_CORE build [ 0.006490] Platform MSI: gic-its@6020000 domain created [ 0.006591] PCI/MSI: /interrupt-controller@6000000/gic-its@6020000 domain created [ 0.006680] fsl-mc MSI: gic-its@6020000 domain created [ 0.007278] EFI services will not be available. [ 0.008971] smp: Bringing up secondary CPUs ... [ 0.010766] Detected PIPT I-cache on CPU1 [ 0.010791] GICv3: CPU1: found redistributor 1 region 0:0x0000000006220000 [ 0.010818] GICv3: CPU1: using reserved LPI pending table @0x000000277fe20000 [ 0.010951] CPU1: Booted secondary processor 0x0000000001 [0x410fd083] [ 0.013072] Detected PIPT I-cache on CPU2 [ 0.013095] GICv3: CPU2: found redistributor 100 region 0:0x0000000006240000 [ 0.013120] GICv3: CPU2: using reserved LPI pending table @0x000000277fe30000 [ 0.013258] CPU2: Booted secondary processor 0x0000000100 [0x410fd083] [ 0.015350] Detected PIPT I-cache on CPU3 [ 0.015367] GICv3: CPU3: found redistributor 101 region 0:0x0000000006260000 [ 0.015388] GICv3: CPU3: using reserved LPI pending table @0x000000277fe40000 [ 0.015511] CPU3: Booted secondary processor 0x0000000101 [0x410fd083] [ 0.017419] Detected PIPT I-cache on CPU4 [ 0.017444] GICv3: CPU4: found redistributor 200 region 0:0x0000000006280000 [ 0.017467] GICv3: CPU4: using reserved LPI pending table @0x000000277fe50000 [ 0.017600] CPU4: Booted secondary processor 0x0000000200 [0x410fd083] [ 0.019496] Detected PIPT I-cache on CPU5 [ 0.019516] GICv3: CPU5: found redistributor 201 region 0:0x00000000062a0000 [ 0.019536] GICv3: CPU5: using reserved LPI pending table @0x000000277fe60000 [ 0.019657] CPU5: Booted secondary processor 0x0000000201 [0x410fd083] [ 0.021614] Detected PIPT I-cache on CPU6 [ 0.021643] GICv3: CPU6: found redistributor 300 region 0:0x00000000062c0000 [ 0.021658] GICv3: CPU6: using reserved LPI pending table @0x000000277fe70000 [ 0.021791] CPU6: Booted secondary processor 0x0000000300 [0x410fd083] [ 0.022061] Callback from call_rcu_tasks_trace() invoked. [ 0.023813] Detected PIPT I-cache on CPU7 [ 0.023838] GICv3: CPU7: found redistributor 301 region 0:0x00000000062e0000 [ 0.023858] GICv3: CPU7: using reserved LPI pending table @0x000000277fe80000 [ 0.023978] CPU7: Booted secondary processor 0x0000000301 [0x410fd083] [ 0.025863] Detected PIPT I-cache on CPU8 [ 0.025886] GICv3: CPU8: found redistributor 400 region 0:0x0000000006300000 [ 0.025902] GICv3: CPU8: using reserved LPI pending table @0x000000277fe90000 [ 0.026032] CPU8: Booted secondary processor 0x0000000400 [0x410fd083] [ 0.027921] Detected PIPT I-cache on CPU9 [ 0.027943] GICv3: CPU9: found redistributor 401 region 0:0x0000000006320000 [ 0.027956] GICv3: CPU9: using reserved LPI pending table @0x000000277fea0000 [ 0.028078] CPU9: Booted secondary processor 0x0000000401 [0x410fd083] [ 0.030006] Detected PIPT I-cache on CPU10 [ 0.030033] GICv3: CPU10: found redistributor 500 region 0:0x0000000006340000 [ 0.030054] GICv3: CPU10: using reserved LPI pending table @0x000000277feb0000 [ 0.030189] CPU10: Booted secondary processor 0x0000000500 [0x410fd083] [ 0.032076] Detected PIPT I-cache on CPU11 [ 0.032099] GICv3: CPU11: found redistributor 501 region 0:0x0000000006360000 [ 0.032113] GICv3: CPU11: using reserved LPI pending table @0x000000277fec0000 [ 0.032235] CPU11: Booted secondary processor 0x0000000501 [0x410fd083] [ 0.034091] Detected PIPT I-cache on CPU12 [ 0.034118] GICv3: CPU12: found redistributor 600 region 0:0x0000000006380000 [ 0.034135] GICv3: CPU12: using reserved LPI pending table @0x000000277fed0000 [ 0.034271] CPU12: Booted secondary processor 0x0000000600 [0x410fd083] [ 0.036130] Detected PIPT I-cache on CPU13 [ 0.036151] GICv3: CPU13: found redistributor 601 region 0:0x00000000063a0000 [ 0.036165] GICv3: CPU13: using reserved LPI pending table @0x000000277fee0000 [ 0.036287] CPU13: Booted secondary processor 0x0000000601 [0x410fd083] [ 0.038161] Detected PIPT I-cache on CPU14 [ 0.038189] GICv3: CPU14: found redistributor 700 region 0:0x00000000063c0000 [ 0.038205] GICv3: CPU14: using reserved LPI pending table @0x000000277fef0000 [ 0.038336] CPU14: Booted secondary processor 0x0000000700 [0x410fd083] [ 0.040221] Detected PIPT I-cache on CPU15 [ 0.040242] GICv3: CPU15: found redistributor 701 region 0:0x00000000063e0000 [ 0.040256] GICv3: CPU15: using reserved LPI pending table @0x000000277ff00000 [ 0.040377] CPU15: Booted secondary processor 0x0000000701 [0x410fd083] [ 0.040608] smp: Brought up 1 node, 16 CPUs [ 0.040729] SMP: Total of 16 processors activated. [ 0.040738] CPU features: detected: 32-bit EL0 Support [ 0.040747] CPU features: detected: 32-bit EL1 Support [ 0.040756] CPU features: detected: CRC32 instructions [ 0.067039] CPU: All CPU(s) started at EL2 [ 0.067176] alternatives: patching kernel code [ 0.069867] devtmpfs: initialized [ 0.099164] KASLR enabled [ 0.100410] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.100433] futex hash table entries: 4096 (order: 7, 524288 bytes, linear) [ 0.101547] pinctrl core: initialized pinctrl subsystem [ 0.104113] DMI not present or invalid. [ 0.104964] NET: Registered PF_NETLINK/PF_ROUTE protocol family [ 0.108803] DMA: preallocated 4096 KiB GFP_KERNEL pool for atomic allocations [ 0.109609] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.110392] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.110509] audit: initializing netlink subsys (disabled) [ 0.110774] audit: type=2000 audit(0.108:1): state=initialized audit_enabled=0 res=1 [ 0.113114] thermal_sys: Registered thermal governor 'step_wise' [ 0.115915] cpuidle: using governor menu [ 0.116336] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers. [ 0.116889] ASID allocator initialised with 32768 entries [ 0.119665] Serial: AMBA PL011 UART driver [ 0.131350] Machine: NXP Layerscape LX2160ABLUEBOX3 [ 0.131364] SoC family: QorIQ LX2160A [ 0.131372] SoC ID: svr:0x87360020, Revision: 2.0 [ 0.139246] 21c0000.serial: ttyAMA0 at MMIO 0x21c0000 (irq = 26, base_baud = 0) is a SBSA [ 1.220238] printk: console [ttyAMA0] enabled [ 1.226490] sbsa-uart 21d0000.serial: aliased and non-aliased serial devices found in device tree. Serial port enumeration may be unpredictable. [ 1.239476] 21d0000.serial: ttyAMA1 at MMIO 0x21d0000 (irq = 27, base_baud = 0) is a SBSA [ 1.251594] platform 2000000.i2c: Linked as a consumer to 2320000.gpio [ 1.283271] fsl_mc_bus 80c000000.fsl-mc: MC firmware version: 10.24.1 [ 1.292523] fsl_mc_dprc dprc.1: DMA mask not set [ 1.297742] platform 2000000.i2c: Linked as a sync state only consumer to regulator-sb3v3 [ 1.343358] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 1.350096] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages [ 1.356817] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 1.363531] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages [ 1.374376] cryptd: max_cpu_qlen set to 1000 [ 1.450431] raid6: neonx8 gen() 5557 MB/s [ 1.522595] raid6: neonx8 xor() 3634 MB/s [ 1.594761] raid6: neonx4 gen() 5519 MB/s [ 1.666929] raid6: neonx4 xor() 3681 MB/s [ 1.739094] raid6: neonx2 gen() 4919 MB/s [ 1.811261] raid6: neonx2 xor() 3562 MB/s [ 1.883430] raid6: neonx1 gen() 3912 MB/s [ 1.955594] raid6: neonx1 xor() 2937 MB/s [ 2.027759] raid6: int64x8 gen() 3306 MB/s [ 2.099922] raid6: int64x8 xor() 1955 MB/s [ 2.172090] raid6: int64x4 gen() 3882 MB/s [ 2.244255] raid6: int64x4 xor() 2124 MB/s [ 2.316423] raid6: int64x2 gen() 3454 MB/s [ 2.388593] raid6: int64x2 xor() 1853 MB/s [ 2.460766] raid6: int64x1 gen() 2696 MB/s [ 2.532929] raid6: int64x1 xor() 1401 MB/s [ 2.537204] raid6: using algorithm neonx8 gen() 5557 MB/s [ 2.542607] raid6: .... xor() 3634 MB/s, rmw enabled [ 2.547575] raid6: using neon recovery algorithm [ 2.552869] ACPI: Interpreter disabled. [ 2.561168] iommu: Default domain type: Passthrough [ 2.566610] vgaarb: loaded [ 2.570380] SCSI subsystem initialized [ 2.575023] usbcore: registered new interface driver usbfs [ 2.580638] usbcore: registered new interface driver hub [ 2.586088] usbcore: registered new device driver usb [ 2.592112] imx-i2c 2000000.i2c: can't get pinctrl, bus recovery not supported [ 2.600269] i2c 0-0077: Linked as a sync state only consumer to regulator-sb3v3 [ 2.607887] i2c i2c-0: IMX I2C adapter registered [ 2.612640] imx-i2c 2000000.i2c: Dropping the link to regulator-sb3v3 [ 2.619718] imx-i2c 2050000.i2c: can't get pinctrl, bus recovery not supported [ 2.627406] i2c i2c-1: IMX I2C adapter registered [ 2.632830] pps_core: LinuxPPS API ver. 1 registered [ 2.637809] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 2.646975] PTP clock support registered [ 2.651131] EDAC MC: Ver: 3.0.0 [ 2.657363] FPGA manager framework [ 2.661061] Advanced Linux Sound Architecture Driver Initialized. [ 2.668553] Bluetooth: Core ver 2.22 [ 2.672194] NET: Registered PF_BLUETOOTH protocol family [ 2.677514] Bluetooth: HCI device and connection manager initialized [ 2.683893] Bluetooth: HCI socket layer initialized [ 2.688785] Bluetooth: L2CAP socket layer initialized [ 2.693888] Bluetooth: SCO socket layer initialized [ 2.699792] clocksource: Switched to clocksource arch_sys_counter [ 2.893095] VFS: Disk quotas dquot_6.6.0 [ 2.897133] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.904658] pnp: PnP ACPI: disabled [ 2.925319] NET: Registered PF_INET protocol family [ 2.930703] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 2.941965] tcp_listen_portaddr_hash hash table entries: 16384 (order: 8, 1441792 bytes, linear) [ 2.952166] TCP established hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 2.961285] TCP bind hash table entries: 65536 (order: 10, 5242880 bytes, vmalloc) [ 2.974112] TCP: Hash tables configured (established 262144 bind 65536) [ 2.981465] UDP hash table entries: 16384 (order: 9, 3145728 bytes, linear) [ 2.991484] UDP-Lite hash table entries: 16384 (order: 9, 3145728 bytes, linear) [ 3.002151] NET: Registered PF_UNIX/PF_LOCAL protocol family [ 3.009011] RPC: Registered named UNIX socket transport module. [ 3.014964] RPC: Registered udp transport module. [ 3.019678] RPC: Registered tcp transport module. [ 3.024396] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 3.030852] PCI: CLS 0 bytes, default 64 [ 3.182026] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available [ 3.190536] kvm [1]: IPA Size Limit: 44 bits [ 3.202176] kvm [1]: vgic-v2@c0e0000 [ 3.205790] kvm [1]: GIC system register CPU interface enabled [ 3.212408] kvm [1]: vgic interrupt IRQ9 [ 3.217322] kvm [1]: Hyp mode initialized successfully [ 3.233022] Initialise system trusted keyrings [ 3.237780] workingset: timestamp_bits=42 max_order=23 bucket_order=0 [ 3.246308] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 3.252912] NFS: Registering the id_resolver key type [ 3.258046] Key type id_resolver registered [ 3.262255] Key type id_legacy registered [ 3.266338] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 3.273063] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering... [ 3.280763] fuse: init (API version 7.34) [ 3.285101] 9p: Installing v9fs 9p2000 file system support [ 3.309033] xor: measuring software checksum speed [ 3.314861] 8regs : 9817 MB/sec [ 3.320228] 32regs : 10028 MB/sec [ 3.327078] arm64_neon : 3973 MB/sec [ 3.331443] xor: using function: 32regs (10028 MB/sec) [ 3.336594] Key type asymmetric registered [ 3.340719] Asymmetric key parser 'x509' registered [ 3.345674] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245) [ 3.353367] io scheduler mq-deadline registered [ 3.357957] io scheduler kyber registered [ 3.382684] layerscape-pcie 3400000.pcie: host bridge /soc/pcie@3400000 ranges: [ 3.390064] layerscape-pcie 3400000.pcie: MEM 0x8040000000..0x807fffffff -> 0x0040000000 [ 3.398648] layerscape-pcie 3400000.pcie: iATU unroll: disabled [ 3.404579] layerscape-pcie 3400000.pcie: Detected iATU regions: 8 outbound, 8 inbound [ 3.512561] layerscape-pcie 3400000.pcie: Link up [ 3.517626] layerscape-pcie 3400000.pcie: PCI host bridge to bus 0000:00 [ 3.524348] pci_bus 0000:00: root bus resource [bus 00-ff] [ 3.529851] pci_bus 0000:00: root bus resource [mem 0x8040000000-0x807fffffff] (bus address [0x40000000-0x7fffffff]) [ 3.540459] pci 0000:00:00.0: [1957:8d80] type 01 class 0x060400 [ 3.546650] pci 0000:00:00.0: supports D1 D2 [ 3.550934] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot [ 3.558540] pci 0000:01:00.0: [1344:6001] type 00 class 0x010802 [ 3.564674] pci 0000:01:00.0: reg 0x10: [mem 0x8040000000-0x8040003fff 64bit] [ 3.572659] pci 0000:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link) [ 3.588748] pci 0000:00:00.0: BAR 14: assigned [mem 0x8040000000-0x80400fffff] [ 3.595997] pci 0000:01:00.0: BAR 0: assigned [mem 0x8040000000-0x8040003fff 64bit] [ 3.603714] pci 0000:00:00.0: PCI bridge to [bus 01-ff] [ 3.608958] pci 0000:00:00.0: bridge window [mem 0x8040000000-0x80400fffff] [ 3.616591] layerscape-pcie 3500000.pcie: host bridge /soc/pcie@3500000 ranges: [ 3.623960] layerscape-pcie 3500000.pcie: MEM 0x8840000000..0x887fffffff -> 0x0040000000 [ 3.632548] layerscape-pcie 3500000.pcie: iATU unroll: disabled [ 3.638481] layerscape-pcie 3500000.pcie: Detected iATU regions: 8 outbound, 8 inbound [ 3.746460] layerscape-pcie 3500000.pcie: Link up [ 3.751490] layerscape-pcie 3500000.pcie: PCI host bridge to bus 0001:00 [ 3.758210] pci_bus 0001:00: root bus resource [bus 00-ff] [ 3.763713] pci_bus 0001:00: root bus resource [mem 0x8840000000-0x887fffffff] (bus address [0x40000000-0x7fffffff]) [ 3.774286] pci 0001:00:00.0: [1957:8d80] type 01 class 0x060400 [ 3.780456] pci 0001:00:00.0: supports D1 D2 [ 3.784741] pci 0001:00:00.0: PME# supported from D0 D1 D2 D3hot [ 3.792301] pci 0001:01:00.0: [1957:4002] type 00 class 0x000000 [ 3.798427] pci 0001:01:00.0: reg 0x10: [mem 0x8840000000-0x88400fffff 64bit pref] [ 3.806053] pci 0001:01:00.0: reg 0x18: [mem 0x8840100000-0x88401fffff] [ 3.812723] pci 0001:01:00.0: reg 0x1c: [io 0x1000-0x10ff] [ 3.818351] pci 0001:01:00.0: reg 0x20: [mem 0x8840200000-0x88402fffff] [ 3.825023] pci 0001:01:00.0: reg 0x24: [mem 0x8840300000-0x88403fffff] [ 3.831693] pci 0001:01:00.0: reg 0x30: [mem 0x8840400000-0x884040ffff pref] [ 3.839142] pci 0001:01:00.0: supports D1 [ 3.843165] pci 0001:01:00.0: PME# supported from D0 D1 D3hot D3cold [ 3.849737] pci 0001:01:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x2 link at 0001:00:00.0 (capable of 15.752 Gb/s with 8.0 GT/s PCIe x2 link) [ 3.865871] pci 0001:00:00.0: BAR 14: assigned [mem 0x8840000000-0x88403fffff] [ 3.873111] pci 0001:00:00.0: BAR 15: assigned [mem 0x8840400000-0x88404fffff 64bit pref] [ 3.881302] pci 0001:00:00.0: BAR 13: no space for [io size 0x1000] [ 3.887668] pci 0001:00:00.0: BAR 13: failed to assign [io size 0x1000] [ 3.894384] pci 0001:00:00.0: PCI bridge to [bus 01-ff] [ 3.899626] pci 0001:00:00.0: bridge window [mem 0x8840000000-0x88403fffff] [ 3.906778] pci 0001:00:00.0: bridge window [mem 0x8840400000-0x88404fffff 64bit pref] [ 3.915327] layerscape-pcie 3600000.pcie: host bridge /soc/pcie@3600000 ranges: [ 3.922695] layerscape-pcie 3600000.pcie: MEM 0x9040000000..0x907fffffff -> 0x0040000000 [ 3.931658] layerscape-pcie 3600000.pcie: iATU unroll: disabled [ 3.937587] layerscape-pcie 3600000.pcie: Detected iATU regions: 256 outbound, 24 inbound [ 4.946287] layerscape-pcie 3600000.pcie: Phy link never came up [ 4.952621] layerscape-pcie 3600000.pcie: PCI host bridge to bus 0002:00 [ 4.959342] pci_bus 0002:00: root bus resource [bus 00-ff] [ 4.964846] pci_bus 0002:00: root bus resource [mem 0x9040000000-0x907fffffff] (bus address [0x40000000-0x7fffffff]) [ 4.975420] pci 0002:00:00.0: [1957:8d80] type 01 class 0x060400 [ 4.981603] pci 0002:00:00.0: supports D1 D2 [ 4.985886] pci 0002:00:00.0: PME# supported from D0 D1 D2 D3hot [ 4.993941] pci 0002:00:00.0: PCI bridge to [bus 01-ff] [ 4.999605] layerscape-pcie 3800000.pcie: host bridge /soc/pcie@3800000 ranges: [ 5.006973] layerscape-pcie 3800000.pcie: MEM 0xa040000000..0xa07fffffff -> 0x0040000000 [ 5.016091] layerscape-pcie 3800000.pcie: iATU unroll: disabled [ 5.022023] layerscape-pcie 3800000.pcie: Detected iATU regions: 256 outbound, 24 inbound [ 6.030799] layerscape-pcie 3800000.pcie: Phy link never came up [ 6.037113] layerscape-pcie 3800000.pcie: PCI host bridge to bus 0003:00 [ 6.043833] pci_bus 0003:00: root bus resource [bus 00-ff] [ 6.049336] pci_bus 0003:00: root bus resource [mem 0xa040000000-0xa07fffffff] (bus address [0x40000000-0x7fffffff]) [ 6.059912] pci 0003:00:00.0: [1957:8d80] type 01 class 0x060400 [ 6.066109] pci 0003:00:00.0: supports D1 D2 [ 6.070392] pci 0003:00:00.0: PME# supported from D0 D1 D2 D3hot [ 6.078457] pci 0003:00:00.0: PCI bridge to [bus 01-ff] [ 6.085845] EINJ: ACPI disabled. [ 6.121151] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.133080] SuperH (H)SCI(F) driver initialized [ 6.138419] msm_serial: driver initialized [ 6.147208] arm-smmu 5000000.iommu: probing hardware configuration... [ 6.153677] arm-smmu 5000000.iommu: SMMUv2 with: [ 6.158310] arm-smmu 5000000.iommu: stage 1 translation [ 6.163635] arm-smmu 5000000.iommu: stage 2 translation [ 6.168959] arm-smmu 5000000.iommu: nested translation [ 6.174196] arm-smmu 5000000.iommu: coherent table walk [ 6.179520] arm-smmu 5000000.iommu: (IDR0.CTTW overridden by FW configuration) [ 6.186845] arm-smmu 5000000.iommu: stream matching with 128 register groups [ 6.193999] arm-smmu 5000000.iommu: 64 context banks (0 stage-2 only) [ 6.200547] arm-smmu 5000000.iommu: Supported page sizes: 0x61311000 [ 6.207000] arm-smmu 5000000.iommu: Stage-1: 48-bit VA -> 48-bit IPA [ 6.213453] arm-smmu 5000000.iommu: Stage-2: 48-bit IPA -> 48-bit PA [ 6.251758] brd: module loaded [ 6.298301] loop: module loaded [ 6.303363] megasas: 07.717.02.00-rc1 [ 6.315088] spi-nor spi0.0: mt35xu512aba (65536 Kbytes) [ 6.329741] spi-nor spi0.1: mt35xu512aba (65536 Kbytes) [ 6.351635] libphy: Fixed MDIO Bus: probed [ 6.361042] tun: Universal TUN/TAP device driver, 1.6 [ 6.366645] CAN device driver interface [ 6.376344] thunder_xcv, ver 1.0 [ 6.379680] thunder_bgx, ver 1.0 [ 6.383029] nicpf, ver 1.0 [ 6.387457] libphy: Freescale XGMAC MDIO Bus: probed [ 6.397953] mv88x3310 0x0000000008b96000:00: PHY failed to boot firmware, status=ffff [ 6.411222] mv88x3310 0x0000000008b96000:08: PHY failed to boot firmware, status=ffff [ 6.420253] i2c 0-0077: Linked as a sync state only consumer to 0x0000000008b96000:01 [ 6.429250] i2c 0-0077: Linked as a sync state only consumer to 0x0000000008b96000:02 [ 6.438742] libphy: Freescale XGMAC MDIO Bus: probed [ 6.448826] mv88x3310 0x0000000008b97000:00: PHY failed to boot firmware, status=ffff [ 6.462097] mv88x3310 0x0000000008b97000:08: PHY failed to boot firmware, status=ffff [ 6.471559] libphy: Freescale XGMAC MDIO Bus: probed [ 6.479068] libphy: Freescale XGMAC MDIO Bus: probed [ 6.486585] libphy: Freescale XGMAC MDIO Bus: probed [ 6.494898] hclge is initializing [ 6.498345] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version [ 6.505579] hns3: Copyright (c) 2017 Huawei Corporation. [ 6.511048] e1000: Intel(R) PRO/1000 Network Driver [ 6.515934] e1000: Copyright (c) 1999-2006 Intel Corporation. [ 6.521805] e1000e: Intel(R) PRO/1000 Network Driver [ 6.526777] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. [ 6.532823] igb: Intel(R) Gigabit Ethernet Network Driver [ 6.538231] igb: Copyright (c) 2007-2014 Intel Corporation. [ 6.543895] igbvf: Intel(R) Gigabit Virtual Function Network Driver [ 6.550169] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. [ 6.556694] sky2: driver version 1.30 [ 6.563181] tlan: ThunderLAN driver v1.17 [ 6.567312] tlan: 0 devices installed, PCI: 0 EISA: 0 [ 6.573916] VFIO - User Level meta-driver version: 0.3 [ 6.580381] dwc3 3100000.usb: Linked as a consumer to 5000000.iommu [ 6.586783] dwc3 3100000.usb: Adding to iommu group 0 [ 6.595924] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 6.602466] ehci-pci: EHCI PCI platform driver [ 6.606996] ehci-platform: EHCI generic platform driver [ 6.612405] ehci-orion: EHCI orion driver [ 6.616588] ehci-exynos: EHCI Exynos driver [ 6.620947] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 6.627169] ohci-pci: OHCI PCI platform driver [ 6.631702] ohci-platform: OHCI generic platform driver [ 6.637119] ohci-exynos: OHCI Exynos driver [ 6.641563] usbcore: registered new interface driver cdc_acm [ 6.647231] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters [ 6.655342] usbcore: registered new interface driver usb-storage [ 6.661514] usbcore: registered new interface driver usbserial_generic [ 6.668170] usbserial: USB Serial support registered for generic [ 6.674255] usbcore: registered new interface driver aircable [ 6.680053] usbserial: USB Serial support registered for aircable [ 6.686210] usbcore: registered new interface driver ark3116 [ 6.691920] usbserial: USB Serial support registered for ark3116 [ 6.697993] usbcore: registered new interface driver belkin_sa [ 6.703877] usbserial: USB Serial support registered for Belkin / Peracom / GoHubs USB Serial Adapter [ 6.713164] usbcore: registered new interface driver ch341 [ 6.718700] usbserial: USB Serial support registered for ch341-uart [ 6.725030] usbcore: registered new interface driver cp210x [ 6.730653] usbserial: USB Serial support registered for cp210x [ 6.736636] usbcore: registered new interface driver cyberjack [ 6.742520] usbserial: USB Serial support registered for Reiner SCT Cyberjack USB card reader [ 6.751110] usbcore: registered new interface driver cypress_m8 [ 6.757088] usbserial: USB Serial support registered for DeLorme Earthmate USB [ 6.764362] usbserial: USB Serial support registered for HID->COM RS232 Adapter [ 6.771720] usbserial: USB Serial support registered for Nokia CA-42 V2 Adapter [ 6.779091] usbcore: registered new interface driver digi_acceleport [ 6.785496] usbserial: USB Serial support registered for Digi 2 port USB adapter [ 6.792942] usbserial: USB Serial support registered for Digi 4 port USB adapter [ 6.800401] usbcore: registered new interface driver io_edgeport [ 6.806460] usbserial: USB Serial support registered for Edgeport 2 port adapter [ 6.813907] usbserial: USB Serial support registered for Edgeport 4 port adapter [ 6.821354] usbserial: USB Serial support registered for Edgeport 8 port adapter [ 6.828800] usbserial: USB Serial support registered for EPiC device [ 6.835215] usbcore: registered new interface driver io_ti [ 6.840753] usbserial: USB Serial support registered for Edgeport TI 1 port adapter [ 6.848461] usbserial: USB Serial support registered for Edgeport TI 2 port adapter [ 6.856184] usbcore: registered new interface driver empeg [ 6.861725] usbserial: USB Serial support registered for empeg [ 6.867632] usbcore: registered new interface driver f81534a_ctrl [ 6.873786] usbcore: registered new interface driver f81232 [ 6.879409] usbserial: USB Serial support registered for f81232 [ 6.885380] usbserial: USB Serial support registered for f81534a [ 6.891452] usbcore: registered new interface driver f81534 [ 6.897075] usbserial: USB Serial support registered for Fintek F81532/F81534 [ 6.904274] usbcore: registered new interface driver ftdi_sio [ 6.910069] usbserial: USB Serial support registered for FTDI USB Serial Device [ 6.917440] usbcore: registered new interface driver garmin_gps [ 6.923411] usbserial: USB Serial support registered for Garmin GPS usb/tty [ 6.930434] usbcore: registered new interface driver ipaq [ 6.935888] usbserial: USB Serial support registered for PocketPC PDA [ 6.942390] usbcore: registered new interface driver ipw [ 6.947751] usbserial: USB Serial support registered for IPWireless converter [ 6.954950] usbcore: registered new interface driver ir_usb [ 6.960575] usbserial: USB Serial support registered for IR Dongle [ 6.966820] usbcore: registered new interface driver iuu_phoenix [ 6.972880] usbserial: USB Serial support registered for iuu_phoenix [ 6.979295] usbcore: registered new interface driver keyspan [ 6.985005] usbserial: USB Serial support registered for Keyspan - (without firmware) [ 6.992886] usbserial: USB Serial support registered for Keyspan 1 port adapter [ 7.000246] usbserial: USB Serial support registered for Keyspan 2 port adapter [ 7.007605] usbserial: USB Serial support registered for Keyspan 4 port adapter [ 7.014976] usbcore: registered new interface driver keyspan_pda [ 7.021036] usbserial: USB Serial support registered for Keyspan PDA [ 7.027440] usbserial: USB Serial support registered for Keyspan PDA - (prerenumeration) [ 7.035593] usbcore: registered new interface driver kl5kusb105 [ 7.041564] usbserial: USB Serial support registered for KL5KUSB105D / PalmConnect [ 7.049196] usbcore: registered new interface driver kobil_sct [ 7.055079] usbserial: USB Serial support registered for KOBIL USB smart card terminal [ 7.063057] usbcore: registered new interface driver mct_u232 [ 7.068858] usbserial: USB Serial support registered for MCT U232 [ 7.075013] usbcore: registered new interface driver metro_usb [ 7.080897] usbserial: USB Serial support registered for Metrologic USB to Serial [ 7.088444] usbcore: registered new interface driver mos7720 [ 7.094154] usbserial: USB Serial support registered for Moschip 2 port adapter [ 7.101527] usbcore: registered new interface driver mos7840 [ 7.107238] usbserial: USB Serial support registered for Moschip 7840/7820 USB Serial Driver [ 7.115740] usbcore: registered new interface driver mxuport [ 7.121450] usbserial: USB Serial support registered for MOXA UPort [ 7.127796] usbcore: registered new interface driver navman [ 7.133419] usbserial: USB Serial support registered for navman [ 7.139402] usbcore: registered new interface driver omninet [ 7.145122] usbserial: USB Serial support registered for ZyXEL - omni.net usb [ 7.152323] usbcore: registered new interface driver opticon [ 7.158033] usbserial: USB Serial support registered for opticon [ 7.164101] usbcore: registered new interface driver option [ 7.169727] usbserial: USB Serial support registered for GSM modem (1-port) [ 7.176751] usbcore: registered new interface driver oti6858 [ 7.182461] usbserial: USB Serial support registered for oti6858 [ 7.188531] usbcore: registered new interface driver pl2303 [ 7.194172] usbserial: USB Serial support registered for pl2303 [ 7.200156] usbcore: registered new interface driver qcaux [ 7.205694] usbserial: USB Serial support registered for qcaux [ 7.211589] usbcore: registered new interface driver qcserial [ 7.217386] usbserial: USB Serial support registered for Qualcomm USB modem [ 7.224411] usbcore: registered new interface driver quatech2 [ 7.230210] usbserial: USB Serial support registered for Quatech 2nd gen USB to Serial Driver [ 7.238796] usbcore: registered new interface driver safe_serial [ 7.244854] usbserial: USB Serial support registered for safe_serial [ 7.251268] usbcore: registered new interface driver sierra [ 7.256891] usbserial: USB Serial support registered for Sierra USB modem [ 7.263742] usbcore: registered new interface driver usb_serial_simple [ 7.270323] usbserial: USB Serial support registered for carelink [ 7.276471] usbserial: USB Serial support registered for zio [ 7.282181] usbserial: USB Serial support registered for funsoft [ 7.288239] usbserial: USB Serial support registered for flashloader [ 7.294644] usbserial: USB Serial support registered for google [ 7.300626] usbserial: USB Serial support registered for libtransistor [ 7.307204] usbserial: USB Serial support registered for vivopay [ 7.313264] usbserial: USB Serial support registered for moto_modem [ 7.319583] usbserial: USB Serial support registered for motorola_tetra [ 7.326251] usbserial: USB Serial support registered for novatel_gps [ 7.332656] usbserial: USB Serial support registered for hp4x [ 7.338454] usbserial: USB Serial support registered for suunto [ 7.344424] usbserial: USB Serial support registered for siemens_mpi [ 7.350842] usbcore: registered new interface driver spcp8x5 [ 7.356552] usbserial: USB Serial support registered for SPCP8x5 [ 7.362620] usbcore: registered new interface driver ssu100 [ 7.368248] usbserial: USB Serial support registered for Quatech SSU-100 USB to Serial Driver [ 7.376836] usbcore: registered new interface driver symbolserial [ 7.382980] usbserial: USB Serial support registered for symbol [ 7.388963] usbcore: registered new interface driver ti_usb_3410_5052 [ 7.395454] usbserial: USB Serial support registered for TI USB 3410 1 port adapter [ 7.403162] usbserial: USB Serial support registered for TI USB 5052 2 port adapter [ 7.410883] usbcore: registered new interface driver upd78f0730 [ 7.416854] usbserial: USB Serial support registered for upd78f0730 [ 7.423183] usbcore: registered new interface driver visor [ 7.428723] usbserial: USB Serial support registered for Handspring Visor / Palm OS [ 7.436431] usbserial: USB Serial support registered for Sony Clie 5.0 [ 7.443009] usbserial: USB Serial support registered for Sony Clie 3.5 [ 7.449600] usbcore: registered new interface driver wishbone_serial [ 7.456007] usbserial: USB Serial support registered for wishbone_serial [ 7.462771] usbcore: registered new interface driver whiteheat [ 7.468655] usbserial: USB Serial support registered for Connect Tech - WhiteHEAT - (prerenumeration) [ 7.477926] usbserial: USB Serial support registered for Connect Tech - WhiteHEAT [ 7.485471] usbcore: registered new interface driver xsens_mt [ 7.491268] usbserial: USB Serial support registered for xsens_mt [ 7.502466] i2c /dev entries driver [ 7.510828] i2c i2c-0: Added multiplexed i2c bus 2 [ 7.516281] i2c i2c-0: Added multiplexed i2c bus 3 [ 7.522021] i2c i2c-0: Added multiplexed i2c bus 4 [ 7.527794] i2c 5-004c: Linked as a consumer to regulator-sb3v3 [ 7.534193] i2c 5-0048: Linked as a consumer to regulator-sb3v3 [ 7.540291] i2c i2c-0: Added multiplexed i2c bus 5 [ 7.546497] genirq: Setting trigger mode 8 for irq 55 failed (gic_set_type+0x0/0x180) [ 7.554464] [ 7.554465] ============================= [ 7.554466] [ BUG: Invalid wait context ] [ 7.554467] 5.14.0-rc5+ #440 Not tainted [ 7.554469] ----------------------------- [ 7.554469] swapper/0/1 is trying to lock: [ 7.554470] ffff309701582898 (&port_lock_key){....}-{3:3}, at: pl011_console_write+0x134/0x210 [ 7.554475] other info that might help us debug this: [ 7.554476] context-{5:5} [ 7.554477] 6 locks held by swapper/0/1: [ 7.554478] #0: ffff3097018861b0 (&dev->mutex){....}-{4:4}, at: __device_driver_lock+0x34/0x70 [ 7.554483] #1: ffff3097010da1b0 (&dev->mutex){....}-{4:4}, at: __device_attach+0x40/0x148 [ 7.554487] #2: ffff3097015bb688 (&desc->request_mutex){+.+.}-{4:4}, at: __setup_irq+0xb8/0x798 [ 7.554491] #3: ffff3097015bb4f8 (&irq_desc_lock_class){-...}-{2:2}, at: __setup_irq+0xdc/0x798 [ 7.554496] #4: ffffd5d447719200 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x154/0x328 [ 7.554500] #5: ffffd5d447719388 (console_owner){....}-{0:0}, at: console_unlock+0x210/0x658 [ 7.554504] stack backtrace: [ 7.554505] CPU: 4 PID: 1 Comm: swapper/0 Not tainted 5.14.0-rc5+ #440 [ 7.554506] Hardware name: NXP Layerscape LX2160ABLUEBOX3 (DT) [ 7.554507] Call trace: [ 7.554508] dump_backtrace+0x0/0x1c0 [ 7.554509] show_stack+0x18/0x28 [ 7.554510] dump_stack_lvl+0x8c/0xb8 [ 7.554511] dump_stack+0x18/0x34 [ 7.554512] __lock_acquire+0x14a8/0x1500 [ 7.554513] lock_acquire+0x314/0x3a8 [ 7.554513] _raw_spin_lock+0x6c/0x110 [ 7.554514] pl011_console_write+0x134/0x210 [ 7.554515] console_unlock+0x5f4/0x658 [ 7.554516] vprintk_emit+0x15c/0x328 [ 7.554517] vprintk_default+0x38/0x48 [ 7.554518] vprintk+0xb4/0x2a8 [ 7.554519] printk+0x68/0x8c [ 7.554520] __irq_set_trigger+0xa0/0x190 [ 7.554521] __setup_irq+0x68c/0x798 [ 7.554521] request_threaded_irq+0xf8/0x1d0 [ 7.554522] devm_request_threaded_irq+0x84/0x108 [ 7.554523] pcf2127_probe.isra.10+0x290/0x4b8 [ 7.554524] pcf2127_i2c_probe+0x94/0xe0 [ 7.554525] i2c_device_probe+0x328/0x358 [ 7.554526] really_probe+0xb4/0x308 [ 7.554527] __driver_probe_device+0x7c/0xe8 [ 7.554528] driver_probe_device+0x8c/0x130 [ 7.554529] __device_attach_driver+0x98/0xd0 [ 7.554530] bus_for_each_drv+0x68/0xd0 [ 7.554530] __device_attach+0xec/0x148 [ 7.554531] device_initial_probe+0x14/0x20 [ 7.554532] bus_probe_device+0x9c/0xa8 [ 7.554533] device_add+0x408/0x8e8 [ 7.554534] device_register+0x20/0x30 [ 7.554535] i2c_new_client_device+0x1a0/0x288 [ 7.554536] of_i2c_register_device+0x80/0xa8 [ 7.554537] of_i2c_register_devices+0xac/0x1a0 [ 7.554538] i2c_register_adapter+0x2b8/0x6a0 [ 7.554539] i2c_add_adapter+0x80/0xd8 [ 7.554540] i2c_mux_add_adapter+0x280/0x530 [ 7.554541] pca954x_probe+0x2b0/0x470 [ 7.554541] i2c_device_probe+0x328/0x358 [ 7.554542] really_probe+0xb4/0x308 [ 7.554543] __driver_probe_device+0x7c/0xe8 [ 7.554544] driver_probe_device+0x8c/0x130 [ 7.554545] __driver_attach+0x9c/0x100 [ 7.554546] bus_for_each_dev+0x7c/0xd8 [ 7.554547] driver_attach+0x24/0x30 [ 7.554548] bus_add_driver+0x158/0x200 [ 7.554549] driver_register+0x64/0x120 [ 7.554550] i2c_register_driver+0x48/0xb0 [ 7.554551] pca954x_driver_init+0x1c/0x28 [ 7.554552] do_one_initcall+0x88/0x410 [ 7.554553] kernel_init_freeable+0x328/0x39c [ 7.554554] kernel_init+0x24/0x128 [ 7.554555] ret_from_fork+0x10/0x18 [ 7.867477] rtc-pcf2127-i2c 6-0051: failed to request alarm irq [ 7.873459] rtc-pcf2127-i2c: probe of 6-0051 failed with error -22 [ 7.879679] i2c i2c-0: Added multiplexed i2c bus 6 [ 7.884658] i2c i2c-0: Added multiplexed i2c bus 7 [ 7.889643] i2c i2c-0: Added multiplexed i2c bus 8 [ 7.894823] i2c 9-0075: Linked as a sync state only consumer to 0x0000000008b96000:01 [ 7.902744] i2c 9-0075: Linked as a sync state only consumer to 0x0000000008b96000:02 [ 7.911910] i2c 10-0028: Linked as a sync state only consumer to 0x0000000008b96000:01 [ 7.919908] i2c 10-0028: Linked as a sync state only consumer to 0x0000000008b96000:02 [ 7.928362] spi spi1.0: Linked as a sync state only consumer to 0x0000000008b96000:01 [ 7.943497] sja1105 spi1.0: Probed switch chip: SJA1110A [ 7.949125] spi spi1.2: Linked as a sync state only consumer to 0x0000000008b96000:02 [ 7.963340] sja1105 spi1.2: Probed switch chip: SJA1110A [ 7.968746] sc18is602 10-0028: Dropping the link to 0x0000000008b96000:01 [ 7.975568] sc18is602 10-0028: Dropping the link to 0x0000000008b96000:02 [ 7.982447] i2c i2c-9: Added multiplexed i2c bus 10 [ 7.987515] i2c i2c-9: Added multiplexed i2c bus 11 [ 7.992614] i2c i2c-9: Added multiplexed i2c bus 12 [ 7.997706] i2c i2c-9: Added multiplexed i2c bus 13 [ 8.002766] i2c i2c-9: Added multiplexed i2c bus 14 [ 8.007829] i2c i2c-9: Added multiplexed i2c bus 15 [ 8.012877] i2c i2c-9: Added multiplexed i2c bus 16 [ 8.017932] i2c i2c-9: Added multiplexed i2c bus 17 [ 8.022814] pca954x 9-0075: registered 8 multiplexed busses for I2C mux pca9547 [ 8.030130] pca954x 9-0075: Dropping the link to 0x0000000008b96000:01 [ 8.036698] pca954x 9-0075: Dropping the link to 0x0000000008b96000:02 [ 8.043320] i2c i2c-0: Added multiplexed i2c bus 9 [ 8.048112] pca954x 0-0077: registered 8 multiplexed busses for I2C mux pca9547 [ 8.055440] pca954x 0-0077: Dropping the link to regulator-sb3v3 [ 8.061480] pca954x 0-0077: Dropping the link to 0x0000000008b96000:01 [ 8.068041] pca954x 0-0077: Dropping the link to 0x0000000008b96000:02 [ 8.076133] i2c i2c-1: Added multiplexed i2c bus 18 [ 8.081205] i2c i2c-1: Added multiplexed i2c bus 19 [ 8.086263] i2c i2c-1: Added multiplexed i2c bus 20 [ 8.091325] i2c i2c-1: Added multiplexed i2c bus 21 [ 8.096208] pca954x 1-0077: registered 4 multiplexed busses for I2C switch pca9846 [ 8.105888] sbsa-gwdt 23a0000.watchdog: Initialized with 30s timeout @ 25000050 Hz, action=0. [enabled] [ 8.118030] sdhci: Secure Digital Host Controller Interface driver [ 8.124216] sdhci: Copyright(c) Pierre Ossman [ 8.129049] Synopsys Designware Multimedia Card Interface Driver [ 8.135758] sdhci-pltfm: SDHCI platform and OF driver helper [ 8.141984] sdhci-esdhc 2150000.esdhc: Linked as a consumer to 5000000.iommu [ 8.149084] sdhci-esdhc 2150000.esdhc: Adding to iommu group 1 [ 8.149105] sdhci-esdhc 2140000.esdhc: Linked as a consumer to 5000000.iommu [ 8.161986] sdhci-esdhc 2140000.esdhc: Adding to iommu group 2 [ 8.168313] ledtrig-cpu: registered to indicate activity on CPUs [ 8.175015] SMCCC: SOC_ID: ARCH_FEATURES(ARCH_SOC_ID) returned error: fffffffffffffffd [ 8.183506] usbcore: registered new interface driver usbhid [ 8.187157] mmc1: SDHCI controller on 2150000.esdhc [2150000.esdhc] using ADMA 64-bit [ 8.189078] usbhid: USB HID core driver [ 8.193629] mmc0: SDHCI controller on 2140000.esdhc [2140000.esdhc] using ADMA 64-bit [ 8.213167] GACT probability NOT on [ 8.216671] Mirror/redirect action on [ 8.220366] u32 classifier [ 8.223063] input device check on [ 8.226720] Actions configured [ 8.230175] Initializing XFRM netlink socket [ 8.234539] NET: Registered PF_INET6 protocol family [ 8.240232] Segment Routing with IPv6 [ 8.243912] In-situ OAM (IOAM) with IPv6 [ 8.247865] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver [ 8.254117] NET: Registered PF_PACKET protocol family [ 8.259181] NET: Registered PF_KEY protocol family [ 8.264005] can: controller area network core [ 8.268388] NET: Registered PF_CAN protocol family [ 8.270836] mmc0: new ultra high speed SDR104 SDHC card at address aaaa [ 8.273178] can: raw protocol [ 8.280291] mmcblk0: mmc0:aaaa SD32G 29.7 GiB [ 8.282745] can: broadcast manager protocol [ 8.291361] can: netlink gateway - max_hops=1 [ 8.291988] mmcblk0: p1 p2 [ 8.295781] 8021q: 802.1Q VLAN Support v1.8 [ 8.301381] random: fast init done [ 8.302700] 9pnet: Installing 9P2000 support [ 8.306072] mmc1: new HS400 MMC card at address 0001 [ 8.310362] Key type dns_resolver registered [ 8.315789] mmcblk1: mmc1:0001 S0J59X 119 GiB [ 8.319819] registered taskstats version 1 [ 8.324243] mmcblk1boot0: mmc1:0001 S0J59X 31.5 MiB [ 8.328078] Loading compiled-in X.509 certificates [ 8.333300] mmcblk1boot1: mmc1:0001 S0J59X 31.5 MiB [ 8.338219] Btrfs loaded, crc32c=crc32c-generic, zoned=no [ 8.342913] mmcblk1rpmb: mmc1:0001 S0J59X 4.00 MiB, chardev (511:0) [ 8.353108] fsl_mc_dprc dprc.1: Linked as a consumer to 5000000.iommu [ 8.360903] fsl_mc_dprc dprc.1: Adding to iommu group 3 [ 8.461866] fsl_mc_allocator dpbp.5: Linked as a consumer to 5000000.iommu [ 8.468758] fsl_mc_allocator dpbp.5: Adding to iommu group 3 [ 8.474571] fsl_mc_allocator dpbp.4: Linked as a consumer to 5000000.iommu [ 8.481453] fsl_mc_allocator dpbp.4: Adding to iommu group 3 [ 8.487259] fsl_mc_allocator dpbp.3: Linked as a consumer to 5000000.iommu [ 8.494140] fsl_mc_allocator dpbp.3: Adding to iommu group 3 [ 8.499948] fsl_mc_allocator dpbp.2: Linked as a consumer to 5000000.iommu [ 8.506828] fsl_mc_allocator dpbp.2: Adding to iommu group 3 [ 8.512633] fsl_mc_allocator dpbp.1: Linked as a consumer to 5000000.iommu [ 8.519514] fsl_mc_allocator dpbp.1: Adding to iommu group 3 [ 8.525323] fsl_mc_allocator dpbp.0: Linked as a consumer to 5000000.iommu [ 8.532204] fsl_mc_allocator dpbp.0: Adding to iommu group 3 [ 8.539018] fsl_mc_allocator dpmcp.39: Linked as a consumer to 5000000.iommu [ 8.546072] fsl_mc_allocator dpmcp.39: Adding to iommu group 3 [ 8.552558] fsl_mc_allocator dpmcp.38: Linked as a consumer to 5000000.iommu [ 8.559612] fsl_mc_allocator dpmcp.38: Adding to iommu group 3 [ 8.566099] fsl_mc_allocator dpmcp.37: Linked as a consumer to 5000000.iommu [ 8.573153] fsl_mc_allocator dpmcp.37: Adding to iommu group 3 [ 8.579639] fsl_mc_allocator dpmcp.36: Linked as a consumer to 5000000.iommu [ 8.586694] fsl_mc_allocator dpmcp.36: Adding to iommu group 3 [ 8.593180] fsl_mc_allocator dpmcp.35: Linked as a consumer to 5000000.iommu [ 8.600234] fsl_mc_allocator dpmcp.35: Adding to iommu group 3 [ 8.606720] fsl_mc_allocator dpmcp.34: Linked as a consumer to 5000000.iommu [ 8.613775] fsl_mc_allocator dpmcp.34: Adding to iommu group 3 [ 8.620268] fsl_mc_allocator dpmcp.33: Linked as a consumer to 5000000.iommu [ 8.627322] fsl_mc_allocator dpmcp.33: Adding to iommu group 3 [ 8.633811] fsl_mc_allocator dpmcp.32: Linked as a consumer to 5000000.iommu [ 8.640865] fsl_mc_allocator dpmcp.32: Adding to iommu group 3 [ 8.647359] fsl_mc_allocator dpmcp.31: Linked as a consumer to 5000000.iommu [ 8.654414] fsl_mc_allocator dpmcp.31: Adding to iommu group 3 [ 8.660905] fsl_mc_allocator dpmcp.30: Linked as a consumer to 5000000.iommu [ 8.667959] fsl_mc_allocator dpmcp.30: Adding to iommu group 3 [ 8.674447] fsl_mc_allocator dpmcp.29: Linked as a consumer to 5000000.iommu [ 8.681503] fsl_mc_allocator dpmcp.29: Adding to iommu group 3 [ 8.687999] fsl_mc_allocator dpmcp.28: Linked as a consumer to 5000000.iommu [ 8.695054] fsl_mc_allocator dpmcp.28: Adding to iommu group 3 [ 8.701545] fsl_mc_allocator dpmcp.27: Linked as a consumer to 5000000.iommu [ 8.708599] fsl_mc_allocator dpmcp.27: Adding to iommu group 3 [ 8.715091] fsl_mc_allocator dpmcp.26: Linked as a consumer to 5000000.iommu [ 8.722145] fsl_mc_allocator dpmcp.26: Adding to iommu group 3 [ 8.728636] fsl_mc_allocator dpmcp.25: Linked as a consumer to 5000000.iommu [ 8.735690] fsl_mc_allocator dpmcp.25: Adding to iommu group 3 [ 8.742185] fsl_mc_allocator dpmcp.24: Linked as a consumer to 5000000.iommu [ 8.749239] fsl_mc_allocator dpmcp.24: Adding to iommu group 3 [ 8.755731] fsl_mc_allocator dpmcp.23: Linked as a consumer to 5000000.iommu [ 8.762787] fsl_mc_allocator dpmcp.23: Adding to iommu group 3 [ 8.769279] fsl_mc_allocator dpmcp.22: Linked as a consumer to 5000000.iommu [ 8.776334] fsl_mc_allocator dpmcp.22: Adding to iommu group 3 [ 8.782827] fsl_mc_allocator dpmcp.21: Linked as a consumer to 5000000.iommu [ 8.789882] fsl_mc_allocator dpmcp.21: Adding to iommu group 3 [ 8.796383] fsl_mc_allocator dpmcp.20: Linked as a consumer to 5000000.iommu [ 8.803436] fsl_mc_allocator dpmcp.20: Adding to iommu group 3 [ 8.809932] fsl_mc_allocator dpmcp.19: Linked as a consumer to 5000000.iommu [ 8.816986] fsl_mc_allocator dpmcp.19: Adding to iommu group 3 [ 8.823481] fsl_mc_allocator dpmcp.18: Linked as a consumer to 5000000.iommu [ 8.830535] fsl_mc_allocator dpmcp.18: Adding to iommu group 3 [ 8.837030] fsl_mc_allocator dpmcp.17: Linked as a consumer to 5000000.iommu [ 8.844085] fsl_mc_allocator dpmcp.17: Adding to iommu group 3 [ 8.850581] fsl_mc_allocator dpmcp.16: Linked as a consumer to 5000000.iommu [ 8.857648] fsl_mc_allocator dpmcp.16: Adding to iommu group 3 [ 8.864153] fsl_mc_allocator dpmcp.15: Linked as a consumer to 5000000.iommu [ 8.871207] fsl_mc_allocator dpmcp.15: Adding to iommu group 3 [ 8.877705] fsl_mc_allocator dpmcp.14: Linked as a consumer to 5000000.iommu [ 8.884760] fsl_mc_allocator dpmcp.14: Adding to iommu group 3 [ 8.891258] fsl_mc_allocator dpmcp.13: Linked as a consumer to 5000000.iommu [ 8.898313] fsl_mc_allocator dpmcp.13: Adding to iommu group 3 [ 8.904810] fsl_mc_allocator dpmcp.12: Linked as a consumer to 5000000.iommu [ 8.911874] fsl_mc_allocator dpmcp.12: Adding to iommu group 3 [ 8.918373] fsl_mc_allocator dpmcp.11: Linked as a consumer to 5000000.iommu [ 8.925427] fsl_mc_allocator dpmcp.11: Adding to iommu group 3 [ 8.931935] fsl_mc_allocator dpmcp.10: Linked as a consumer to 5000000.iommu [ 8.938989] fsl_mc_allocator dpmcp.10: Adding to iommu group 3 [ 8.945493] fsl_mc_allocator dpmcp.9: Linked as a consumer to 5000000.iommu [ 8.952460] fsl_mc_allocator dpmcp.9: Adding to iommu group 3 [ 8.958875] fsl_mc_allocator dpmcp.8: Linked as a consumer to 5000000.iommu [ 8.965843] fsl_mc_allocator dpmcp.8: Adding to iommu group 3 [ 8.972263] fsl_mc_allocator dpmcp.7: Linked as a consumer to 5000000.iommu [ 8.979230] fsl_mc_allocator dpmcp.7: Adding to iommu group 3 [ 8.985647] fsl_mc_allocator dpmcp.6: Linked as a consumer to 5000000.iommu [ 8.992614] fsl_mc_allocator dpmcp.6: Adding to iommu group 3 [ 8.999029] fsl_mc_allocator dpmcp.5: Linked as a consumer to 5000000.iommu [ 9.005997] fsl_mc_allocator dpmcp.5: Adding to iommu group 3 [ 9.012420] fsl_mc_allocator dpmcp.4: Linked as a consumer to 5000000.iommu [ 9.019387] fsl_mc_allocator dpmcp.4: Adding to iommu group 3 [ 9.025806] fsl_mc_allocator dpmcp.3: Linked as a consumer to 5000000.iommu [ 9.032773] fsl_mc_allocator dpmcp.3: Adding to iommu group 3 [ 9.039191] fsl_mc_allocator dpmcp.2: Linked as a consumer to 5000000.iommu [ 9.046159] fsl_mc_allocator dpmcp.2: Adding to iommu group 3 [ 9.052574] fsl_mc_allocator dpmcp.1: Linked as a consumer to 5000000.iommu [ 9.059541] fsl_mc_allocator dpmcp.1: Adding to iommu group 3 [ 9.065458] fsl_mc_allocator dpcon.95: Linked as a consumer to 5000000.iommu [ 9.072514] fsl_mc_allocator dpcon.95: Adding to iommu group 3 [ 9.078515] fsl_mc_allocator dpcon.94: Linked as a consumer to 5000000.iommu [ 9.085570] fsl_mc_allocator dpcon.94: Adding to iommu group 3 [ 9.091573] fsl_mc_allocator dpcon.93: Linked as a consumer to 5000000.iommu [ 9.098628] fsl_mc_allocator dpcon.93: Adding to iommu group 3 [ 9.104630] fsl_mc_allocator dpcon.92: Linked as a consumer to 5000000.iommu [ 9.111685] fsl_mc_allocator dpcon.92: Adding to iommu group 3 [ 9.117690] fsl_mc_allocator dpcon.91: Linked as a consumer to 5000000.iommu [ 9.124745] fsl_mc_allocator dpcon.91: Adding to iommu group 3 [ 9.130746] fsl_mc_allocator dpcon.90: Linked as a consumer to 5000000.iommu [ 9.137800] fsl_mc_allocator dpcon.90: Adding to iommu group 3 [ 9.143815] fsl_mc_allocator dpcon.89: Linked as a consumer to 5000000.iommu [ 9.150868] fsl_mc_allocator dpcon.89: Adding to iommu group 3 [ 9.156874] fsl_mc_allocator dpcon.88: Linked as a consumer to 5000000.iommu [ 9.163928] fsl_mc_allocator dpcon.88: Adding to iommu group 3 [ 9.169944] fsl_mc_allocator dpcon.87: Linked as a consumer to 5000000.iommu [ 9.176999] fsl_mc_allocator dpcon.87: Adding to iommu group 3 [ 9.183003] fsl_mc_allocator dpcon.86: Linked as a consumer to 5000000.iommu [ 9.190057] fsl_mc_allocator dpcon.86: Adding to iommu group 3 [ 9.196068] fsl_mc_allocator dpcon.85: Linked as a consumer to 5000000.iommu [ 9.203122] fsl_mc_allocator dpcon.85: Adding to iommu group 3 [ 9.209128] fsl_mc_allocator dpcon.84: Linked as a consumer to 5000000.iommu [ 9.216183] fsl_mc_allocator dpcon.84: Adding to iommu group 3 [ 9.222192] fsl_mc_allocator dpcon.83: Linked as a consumer to 5000000.iommu [ 9.229246] fsl_mc_allocator dpcon.83: Adding to iommu group 3 [ 9.235254] fsl_mc_allocator dpcon.82: Linked as a consumer to 5000000.iommu [ 9.242310] fsl_mc_allocator dpcon.82: Adding to iommu group 3 [ 9.248321] fsl_mc_allocator dpcon.81: Linked as a consumer to 5000000.iommu [ 9.255376] fsl_mc_allocator dpcon.81: Adding to iommu group 3 [ 9.261384] fsl_mc_allocator dpcon.80: Linked as a consumer to 5000000.iommu [ 9.268439] fsl_mc_allocator dpcon.80: Adding to iommu group 3 [ 9.274450] fsl_mc_allocator dpcon.79: Linked as a consumer to 5000000.iommu [ 9.281505] fsl_mc_allocator dpcon.79: Adding to iommu group 3 [ 9.287517] fsl_mc_allocator dpcon.78: Linked as a consumer to 5000000.iommu [ 9.294571] fsl_mc_allocator dpcon.78: Adding to iommu group 3 [ 9.300584] fsl_mc_allocator dpcon.77: Linked as a consumer to 5000000.iommu [ 9.307638] fsl_mc_allocator dpcon.77: Adding to iommu group 3 [ 9.313649] fsl_mc_allocator dpcon.76: Linked as a consumer to 5000000.iommu [ 9.320703] fsl_mc_allocator dpcon.76: Adding to iommu group 3 [ 9.326717] fsl_mc_allocator dpcon.75: Linked as a consumer to 5000000.iommu [ 9.333771] fsl_mc_allocator dpcon.75: Adding to iommu group 3 [ 9.339794] fsl_mc_allocator dpcon.74: Linked as a consumer to 5000000.iommu [ 9.346847] fsl_mc_allocator dpcon.74: Adding to iommu group 3 [ 9.352862] fsl_mc_allocator dpcon.73: Linked as a consumer to 5000000.iommu [ 9.359917] fsl_mc_allocator dpcon.73: Adding to iommu group 3 [ 9.365941] fsl_mc_allocator dpcon.72: Linked as a consumer to 5000000.iommu [ 9.372996] fsl_mc_allocator dpcon.72: Adding to iommu group 3 [ 9.379017] fsl_mc_allocator dpcon.71: Linked as a consumer to 5000000.iommu [ 9.386072] fsl_mc_allocator dpcon.71: Adding to iommu group 3 [ 9.392088] fsl_mc_allocator dpcon.70: Linked as a consumer to 5000000.iommu [ 9.399142] fsl_mc_allocator dpcon.70: Adding to iommu group 3 [ 9.405158] fsl_mc_allocator dpcon.69: Linked as a consumer to 5000000.iommu [ 9.412213] fsl_mc_allocator dpcon.69: Adding to iommu group 3 [ 9.418231] fsl_mc_allocator dpcon.68: Linked as a consumer to 5000000.iommu [ 9.425285] fsl_mc_allocator dpcon.68: Adding to iommu group 3 [ 9.431312] fsl_mc_allocator dpcon.67: Linked as a consumer to 5000000.iommu [ 9.438366] fsl_mc_allocator dpcon.67: Adding to iommu group 3 [ 9.444384] fsl_mc_allocator dpcon.66: Linked as a consumer to 5000000.iommu [ 9.451438] fsl_mc_allocator dpcon.66: Adding to iommu group 3 [ 9.457458] fsl_mc_allocator dpcon.65: Linked as a consumer to 5000000.iommu [ 9.464512] fsl_mc_allocator dpcon.65: Adding to iommu group 3 [ 9.470530] fsl_mc_allocator dpcon.64: Linked as a consumer to 5000000.iommu [ 9.477585] fsl_mc_allocator dpcon.64: Adding to iommu group 3 [ 9.483605] fsl_mc_allocator dpcon.63: Linked as a consumer to 5000000.iommu [ 9.490660] fsl_mc_allocator dpcon.63: Adding to iommu group 3 [ 9.496680] fsl_mc_allocator dpcon.62: Linked as a consumer to 5000000.iommu [ 9.503735] fsl_mc_allocator dpcon.62: Adding to iommu group 3 [ 9.509757] fsl_mc_allocator dpcon.61: Linked as a consumer to 5000000.iommu [ 9.516811] fsl_mc_allocator dpcon.61: Adding to iommu group 3 [ 9.522831] fsl_mc_allocator dpcon.60: Linked as a consumer to 5000000.iommu [ 9.529885] fsl_mc_allocator dpcon.60: Adding to iommu group 3 [ 9.535917] fsl_mc_allocator dpcon.59: Linked as a consumer to 5000000.iommu [ 9.542973] fsl_mc_allocator dpcon.59: Adding to iommu group 3 [ 9.548995] fsl_mc_allocator dpcon.58: Linked as a consumer to 5000000.iommu [ 9.556050] fsl_mc_allocator dpcon.58: Adding to iommu group 3 [ 9.562073] fsl_mc_allocator dpcon.57: Linked as a consumer to 5000000.iommu [ 9.569128] fsl_mc_allocator dpcon.57: Adding to iommu group 3 [ 9.575150] fsl_mc_allocator dpcon.56: Linked as a consumer to 5000000.iommu [ 9.582204] fsl_mc_allocator dpcon.56: Adding to iommu group 3 [ 9.588232] fsl_mc_allocator dpcon.55: Linked as a consumer to 5000000.iommu [ 9.595285] fsl_mc_allocator dpcon.55: Adding to iommu group 3 [ 9.601308] fsl_mc_allocator dpcon.54: Linked as a consumer to 5000000.iommu [ 9.608363] fsl_mc_allocator dpcon.54: Adding to iommu group 3 [ 9.614388] fsl_mc_allocator dpcon.53: Linked as a consumer to 5000000.iommu [ 9.621443] fsl_mc_allocator dpcon.53: Adding to iommu group 3 [ 9.627469] fsl_mc_allocator dpcon.52: Linked as a consumer to 5000000.iommu [ 9.634523] fsl_mc_allocator dpcon.52: Adding to iommu group 3 [ 9.640551] fsl_mc_allocator dpcon.51: Linked as a consumer to 5000000.iommu [ 9.647605] fsl_mc_allocator dpcon.51: Adding to iommu group 3 [ 9.653632] fsl_mc_allocator dpcon.50: Linked as a consumer to 5000000.iommu [ 9.660686] fsl_mc_allocator dpcon.50: Adding to iommu group 3 [ 9.666715] fsl_mc_allocator dpcon.49: Linked as a consumer to 5000000.iommu [ 9.673771] fsl_mc_allocator dpcon.49: Adding to iommu group 3 [ 9.679805] fsl_mc_allocator dpcon.48: Linked as a consumer to 5000000.iommu [ 9.686860] fsl_mc_allocator dpcon.48: Adding to iommu group 3 [ 9.692897] fsl_mc_allocator dpcon.47: Linked as a consumer to 5000000.iommu [ 9.699953] fsl_mc_allocator dpcon.47: Adding to iommu group 3 [ 9.705980] fsl_mc_allocator dpcon.46: Linked as a consumer to 5000000.iommu [ 9.713034] fsl_mc_allocator dpcon.46: Adding to iommu group 3 [ 9.719065] fsl_mc_allocator dpcon.45: Linked as a consumer to 5000000.iommu [ 9.726119] fsl_mc_allocator dpcon.45: Adding to iommu group 3 [ 9.732151] fsl_mc_allocator dpcon.44: Linked as a consumer to 5000000.iommu [ 9.739205] fsl_mc_allocator dpcon.44: Adding to iommu group 3 [ 9.745234] fsl_mc_allocator dpcon.43: Linked as a consumer to 5000000.iommu [ 9.752289] fsl_mc_allocator dpcon.43: Adding to iommu group 3 [ 9.758320] fsl_mc_allocator dpcon.42: Linked as a consumer to 5000000.iommu [ 9.765375] fsl_mc_allocator dpcon.42: Adding to iommu group 3 [ 9.771408] fsl_mc_allocator dpcon.41: Linked as a consumer to 5000000.iommu [ 9.778462] fsl_mc_allocator dpcon.41: Adding to iommu group 3 [ 9.784495] fsl_mc_allocator dpcon.40: Linked as a consumer to 5000000.iommu [ 9.791549] fsl_mc_allocator dpcon.40: Adding to iommu group 3 [ 9.797584] fsl_mc_allocator dpcon.39: Linked as a consumer to 5000000.iommu [ 9.804639] fsl_mc_allocator dpcon.39: Adding to iommu group 3 [ 9.810671] fsl_mc_allocator dpcon.38: Linked as a consumer to 5000000.iommu [ 9.817726] fsl_mc_allocator dpcon.38: Adding to iommu group 3 [ 9.823762] fsl_mc_allocator dpcon.37: Linked as a consumer to 5000000.iommu [ 9.830817] fsl_mc_allocator dpcon.37: Adding to iommu group 3 [ 9.836851] fsl_mc_allocator dpcon.36: Linked as a consumer to 5000000.iommu [ 9.843907] fsl_mc_allocator dpcon.36: Adding to iommu group 3 [ 9.849943] fsl_mc_allocator dpcon.35: Linked as a consumer to 5000000.iommu [ 9.856997] fsl_mc_allocator dpcon.35: Adding to iommu group 3 [ 9.863034] fsl_mc_allocator dpcon.34: Linked as a consumer to 5000000.iommu [ 9.870088] fsl_mc_allocator dpcon.34: Adding to iommu group 3 [ 9.876137] fsl_mc_allocator dpcon.33: Linked as a consumer to 5000000.iommu [ 9.883191] fsl_mc_allocator dpcon.33: Adding to iommu group 3 [ 9.889228] fsl_mc_allocator dpcon.32: Linked as a consumer to 5000000.iommu [ 9.896283] fsl_mc_allocator dpcon.32: Adding to iommu group 3 [ 9.902321] fsl_mc_allocator dpcon.31: Linked as a consumer to 5000000.iommu [ 9.909376] fsl_mc_allocator dpcon.31: Adding to iommu group 3 [ 9.915413] fsl_mc_allocator dpcon.30: Linked as a consumer to 5000000.iommu [ 9.922467] fsl_mc_allocator dpcon.30: Adding to iommu group 3 [ 9.928511] fsl_mc_allocator dpcon.29: Linked as a consumer to 5000000.iommu [ 9.935565] fsl_mc_allocator dpcon.29: Adding to iommu group 3 [ 9.941604] fsl_mc_allocator dpcon.28: Linked as a consumer to 5000000.iommu [ 9.948659] fsl_mc_allocator dpcon.28: Adding to iommu group 3 [ 9.954705] fsl_mc_allocator dpcon.27: Linked as a consumer to 5000000.iommu [ 9.961760] fsl_mc_allocator dpcon.27: Adding to iommu group 3 [ 9.967807] fsl_mc_allocator dpcon.26: Linked as a consumer to 5000000.iommu [ 9.974863] fsl_mc_allocator dpcon.26: Adding to iommu group 3 [ 9.980907] fsl_mc_allocator dpcon.25: Linked as a consumer to 5000000.iommu [ 9.987962] fsl_mc_allocator dpcon.25: Adding to iommu group 3 [ 9.994003] fsl_mc_allocator dpcon.24: Linked as a consumer to 5000000.iommu [ 10.001057] fsl_mc_allocator dpcon.24: Adding to iommu group 3 [ 10.007101] fsl_mc_allocator dpcon.23: Linked as a consumer to 5000000.iommu [ 10.014155] fsl_mc_allocator dpcon.23: Adding to iommu group 3 [ 10.020199] fsl_mc_allocator dpcon.22: Linked as a consumer to 5000000.iommu [ 10.027253] fsl_mc_allocator dpcon.22: Adding to iommu group 3 [ 10.033297] fsl_mc_allocator dpcon.21: Linked as a consumer to 5000000.iommu [ 10.040352] fsl_mc_allocator dpcon.21: Adding to iommu group 3 [ 10.046394] fsl_mc_allocator dpcon.20: Linked as a consumer to 5000000.iommu [ 10.053449] fsl_mc_allocator dpcon.20: Adding to iommu group 3 [ 10.059496] fsl_mc_allocator dpcon.19: Linked as a consumer to 5000000.iommu [ 10.066550] fsl_mc_allocator dpcon.19: Adding to iommu group 3 [ 10.072595] fsl_mc_allocator dpcon.18: Linked as a consumer to 5000000.iommu [ 10.079649] fsl_mc_allocator dpcon.18: Adding to iommu group 3 [ 10.085696] fsl_mc_allocator dpcon.17: Linked as a consumer to 5000000.iommu [ 10.092751] fsl_mc_allocator dpcon.17: Adding to iommu group 3 [ 10.098798] fsl_mc_allocator dpcon.16: Linked as a consumer to 5000000.iommu [ 10.105853] fsl_mc_allocator dpcon.16: Adding to iommu group 3 [ 10.111909] fsl_mc_allocator dpcon.15: Linked as a consumer to 5000000.iommu [ 10.118966] fsl_mc_allocator dpcon.15: Adding to iommu group 3 [ 10.125014] fsl_mc_allocator dpcon.14: Linked as a consumer to 5000000.iommu [ 10.132069] fsl_mc_allocator dpcon.14: Adding to iommu group 3 [ 10.138118] fsl_mc_allocator dpcon.13: Linked as a consumer to 5000000.iommu [ 10.145174] fsl_mc_allocator dpcon.13: Adding to iommu group 3 [ 10.151224] fsl_mc_allocator dpcon.12: Linked as a consumer to 5000000.iommu [ 10.158278] fsl_mc_allocator dpcon.12: Adding to iommu group 3 [ 10.164327] fsl_mc_allocator dpcon.11: Linked as a consumer to 5000000.iommu [ 10.171381] fsl_mc_allocator dpcon.11: Adding to iommu group 3 [ 10.177431] fsl_mc_allocator dpcon.10: Linked as a consumer to 5000000.iommu [ 10.184485] fsl_mc_allocator dpcon.10: Adding to iommu group 3 [ 10.190538] fsl_mc_allocator dpcon.9: Linked as a consumer to 5000000.iommu [ 10.197506] fsl_mc_allocator dpcon.9: Adding to iommu group 3 [ 10.203471] fsl_mc_allocator dpcon.8: Linked as a consumer to 5000000.iommu [ 10.210438] fsl_mc_allocator dpcon.8: Adding to iommu group 3 [ 10.216412] fsl_mc_allocator dpcon.7: Linked as a consumer to 5000000.iommu [ 10.223379] fsl_mc_allocator dpcon.7: Adding to iommu group 3 [ 10.229347] fsl_mc_allocator dpcon.6: Linked as a consumer to 5000000.iommu [ 10.236315] fsl_mc_allocator dpcon.6: Adding to iommu group 3 [ 10.242281] fsl_mc_allocator dpcon.5: Linked as a consumer to 5000000.iommu [ 10.249249] fsl_mc_allocator dpcon.5: Adding to iommu group 3 [ 10.255216] fsl_mc_allocator dpcon.4: Linked as a consumer to 5000000.iommu [ 10.262184] fsl_mc_allocator dpcon.4: Adding to iommu group 3 [ 10.268152] fsl_mc_allocator dpcon.3: Linked as a consumer to 5000000.iommu [ 10.275121] fsl_mc_allocator dpcon.3: Adding to iommu group 3 [ 10.281088] fsl_mc_allocator dpcon.2: Linked as a consumer to 5000000.iommu [ 10.288057] fsl_mc_allocator dpcon.2: Adding to iommu group 3 [ 10.294027] fsl_mc_allocator dpcon.1: Linked as a consumer to 5000000.iommu [ 10.300995] fsl_mc_allocator dpcon.1: Adding to iommu group 3 [ 10.306966] fsl_mc_allocator dpcon.0: Linked as a consumer to 5000000.iommu [ 10.313934] fsl_mc_allocator dpcon.0: Adding to iommu group 3 [ 10.319917] fsl_dpaa2_eth dpni.5: Linked as a consumer to 5000000.iommu [ 10.326539] fsl_dpaa2_eth dpni.5: Adding to iommu group 3 [ 10.332601] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpmcp.39 [ 10.372876] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.95 [ 10.412736] fsl_dpaa2_eth dpni.5: Dropping the link to dpcon.95 [ 10.418700] fsl_dpaa2_eth dpni.5: Dropping the link to dpmcp.39 [ 10.424909] fsl_dpaa2_eth dpni.4: Linked as a consumer to 5000000.iommu [ 10.431534] fsl_dpaa2_eth dpni.4: Adding to iommu group 3 [ 10.437568] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpmcp.38 [ 10.477841] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.94 [ 10.517717] fsl_dpaa2_eth dpni.4: Dropping the link to dpcon.94 [ 10.523700] fsl_dpaa2_eth dpni.4: Dropping the link to dpmcp.38 [ 10.529940] fsl_dpaa2_eth dpni.3: Linked as a consumer to 5000000.iommu [ 10.536567] fsl_dpaa2_eth dpni.3: Adding to iommu group 3 [ 10.542606] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpmcp.37 [ 10.582872] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.93 [ 10.622748] fsl_dpaa2_eth dpni.3: Dropping the link to dpcon.93 [ 10.628723] fsl_dpaa2_eth dpni.3: Dropping the link to dpmcp.37 [ 10.634950] fsl_dpaa2_eth dpni.2: Linked as a consumer to 5000000.iommu [ 10.641577] fsl_dpaa2_eth dpni.2: Adding to iommu group 3 [ 10.647620] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpmcp.36 [ 10.687885] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.92 [ 10.727737] fsl_dpaa2_eth dpni.2: Dropping the link to dpcon.92 [ 10.733702] fsl_dpaa2_eth dpni.2: Dropping the link to dpmcp.36 [ 10.739915] fsl_dpaa2_eth dpni.1: Linked as a consumer to 5000000.iommu [ 10.746537] fsl_dpaa2_eth dpni.1: Adding to iommu group 3 [ 10.752567] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpmcp.35 [ 10.792832] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.91 [ 10.832648] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.91 [ 10.838614] fsl_dpaa2_eth dpni.1: Dropping the link to dpmcp.35 [ 10.844828] fsl_dpaa2_eth dpni.0: Linked as a consumer to 5000000.iommu [ 10.851450] fsl_dpaa2_eth dpni.0: Adding to iommu group 3 [ 10.857478] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpmcp.34 [ 10.897746] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.90 [ 10.937569] fsl_dpaa2_eth dpni.0: Dropping the link to dpcon.90 [ 10.943537] fsl_dpaa2_eth dpni.0: Dropping the link to dpmcp.34 [ 10.950749] fsl_dpaa2_ptp dprtc.0: Linked as a consumer to 5000000.iommu [ 10.957462] fsl_dpaa2_ptp dprtc.0: Adding to iommu group 3 [ 10.963545] fsl_dpaa2_ptp dprtc.0: Linked as a consumer to dpmcp.33 [ 10.971576] ptp_qoriq: device tree node missing required elements, try automatic configuration [ 10.980337] pps pps0: new PPS source ptp0 [ 10.986262] fsl_mc_dpio dpio.15: Linked as a consumer to 5000000.iommu [ 10.992811] fsl_mc_dpio dpio.15: Adding to iommu group 3 [ 10.998716] fsl_mc_dpio dpio.15: Linked as a consumer to dpmcp.32 [ 11.007954] fsl_mc_dpio dpio.15: probed [ 11.014046] fsl_mc_dpio dpio.14: Linked as a consumer to 5000000.iommu [ 11.020583] fsl_mc_dpio dpio.14: Adding to iommu group 3 [ 11.026472] fsl_mc_dpio dpio.14: Linked as a consumer to dpmcp.31 [ 11.035684] fsl_mc_dpio dpio.14: probed [ 11.041777] fsl_mc_dpio dpio.13: Linked as a consumer to 5000000.iommu [ 11.048316] fsl_mc_dpio dpio.13: Adding to iommu group 3 [ 11.054206] fsl_mc_dpio dpio.13: Linked as a consumer to dpmcp.30 [ 11.063417] fsl_mc_dpio dpio.13: probed [ 11.069516] fsl_mc_dpio dpio.12: Linked as a consumer to 5000000.iommu [ 11.076053] fsl_mc_dpio dpio.12: Adding to iommu group 3 [ 11.081941] fsl_mc_dpio dpio.12: Linked as a consumer to dpmcp.29 [ 11.091150] fsl_mc_dpio dpio.12: probed [ 11.097251] fsl_mc_dpio dpio.11: Linked as a consumer to 5000000.iommu [ 11.103786] fsl_mc_dpio dpio.11: Adding to iommu group 3 [ 11.109676] fsl_mc_dpio dpio.11: Linked as a consumer to dpmcp.28 [ 11.118896] fsl_mc_dpio dpio.11: probed [ 11.124999] fsl_mc_dpio dpio.10: Linked as a consumer to 5000000.iommu [ 11.131534] fsl_mc_dpio dpio.10: Adding to iommu group 3 [ 11.137425] fsl_mc_dpio dpio.10: Linked as a consumer to dpmcp.27 [ 11.146635] fsl_mc_dpio dpio.10: probed [ 11.152738] fsl_mc_dpio dpio.9: Linked as a consumer to 5000000.iommu [ 11.159186] fsl_mc_dpio dpio.9: Adding to iommu group 3 [ 11.164990] fsl_mc_dpio dpio.9: Linked as a consumer to dpmcp.26 [ 11.174114] fsl_mc_dpio dpio.9: probed [ 11.180129] fsl_mc_dpio dpio.8: Linked as a consumer to 5000000.iommu [ 11.186578] fsl_mc_dpio dpio.8: Adding to iommu group 3 [ 11.192385] fsl_mc_dpio dpio.8: Linked as a consumer to dpmcp.25 [ 11.201509] fsl_mc_dpio dpio.8: probed [ 11.207512] fsl_mc_dpio dpio.7: Linked as a consumer to 5000000.iommu [ 11.213960] fsl_mc_dpio dpio.7: Adding to iommu group 3 [ 11.219762] fsl_mc_dpio dpio.7: Linked as a consumer to dpmcp.24 [ 11.228897] fsl_mc_dpio dpio.7: probed [ 11.234904] fsl_mc_dpio dpio.6: Linked as a consumer to 5000000.iommu [ 11.241354] fsl_mc_dpio dpio.6: Adding to iommu group 3 [ 11.247155] fsl_mc_dpio dpio.6: Linked as a consumer to dpmcp.23 [ 11.256288] fsl_mc_dpio dpio.6: probed [ 11.262293] fsl_mc_dpio dpio.5: Linked as a consumer to 5000000.iommu [ 11.268744] fsl_mc_dpio dpio.5: Adding to iommu group 3 [ 11.274546] fsl_mc_dpio dpio.5: Linked as a consumer to dpmcp.22 [ 11.283670] fsl_mc_dpio dpio.5: probed [ 11.289681] fsl_mc_dpio dpio.4: Linked as a consumer to 5000000.iommu [ 11.296136] fsl_mc_dpio dpio.4: Adding to iommu group 3 [ 11.301936] fsl_mc_dpio dpio.4: Linked as a consumer to dpmcp.21 [ 11.311058] fsl_mc_dpio dpio.4: probed [ 11.317079] fsl_mc_dpio dpio.3: Linked as a consumer to 5000000.iommu [ 11.323528] fsl_mc_dpio dpio.3: Adding to iommu group 3 [ 11.329330] fsl_mc_dpio dpio.3: Linked as a consumer to dpmcp.20 [ 11.338451] fsl_mc_dpio dpio.3: probed [ 11.344472] fsl_mc_dpio dpio.2: Linked as a consumer to 5000000.iommu [ 11.350920] fsl_mc_dpio dpio.2: Adding to iommu group 3 [ 11.356724] fsl_mc_dpio dpio.2: Linked as a consumer to dpmcp.19 [ 11.365847] fsl_mc_dpio dpio.2: probed [ 11.371866] fsl_mc_dpio dpio.1: Linked as a consumer to 5000000.iommu [ 11.378313] fsl_mc_dpio dpio.1: Adding to iommu group 3 [ 11.384118] fsl_mc_dpio dpio.1: Linked as a consumer to dpmcp.18 [ 11.393241] fsl_mc_dpio dpio.1: probed [ 11.399250] fsl_mc_dpio dpio.0: Linked as a consumer to 5000000.iommu [ 11.405701] fsl_mc_dpio dpio.0: Adding to iommu group 3 [ 11.411503] fsl_mc_dpio dpio.0: Linked as a consumer to dpmcp.17 [ 11.420629] fsl_mc_dpio dpio.0: probed [ 11.428066] fsl_mc_dprc dprc.1: DPRC device bound to driver [ 11.433768] pcieport 0000:00:00.0: Linked as a consumer to 5000000.iommu [ 11.440498] pcieport 0000:00:00.0: Adding to iommu group 4 [ 11.446159] pcieport 0000:00:00.0: PME: Signaling with IRQ 373 [ 11.452239] pcieport 0001:00:00.0: Linked as a consumer to 5000000.iommu [ 11.458966] pcieport 0001:00:00.0: Adding to iommu group 5 [ 11.464596] pcieport 0001:00:00.0: PME: Signaling with IRQ 374 [ 11.470641] pcieport 0002:00:00.0: Linked as a consumer to 5000000.iommu [ 11.477367] pcieport 0002:00:00.0: Adding to iommu group 6 [ 11.482993] pcieport 0002:00:00.0: PME: Signaling with IRQ 375 [ 11.489041] pcieport 0003:00:00.0: Linked as a consumer to 5000000.iommu [ 11.495773] pcieport 0003:00:00.0: Adding to iommu group 7 [ 11.501412] pcieport 0003:00:00.0: PME: Signaling with IRQ 376 [ 11.513888] sja1105 spi1.0: Probed switch chip: SJA1110A [ 11.525754] sja1105 spi1.2: Probed switch chip: SJA1110A [ 11.531850] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpmcp.16 [ 11.572125] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.89 [ 11.580360] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.15 [ 11.586997] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.88 [ 11.595226] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.14 [ 11.601865] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.87 [ 11.610091] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.13 [ 11.616727] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.86 [ 11.624962] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.12 [ 11.631601] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.85 [ 11.639836] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.11 [ 11.646471] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.84 [ 11.654699] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.10 [ 11.661335] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.83 [ 11.669566] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.9 [ 11.676122] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.82 [ 11.684354] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.8 [ 11.690903] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.81 [ 11.699130] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.7 [ 11.705679] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.80 [ 11.713906] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.6 [ 11.720458] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.79 [ 11.728689] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.5 [ 11.735240] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.78 [ 11.743467] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.4 [ 11.750017] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.77 [ 11.758243] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.3 [ 11.764795] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.76 [ 11.773027] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.2 [ 11.779577] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.75 [ 11.787809] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.1 [ 11.794360] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpcon.74 [ 11.802586] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpio.0 [ 11.809207] fsl_dpaa2_eth dpni.5: Linked as a consumer to dpbp.5 [ 11.891236] fsl_dpaa2_eth dpni.5: Probed interface eth0 [ 11.897192] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpmcp.15 [ 11.938466] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.73 [ 11.946693] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.15 [ 11.953329] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.72 [ 11.961562] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.14 [ 11.968203] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.71 [ 11.976448] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.13 [ 11.983088] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.70 [ 11.991316] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.12 [ 11.997956] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.69 [ 12.006184] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.11 [ 12.012821] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.68 [ 12.021064] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.10 [ 12.027700] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.67 [ 12.035933] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.9 [ 12.042482] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.66 [ 12.050707] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.8 [ 12.057256] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.65 [ 12.065488] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.7 [ 12.072044] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.64 [ 12.080276] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.6 [ 12.086828] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.63 [ 12.095056] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.5 [ 12.101607] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.62 [ 12.109834] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.4 [ 12.116389] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.61 [ 12.124621] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.3 [ 12.131169] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.60 [ 12.139394] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.2 [ 12.145950] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.59 [ 12.154177] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.1 [ 12.160726] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpcon.58 [ 12.168958] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpio.0 [ 12.175564] fsl_dpaa2_eth dpni.4: Linked as a consumer to dpbp.4 [ 12.256556] fsl_dpaa2_eth dpni.4: Probed interface eth1 [ 12.262495] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpmcp.14 [ 12.302777] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.57 [ 12.311005] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.15 [ 12.317643] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.56 [ 12.325868] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.14 [ 12.332506] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.55 [ 12.340737] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.13 [ 12.347376] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.54 [ 12.355605] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.12 [ 12.362245] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.53 [ 12.370474] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.11 [ 12.377113] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.52 [ 12.385346] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.10 [ 12.391989] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.51 [ 12.400223] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.9 [ 12.406779] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.50 [ 12.415007] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.8 [ 12.421556] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.49 [ 12.429791] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.7 [ 12.436340] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.48 [ 12.444573] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.6 [ 12.451121] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.47 [ 12.459351] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.5 [ 12.465904] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.46 [ 12.474133] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.4 [ 12.480684] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.45 [ 12.488918] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.3 [ 12.495467] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.44 [ 12.503693] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.2 [ 12.510243] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.43 [ 12.518469] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.1 [ 12.525018] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpcon.42 [ 12.533259] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpio.0 [ 12.539873] fsl_dpaa2_eth dpni.3: Linked as a consumer to dpbp.3 [ 12.619556] fsl_dpaa2_eth dpni.3 (unnamed net_device) (uninitialized): PHY [0x0000000008b97000:00] driver [Generic Clause 45 PHY] (irq=POLL) [ 12.633450] fsl_dpaa2_eth dpni.3: Probed interface eth2 [ 12.639402] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpmcp.13 [ 12.679670] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.41 [ 12.687902] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.15 [ 12.694539] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.40 [ 12.702766] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.14 [ 12.709403] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.39 [ 12.717633] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.13 [ 12.724275] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.38 [ 12.732505] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.12 [ 12.739142] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.37 [ 12.747370] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.11 [ 12.754007] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.36 [ 12.762237] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.10 [ 12.768871] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.35 [ 12.777105] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.9 [ 12.783653] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.34 [ 12.791885] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.8 [ 12.798434] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.33 [ 12.806661] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.7 [ 12.813210] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.32 [ 12.821443] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.6 [ 12.827993] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.31 [ 12.836224] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.5 [ 12.842772] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.30 [ 12.851000] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.4 [ 12.857561] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.29 [ 12.865793] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.3 [ 12.872341] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.28 [ 12.880573] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.2 [ 12.887121] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.27 [ 12.895348] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.1 [ 12.901896] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpcon.26 [ 12.910124] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpio.0 [ 12.916729] fsl_dpaa2_eth dpni.2: Linked as a consumer to dpbp.2 [ 12.996910] fsl_dpaa2_eth dpni.2 (unnamed net_device) (uninitialized): PHY [0x0000000008b97000:08] driver [Generic Clause 45 PHY] (irq=POLL) [ 13.010772] fsl_dpaa2_eth dpni.2: Probed interface eth3 [ 13.016718] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpmcp.12 [ 13.056991] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.25 [ 13.065225] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.15 [ 13.071869] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.24 [ 13.080105] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.14 [ 13.086741] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.23 [ 13.094968] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.13 [ 13.101605] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.22 [ 13.109831] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.12 [ 13.116466] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.21 [ 13.124701] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.11 [ 13.131340] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.20 [ 13.139568] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.10 [ 13.146206] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.19 [ 13.154435] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.9 [ 13.160988] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.18 [ 13.169221] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.8 [ 13.175781] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.17 [ 13.184011] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.7 [ 13.190563] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.16 [ 13.198794] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.6 [ 13.205346] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.15 [ 13.213578] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.5 [ 13.220136] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.14 [ 13.228368] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.4 [ 13.234919] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.13 [ 13.243144] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.3 [ 13.249692] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.12 [ 13.257917] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.2 [ 13.264465] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.11 [ 13.272699] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.1 [ 13.279252] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.10 [ 13.287497] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.0 [ 13.294110] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpbp.1 [ 13.422401] fsl_dpaa2_eth dpni.1: Dropping the link to dpbp.1 [ 13.428201] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.0 [ 13.433981] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.10 [ 13.439932] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.1 [ 13.445710] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.11 [ 13.451659] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.2 [ 13.457435] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.12 [ 13.463383] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.3 [ 13.469156] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.13 [ 13.475103] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.4 [ 13.480881] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.14 [ 13.486827] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.5 [ 13.492601] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.15 [ 13.498550] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.6 [ 13.504327] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.16 [ 13.510276] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.7 [ 13.516051] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.17 [ 13.522001] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.8 [ 13.527782] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.18 [ 13.533730] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.9 [ 13.539507] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.19 [ 13.545457] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.10 [ 13.551317] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.20 [ 13.557264] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.11 [ 13.563127] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.21 [ 13.569078] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.12 [ 13.574938] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.22 [ 13.580885] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.13 [ 13.586748] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.23 [ 13.592698] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.14 [ 13.598559] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.24 [ 13.604506] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.15 [ 13.610377] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.25 [ 13.616326] fsl_dpaa2_eth dpni.1: Dropping the link to dpmcp.12 [ 13.623084] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpmcp.11 [ 13.663371] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.9 [ 13.671516] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.15 [ 13.678155] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.8 [ 13.686298] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.14 [ 13.692934] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.7 [ 13.701077] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.13 [ 13.707712] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.6 [ 13.715856] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.12 [ 13.722489] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.5 [ 13.730628] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.11 [ 13.737264] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.4 [ 13.745408] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.10 [ 13.752048] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.3 [ 13.760191] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.9 [ 13.766739] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.2 [ 13.774878] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.8 [ 13.781427] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.1 [ 13.789572] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.7 [ 13.796126] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.0 [ 13.804270] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.6 [ 13.810817] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.95 [ 13.819043] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.5 [ 13.825592] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.94 [ 13.833821] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.4 [ 13.840369] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.93 [ 13.848599] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.3 [ 13.855147] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.92 [ 13.863375] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.2 [ 13.869927] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.91 [ 13.878161] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.1 [ 13.884712] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpcon.90 [ 13.892942] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpio.0 [ 13.899560] fsl_dpaa2_eth dpni.0: Linked as a consumer to dpbp.0 [ 13.979724] fsl_dpaa2_eth dpni.0 (unnamed net_device) (uninitialized): PHY [0x0000000008b96000:08] driver [Generic Clause 45 PHY] (irq=POLL) [ 13.993607] fsl_dpaa2_eth dpni.0: Probed interface eth4 [ 14.005432] sja1105 spi1.0: Probed switch chip: SJA1110A [ 14.011175] sja1105 spi1.0: Dropping the link to 0x0000000008b96000:01 [ 14.024328] sja1105 spi1.2: Probed switch chip: SJA1110A [ 14.033399] sja1105 spi1.0: H topology detected, cutting RX from DSA link 4 to CPU port 2 to prevent TX packet loops [ 14.288275] libphy: SJA1105 PCS MDIO bus: probed [ 14.445009] libphy: SJA1110 100base-T1 MDIO bus: probed [ 14.950047] sja1105 spi1.2: H topology detected, cutting RX from DSA link 1 to CPU port 2 to prevent TX packet loops [ 15.204881] libphy: SJA1105 PCS MDIO bus: probed [ 15.361009] libphy: SJA1110 100base-T1 MDIO bus: probed [ 15.874265] random: crng init done [ 15.937311] sja1105 spi1.0: configuring for fixed/rgmii-id link mode [ 15.955906] sja1105 spi1.0: Link is Up - 1Gbps/Full - flow control off [ 16.004458] sja1105 spi1.0 1ge_p1 (uninitialized): PHY [0x0000000008b96000:01] driver [Qualcomm Atheros AR8035] (irq=POLL) [ 16.026114] sja1105 spi1.0: configuring for fixed/sgmii link mode [ 16.115819] sja1105 spi1.0: Link is Up - 1Gbps/Full - flow control off [ 16.231439] sja1105 spi1.0 trx1 (uninitialized): PHY [spi1.0-base-t1:01] driver [NXP C45 TJA1103] (irq=POLL) [ 16.365831] sja1105 spi1.0 trx2 (uninitialized): PHY [spi1.0-base-t1:02] driver [NXP C45 TJA1103] (irq=POLL) [ 16.500214] sja1105 spi1.0 trx3 (uninitialized): PHY [spi1.0-base-t1:03] driver [NXP C45 TJA1103] (irq=POLL) [ 16.634609] sja1105 spi1.0 trx4 (uninitialized): PHY [spi1.0-base-t1:04] driver [NXP C45 TJA1103] (irq=POLL) [ 16.768983] sja1105 spi1.0 trx5 (uninitialized): PHY [spi1.0-base-t1:05] driver [NXP C45 TJA1103] (irq=POLL) [ 16.903366] sja1105 spi1.0 trx6 (uninitialized): PHY [spi1.0-base-t1:06] driver [NXP C45 TJA1103] (irq=POLL) [ 16.929408] sja1105 spi1.2: configuring for fixed/sgmii link mode [ 17.010299] sja1105 spi1.2: configuring for fixed/rgmii-id link mode [ 17.021336] sja1105 spi1.2: Link is Up - 1Gbps/Full - flow control off [ 17.033629] sja1105 spi1.2: Link is Up - 1Gbps/Full - flow control off [ 17.076457] sja1105 spi1.2 1ge_p2 (uninitialized): PHY [0x0000000008b96000:02] driver [Qualcomm Atheros AR8035] (irq=POLL) [ 17.212704] sja1105 spi1.2 trx7 (uninitialized): PHY [spi1.2-base-t1:01] driver [NXP C45 TJA1103] (irq=POLL) [ 17.347049] sja1105 spi1.2 trx8 (uninitialized): PHY [spi1.2-base-t1:02] driver [NXP C45 TJA1103] (irq=POLL) [ 17.481461] sja1105 spi1.2 trx9 (uninitialized): PHY [spi1.2-base-t1:03] driver [NXP C45 TJA1103] (irq=POLL) [ 17.615822] sja1105 spi1.2 trx10 (uninitialized): PHY [spi1.2-base-t1:04] driver [NXP C45 TJA1103] (irq=POLL) [ 17.750237] sja1105 spi1.2 trx11 (uninitialized): PHY [spi1.2-base-t1:05] driver [NXP C45 TJA1103] (irq=POLL) [ 17.884602] sja1105 spi1.2 trx12 (uninitialized): PHY [spi1.2-base-t1:06] driver [NXP C45 TJA1103] (irq=POLL) [ 17.895189] sja1105 spi1.0: Linked as a consumer to dpni.4 [ 17.900768] sja1105 spi1.2: Linked as a consumer to dpni.5 [ 17.906258] DSA: tree 0 setup [ 17.909265] sja1105 spi1.2: Dropping the link to 0x0000000008b96000:02 [ 17.916595] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpmcp.10 [ 17.956882] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.25 [ 17.965118] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.15 [ 17.971757] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.24 [ 17.979993] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.14 [ 17.986631] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.23 [ 17.994860] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.13 [ 18.001500] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.22 [ 18.009733] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.12 [ 18.016371] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.21 [ 18.024604] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.11 [ 18.031244] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.20 [ 18.039474] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.10 [ 18.046109] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.19 [ 18.054339] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.9 [ 18.060890] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.18 [ 18.069125] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.8 [ 18.075676] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.17 [ 18.083910] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.7 [ 18.090460] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.16 [ 18.098689] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.6 [ 18.105239] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.15 [ 18.113475] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.5 [ 18.120030] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.14 [ 18.128266] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.4 [ 18.134814] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.13 [ 18.143043] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.3 [ 18.149596] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.12 [ 18.157825] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.2 [ 18.164374] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.11 [ 18.172611] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.1 [ 18.179160] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.10 [ 18.187389] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.0 [ 18.194009] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpbp.1 [ 18.321818] fsl_dpaa2_eth dpni.1: Dropping the link to dpbp.1 [ 18.327621] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.0 [ 18.333400] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.10 [ 18.339351] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.1 [ 18.345133] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.11 [ 18.351081] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.2 [ 18.356856] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.12 [ 18.362806] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.3 [ 18.368583] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.13 [ 18.374530] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.4 [ 18.380304] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.14 [ 18.386254] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.5 [ 18.392032] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.15 [ 18.397979] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.6 [ 18.403755] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.16 [ 18.409704] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.7 [ 18.415479] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.17 [ 18.421427] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.8 [ 18.427203] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.18 [ 18.433153] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.9 [ 18.438927] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.19 [ 18.444876] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.10 [ 18.450738] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.20 [ 18.456689] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.11 [ 18.462549] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.21 [ 18.468497] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.12 [ 18.474359] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.22 [ 18.480309] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.13 [ 18.486171] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.23 [ 18.492118] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.14 [ 18.497980] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.24 [ 18.503930] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.15 [ 18.509798] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.25 [ 18.515749] fsl_dpaa2_eth dpni.1: Dropping the link to dpmcp.10 [ 18.522513] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpmcp.9 [ 18.563705] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.25 [ 18.571935] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.15 [ 18.578573] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.24 [ 18.586799] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.14 [ 18.593436] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.23 [ 18.601665] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.13 [ 18.608302] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.22 [ 18.616541] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.12 [ 18.623176] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.21 [ 18.631404] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.11 [ 18.638039] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.20 [ 18.646263] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.10 [ 18.652896] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.19 [ 18.661126] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.9 [ 18.667674] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.18 [ 18.675904] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.8 [ 18.682452] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.17 [ 18.690676] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.7 [ 18.697222] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.16 [ 18.705453] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.6 [ 18.712003] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.15 [ 18.720232] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.5 [ 18.726779] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.14 [ 18.735005] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.4 [ 18.741557] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.13 [ 18.749785] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.3 [ 18.756332] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.12 [ 18.765561] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.2 [ 18.772123] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.11 [ 18.780353] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.1 [ 18.786901] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpcon.10 [ 18.795128] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpio.0 [ 18.801733] fsl_dpaa2_eth dpni.1: Linked as a consumer to dpbp.1 [ 18.929911] fsl_dpaa2_eth dpni.1: Dropping the link to dpbp.1 [ 18.935705] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.0 [ 18.941486] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.10 [ 18.947434] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.1 [ 18.953213] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.11 [ 18.959161] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.2 [ 18.964934] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.12 [ 18.970882] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.3 [ 18.976655] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.13 [ 18.982603] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.4 [ 18.988377] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.14 [ 18.994324] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.5 [ 19.000098] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.15 [ 19.006046] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.6 [ 19.011820] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.16 [ 19.017767] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.7 [ 19.023540] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.17 [ 19.029488] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.8 [ 19.035261] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.18 [ 19.041210] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.9 [ 19.046984] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.19 [ 19.052931] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.10 [ 19.058790] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.20 [ 19.064738] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.11 [ 19.070598] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.21 [ 19.076545] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.12 [ 19.082405] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.22 [ 19.088353] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.13 [ 19.094212] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.23 [ 19.100160] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.14 [ 19.106020] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.24 [ 19.111968] fsl_dpaa2_eth dpni.1: Dropping the link to dpio.15 [ 19.117829] fsl_dpaa2_eth dpni.1: Dropping the link to dpcon.25 [ 19.123781] fsl_dpaa2_eth dpni.1: Dropping the link to dpmcp.9 [ 19.130149] ALSA device list: [ 19.133136] No soundcards found. [ 19.341696] EXT4-fs (mmcblk0p2): recovery complete [ 19.347355] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none. [ 19.357138] VFS: Mounted root (ext4 filesystem) on device 179:2. [ 19.363844] devtmpfs: mounted [ 19.369671] Freeing unused kernel memory: 9536K [ 19.374300] Run /sbin/init as init process [ 19.378395] with arguments: [ 19.381356] /sbin/init [ 19.384058] with environment: [ 19.387188] HOME=/ [ 19.389542] TERM=linux INIT: version booting Fuse filesystem already available. Mounting fuse control filesystem. Starting udev [ 19.720832] udevd[480]: starting version 3.2.9 [ 19.756177] udevd[481]: starting eudev-3.2.9 [ 19.902508] fsl_dpaa2_eth dpni.3 dpmac10: renamed from eth2 [ 19.908092] net eth2: renaming to dpmac10 [ 19.984284] fsl_dpaa2_eth dpni.4 dpmac17: renamed from eth1 [ 19.989997] net eth1: renaming to dpmac17 [ 20.020231] fsl_dpaa2_eth dpni.2 dpmac9: renamed from eth3 [ 20.025849] net eth3: renaming to dpmac9 [ 20.079989] fsl_dpaa2_eth dpni.0 dpmac5: renamed from eth4 [ 20.085493] net eth4: renaming to dpmac5 [ 20.127996] fsl_dpaa2_eth dpni.5 dpmac18: renamed from eth0 [ 20.133603] net eth0: renaming to dpmac18 [ 20.166859] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck. [ 20.206953] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: none. Starting tcf-agent: OK lx2160abluebox3 login: root root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/ total 0 drwxr-xr-x 2 root root 0 Jan 1 1970 . drwxr-xr-x 4 root root 0 Jan 1 1970 .. lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000:00 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:00 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000:01 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:01 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000:02 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:02 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000:08 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:08 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b97000:00 -> ../../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000/0x0000000008b97000:00 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b97000:08 -> ../../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000/0x0000000008b97000:08 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c17000:00 -> ../../../devices/platform/soc/8c17000.mdio/mdio_bus/0x0000000008c17000/0x0000000008c17000:00 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c27000:00 -> ../../../devices/platform/soc/8c27000.mdio/mdio_bus/0x0000000008c27000/0x0000000008c27000:00 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c2b000:00 -> ../../../devices/platform/soc/8c2b000.mdio/mdio_bus/0x0000000008c2b000/0x0000000008c2b000:00 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:01 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:01 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:02 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:02 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:03 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:03 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:04 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:04 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:05 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:05 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:06 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:06 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:01 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:01 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:02 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:02 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:03 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:03 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:04 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:04 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:05 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:05 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:06 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:06 root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/spi1.0-base-t1\:01 spi1.0-base-t1:01/ root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/spi1.0-base-t1\:01/ total 0 drwxr-xr-x 4 root root 0 Jan 1 1970 . drwxr-xr-x 10 root root 0 Jan 1 1970 .. lrwxrwxrwx 1 root root 0 Mar 9 12:35 driver -> ../../../../../../../../../../../../../../bus/mdio_bus/drivers/NXP C45 TJA1103 lrwxrwxrwx 1 root root 0 Mar 9 12:35 of_node -> ../../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio @0/ethernet-phy@1 -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_dev_flags -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_has_fixups -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_id -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_interface drwxr-xr-x 2 root root 0 Mar 9 12:35 power drwxr-xr-x 2 root root 0 Mar 9 12:35 statistics lrwxrwxrwx 1 root root 0 Mar 9 12:35 subsystem -> ../../../../../../../../../../../../../../bus/mdio_bus -rw-r--r-- 1 root root 4096 Jan 1 1970 uevent root@lx2160abluebox3:~# root@lx2160abluebox3:~# ls -la /sys/bus/spi/devices/spi1.0/ total 0 drwxr-xr-x 7 root root 0 Jan 1 1970 . drwxr-xr-x 6 root root 0 Jan 1 1970 .. lrwxrwxrwx 1 root root 0 Mar 9 12:36 driver -> ../../../../../../../../../../../bus/spi/drivers/sja1105 -rw-r--r-- 1 root root 4096 Mar 9 12:36 driver_override drwxr-xr-x 4 root root 0 Jan 1 1970 mdio_bus -r--r--r-- 1 root root 4096 Mar 9 12:36 modalias drwxr-xr-x 10 root root 0 Jan 1 1970 net lrwxrwxrwx 1 root root 0 Mar 9 12:36 of_node -> ../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0 drwxr-xr-x 2 root root 0 Mar 9 12:36 power drwxr-xr-x 3 root root 0 Jan 1 1970 ptp drwxr-xr-x 2 root root 0 Mar 9 12:36 statistics lrwxrwxrwx 1 root root 0 Jan 1 1970 subsystem -> ../../../../../../../../../../../bus/spi lrwxrwxrwx 1 root root 0 Mar 9 12:36 supplier:fsl-mc:dpni.4 -> ../../../../../../../../../../virtual/devlink/fsl-mc:dpni.4--spi:spi1.0 -rw-r--r-- 1 root root 4096 Jan 1 1970 uevent INIT: Id "S0" respawning too fast: disabled for 5 minutes c lass/mdio_bus/ total 0 drwxr-xr-x 2 root root 0 Jan 1 1970 . drwxr-xr-x 70 root root 0 Jan 1 1970 .. lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000 -> ../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b97000 -> ../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c17000 -> ../../devices/platform/soc/8c17000.mdio/mdio_bus/0x0000000008c17000 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c27000 -> ../../devices/platform/soc/8c27000.mdio/mdio_bus/0x0000000008c27000 lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c2b000 -> ../../devices/platform/soc/8c2b000.mdio/mdio_bus/0x0000000008c2b000 lrwxrwxrwx 1 root root 0 Jan 1 1970 fixed-0 -> ../../devices/platform/Fixed MDIO bus.0/mdio_bus/fixed-0 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1 -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-pcs -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-pcs lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1 -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1 lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-pcs -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-pcs root@lx2160abluebox3:~# ls -la /sys/class/mdio_bus/spi1.0-base-t1/ total 0 drwxr-xr-x 10 root root 0 Jan 1 1970 . drwxr-xr-x 4 root root 0 Jan 1 1970 .. lrwxrwxrwx 1 root root 0 Mar 9 12:37 device -> ../../../spi1.0 lrwxrwxrwx 1 root root 0 Mar 9 12:37 of_node -> ../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio@0 drwxr-xr-x 2 root root 0 Mar 9 12:37 power drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:01 drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:02 drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:03 drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:04 drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:05 drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:06 drwxr-xr-x 2 root root 0 Mar 9 12:37 statistics lrwxrwxrwx 1 root root 0 Jan 1 1970 subsystem -> ../../../../../../../../../../../../../class/mdio_bus -rw-r--r-- 1 root root 4096 Jan 1 1970 uevent -r--r--r-- 1 root root 4096 Mar 9 12:37 waiting_for_supplier root@lx2160abluebox3:~# ls -la /sys/class/mdio_bus/spi1.0-base-t1/spi1.0-base-t1\:01/ total 0 drwxr-xr-x 4 root root 0 Jan 1 1970 . drwxr-xr-x 10 root root 0 Jan 1 1970 .. lrwxrwxrwx 1 root root 0 Mar 9 12:35 driver -> ../../../../../../../../../../../../../../bus/mdio_bus/drivers/NXP C45 TJA1103 lrwxrwxrwx 1 root root 0 Mar 9 12:35 of_node -> ../../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio @0/ethernet-phy@1 -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_dev_flags -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_has_fixups -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_id -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_interface drwxr-xr-x 2 root root 0 Mar 9 12:35 power drwxr-xr-x 2 root root 0 Mar 9 12:35 statistics lrwxrwxrwx 1 root root 0 Mar 9 12:35 subsystem -> ../../../../../../../../../../../../../../bus/mdio_bus -rw-r--r-- 1 root root 4096 Jan 1 1970 uevent As you can see, there was no provider/consumer relationship enforced between the PHY at spi1.0-base-t1\:01 and the ethernet-switch. > > > > > > If it doesn't fix it, can one of you please point me to an upstream > > > dts (not dtsi) file for a platform in which you see this issue? And > > > ideally also the DT nodes and their drivers that are involved in this > > > cycle? With that info, I should be able to root cause this if the > > > patch above doesn't already fix it. > > > > I'm working with a non-upstream dts - maybe Vladimir is using an > > upstream one? The pattern among the drivers/dts is common between our > > two cases. > > Ideally, I can get a fully upstream example where this issue is > happening so that I can look at the actual code that's hitting this > issue and be sure my analysis is right. Mine isn't upstream either, but I have no issue sharing it with you. cat arch/arm64/boot/dts/freescale/fsl-lx2160a-bluebox3.dts // SPDX-License-Identifier: (GPL-2.0 OR MIT) // // Device Tree file for LX2160A BLUEBOX3 // // Copyright 2020 NXP /dts-v1/; #include "fsl-lx2160a.dtsi" / { model = "NXP Layerscape LX2160ABLUEBOX3"; compatible = "fsl,lx2160a-rdb", "fsl,lx2160a"; aliases { crypto = &crypto; serial0 = &uart0; mmc0 = &esdhc0; mmc1 = &esdhc1; }; chosen { stdout-path = "serial0:115200n8"; }; sb_3v3: regulator-sb3v3 { compatible = "regulator-fixed"; regulator-name = "MC34717-3.3VSB"; regulator-min-microvolt = <3300000>; regulator-max-microvolt = <3300000>; regulator-boot-on; regulator-always-on; }; }; &can0 { status = "okay"; can-transceiver { max-bitrate = <5000000>; }; }; &can1 { status = "okay"; can-transceiver { max-bitrate = <5000000>; }; }; &crypto { status = "okay"; }; &emdio1 { status = "okay"; }; &emdio2 { status = "okay"; }; &esdhc0 { sd-uhs-sdr104; sd-uhs-sdr50; sd-uhs-sdr25; sd-uhs-sdr12; status = "okay"; }; &esdhc1 { mmc-hs200-1_8v; mmc-hs400-1_8v; bus-width = <8>; status = "okay"; }; &fspi { status = "okay"; mt35xu512aba0: flash@0 { #address-cells = <1>; #size-cells = <1>; compatible = "jedec,spi-nor"; m25p,fast-read; spi-max-frequency = <50000000>; reg = <0>; spi-rx-bus-width = <8>; spi-tx-bus-width = <8>; }; mt35xu512aba1: flash@1 { #address-cells = <1>; #size-cells = <1>; compatible = "jedec,spi-nor"; m25p,fast-read; spi-max-frequency = <50000000>; reg = <1>; spi-rx-bus-width = <8>; spi-tx-bus-width = <8>; }; }; &i2c0 { status = "okay"; i2c-mux@77 { compatible = "nxp,pca9547"; reg = <0x77>; #address-cells = <1>; #size-cells = <0>; i2c@1 { #address-cells = <1>; #size-cells = <0>; reg = <0x1>; // TODO: Add entries for EMC2305 }; i2c@2 { #address-cells = <1>; #size-cells = <0>; reg = <0x2>; power-monitor@40 { compatible = "ti,ina220"; reg = <0x40>; shunt-resistor = <500>; }; }; i2c@3 { #address-cells = <1>; #size-cells = <0>; reg = <0x3>; temperature-sensor@4c { compatible = "nxp,sa56004"; reg = <0x4c>; vcc-supply = <&sb_3v3>; }; temperature-sensor@48 { compatible = "nxp,sa56004"; reg = <0x48>; vcc-supply = <&sb_3v3>; }; }; i2c@4 { #address-cells = <1>; #size-cells = <0>; reg = <0x4>; rtc@51 { compatible = "nxp,pcf2129"; reg = <0x51>; // IRQ10_B interrupts = <0 150 0x4>; }; }; i2c@7 { #address-cells = <1>; #size-cells = <0>; reg = <0x7>; i2c-mux@75 { compatible = "nxp,pca9547"; reg = <0x75>; #address-cells = <1>; #size-cells = <0>; i2c@0 { #address-cells = <1>; #size-cells = <0>; reg = <0x0>; spi_bridge: spi@28 { compatible = "nxp,sc18is602b"; reg = <0x28>; #address-cells = <1>; #size-cells = <0>; }; }; }; }; }; }; &i2c5 { status = "okay"; i2c-mux@77 { compatible = "nxp,pca9846"; reg = <0x77>; #address-cells = <1>; #size-cells = <0>; // TODO: Add entrier for device connected to mux }; }; &spi_bridge { /* SW1 */ ethernet-switch@0 { compatible = "nxp,sja1110a"; reg = <0>; spi-max-frequency = <4000000>; spi-cpol; dsa,member = <0 0>; ethernet-ports { #address-cells = <1>; #size-cells = <0>; /* Microcontroller port */ port@0 { reg = <0>; status = "disabled"; }; /* SW1_P1 */ port@1 { reg = <1>; label = "con_2x20"; phy-mode = "sgmii"; fixed-link { speed = <1000>; full-duplex; }; }; port@2 { reg = <2>; ethernet = <&dpmac17>; phy-mode = "rgmii-id"; fixed-link { speed = <1000>; full-duplex; }; }; port@3 { reg = <3>; label = "1ge_p1"; phy-mode = "rgmii-id"; phy-handle = <&sw1_mii3_phy>; }; sw1p4: port@4 { reg = <4>; link = <&sw2p1>; phy-mode = "sgmii"; fixed-link { speed = <1000>; full-duplex; }; }; port@5 { reg = <5>; label = "trx1"; phy-mode = "internal"; phy-handle = <&sw1_port5_base_t1_phy>; }; port@6 { reg = <6>; label = "trx2"; phy-mode = "internal"; phy-handle = <&sw1_port6_base_t1_phy>; }; port@7 { reg = <7>; label = "trx3"; phy-mode = "internal"; phy-handle = <&sw1_port7_base_t1_phy>; }; port@8 { reg = <8>; label = "trx4"; phy-mode = "internal"; phy-handle = <&sw1_port8_base_t1_phy>; }; port@9 { reg = <9>; label = "trx5"; phy-mode = "internal"; phy-handle = <&sw1_port9_base_t1_phy>; }; port@a { reg = <10>; label = "trx6"; phy-mode = "internal"; phy-handle = <&sw1_port10_base_t1_phy>; }; }; mdios { #address-cells = <1>; #size-cells = <0>; mdio@0 { reg = <0>; compatible = "nxp,sja1110-base-t1-mdio"; #address-cells = <1>; #size-cells = <0>; sw1_port5_base_t1_phy: ethernet-phy@1 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x1>; }; sw1_port6_base_t1_phy: ethernet-phy@2 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x2>; }; sw1_port7_base_t1_phy: ethernet-phy@3 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x3>; }; sw1_port8_base_t1_phy: ethernet-phy@4 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x4>; }; sw1_port9_base_t1_phy: ethernet-phy@5 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x5>; }; sw1_port10_base_t1_phy: ethernet-phy@6 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x6>; }; }; }; }; /* SW2 */ ethernet-switch@2 { compatible = "nxp,sja1110a"; reg = <2>; spi-max-frequency = <4000000>; spi-cpol; dsa,member = <0 1>; ethernet-ports { #address-cells = <1>; #size-cells = <0>; /* Microcontroller port */ port@0 { reg = <0>; status = "disabled"; }; sw2p1: port@1 { reg = <1>; link = <&sw1p4>; phy-mode = "sgmii"; fixed-link { speed = <1000>; full-duplex; }; }; port@2 { reg = <2>; ethernet = <&dpmac18>; phy-mode = "rgmii-id"; fixed-link { speed = <1000>; full-duplex; }; }; port@3 { reg = <3>; label = "1ge_p2"; phy-mode = "rgmii-id"; phy-handle = <&sw2_mii3_phy>; }; port@4 { reg = <4>; label = "to_sw3"; phy-mode = "2500base-x"; fixed-link { speed = <2500>; full-duplex; }; }; port@5 { reg = <5>; label = "trx7"; phy-mode = "internal"; phy-handle = <&sw2_port5_base_t1_phy>; }; port@6 { reg = <6>; label = "trx8"; phy-mode = "internal"; phy-handle = <&sw2_port6_base_t1_phy>; }; port@7 { reg = <7>; label = "trx9"; phy-mode = "internal"; phy-handle = <&sw2_port7_base_t1_phy>; }; port@8 { reg = <8>; label = "trx10"; phy-mode = "internal"; phy-handle = <&sw2_port8_base_t1_phy>; }; port@9 { reg = <9>; label = "trx11"; phy-mode = "internal"; phy-handle = <&sw2_port9_base_t1_phy>; }; port@a { reg = <10>; label = "trx12"; phy-mode = "internal"; phy-handle = <&sw2_port10_base_t1_phy>; }; }; mdios { #address-cells = <1>; #size-cells = <0>; mdio@0 { reg = <0>; compatible = "nxp,sja1110-base-t1-mdio"; #address-cells = <1>; #size-cells = <0>; sw2_port5_base_t1_phy: ethernet-phy@1 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x1>; }; sw2_port6_base_t1_phy: ethernet-phy@2 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x2>; }; sw2_port7_base_t1_phy: ethernet-phy@3 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x3>; }; sw2_port8_base_t1_phy: ethernet-phy@4 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x4>; }; sw2_port9_base_t1_phy: ethernet-phy@5 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x5>; }; sw2_port10_base_t1_phy: ethernet-phy@6 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x6>; }; }; }; }; }; &uart0 { status = "okay"; }; &uart1 { status = "okay"; }; &usb0 { status = "okay"; }; &emdio1 { aquantia_phy1: ethernet-phy@0 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x0>; }; aquantia_phy2: ethernet-phy@8 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x8>; }; sw1_mii3_phy: ethernet-phy@1 { /* AR8035 */ compatible = "ethernet-phy-id004d.d072"; reg = <0x5>; }; sw2_mii3_phy: ethernet-phy@2 { /* AR8035 */ compatible = "ethernet-phy-id004d.d072"; reg = <0x6>; }; }; &emdio2 { aquantia_phy3: ethernet-phy@0 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x0>; }; aquantia_phy4: ethernet-phy@8 { compatible = "ethernet-phy-ieee802.3-c45"; reg = <0x8>; }; }; &dpmac5 { phy-handle = <&aquantia_phy2>; phy-connection-type = "usxgmii"; managed = "in-band-status"; }; &dpmac6 { phy-handle = <&aquantia_phy1>; phy-connection-type = "usxgmii"; managed = "in-band-status"; }; &dpmac9 { phy-handle = <&aquantia_phy4>; phy-connection-type = "usxgmii"; managed = "in-band-status"; }; &dpmac10 { phy-handle = <&aquantia_phy3>; phy-connection-type = "usxgmii"; managed = "in-band-status"; }; &dpmac17 { phy-connection-type = "rgmii"; status = "okay"; fixed-link { speed = <1000>; full-duplex; }; }; &dpmac18 { phy-connection-type = "rgmii"; status = "okay"; fixed-link { speed = <1000>; full-duplex; }; }; &pcs_mdio5 { status = "okay"; }; &pcs_mdio5 { status = "okay"; }; &pcs_mdio9 { status = "okay"; }; &pcs_mdio10 { status = "okay"; }; > 2. The child devices could defer probe because one of their suppliers > isn't ready yet. Either because of fw_devlink=on or the framework > itself returning -EPROBE_DEFER. > 3. The child devices could be getting probed asynchronously. So the > device_add() would kick off a thread to probe the child devices in a > separate thread. > > (2) is what is happening in this case. fw_devlink=on sees that > "switch" implements the "switch_intc" and "switch" hasn't finished > probing yet. So it has no way of knowing that switch_intc is actually > ready. And even if switch_intc was registered as part of switch's > probe() by the time the PHYs are added, switch_intc could get > deregistered if the probe fails at a later point. So until probe() > returns 0, fw_devlink can't be fully sure the supplier (switch_intc) > is ready. Which is good in general because you won't have to > forcefully unbind (if that is even handled correctly in the first > place) the consumers of a device if it fails probe() half way through > registering a few services. I think you're right and this is the difference between my setup and Alvin's, I don't have any sort of interrupts for the internal PHYs.
> (2) is what is happening in this case. fw_devlink=on sees that > "switch" implements the "switch_intc" and "switch" hasn't finished > probing yet. So it has no way of knowing that switch_intc is actually > ready. And even if switch_intc was registered as part of switch's > probe() by the time the PHYs are added, switch_intc could get > deregistered if the probe fails at a later point. So until probe() > returns 0, fw_devlink can't be fully sure the supplier (switch_intc) > is ready. Which is good in general because you won't have to > forcefully unbind (if that is even handled correctly in the first > place) the consumers of a device if it fails probe() half way through > registering a few services. There are actually a few different circular references with the way switches work. Take for example: &fec1 { phy-mode = "rmii"; pinctrl-names = "default"; pinctrl-0 = <&pinctrl_fec1>; status = "okay"; fixed-link { speed = <100>; full-duplex; }; mdio1: mdio { #address-cells = <1>; #size-cells = <0>; clock-frequency = <12500000>; suppress-preamble; status = "okay"; switch0: switch0@0 { compatible = "marvell,mv88e6190"; pinctrl-0 = <&pinctrl_gpio_switch0>; pinctrl-names = "default"; reg = <0>; eeprom-length = <65536>; interrupt-parent = <&gpio3>; interrupts = <2 IRQ_TYPE_LEVEL_LOW>; interrupt-controller; #interrupt-cells = <2>; ports { #address-cells = <1>; #size-cells = <0>; port@0 { reg = <0>; label = "cpu"; ethernet = <&fec1>; fixed-link { speed = <100>; full-duplex; }; }; FEC is an ethernet controller. It has an MDIO bus, and on the bus is an Ethernet switch. port 0 of the Ethernet switch is connected to the FEC ethernet controller. While the FEC probes, it will at some point register its MDIO bus. At that point, the MDIO bus is probed, the switch is found, and registered with the switch core. The switch core looks for the port with an ethernet property and goes looking for that ethernet interface. But that this point in time, the FEC probe has only got as far as registering the MDIO bus. The interface itself is not registered. So finding the interface fails, and we go into EPROBE_DEFER for probing the switch. It is pretty hard to solve. An Ethernet interface can be used by the kernel itself, e.g. NFS root. At the point you call register_netdev() in the probe function, to register the interface with the core, it needs to be fully ready to go. The networking stack can start using the interface before register_netdev() even returns. So you cannot first register the interface and then register the MDIO bus. I once looked to see if it was possible to tell the driver core to not even bother probing a bus as soon as it is registered, go straight to defer probe handling. Because this is one case we know it cannot work. But it does not seem possible. Andrew
Hi Saravana, Thanks for your lucid analysis, it's very much appreciated. Please find my replies inline - but in summary, I don't think there is anything more I can ask of you right now. On 8/19/21 5:28 AM, Saravana Kannan wrote: > On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: >> >> Hi Saravana, >> >> On 8/18/21 4:46 AM, Saravana Kannan wrote: >>> On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote: >>>> >>>> Hi Alvin, >>>> >>>> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote: >>>>> I have an observation that's slightly out of the scope of your patch, >>>>> but I'll post here on the off chance that you find it relevant. >>>>> Apologies if it's out of place. >>>>> >>>>> Do these integrated NXP PHYs use a specific PHY driver, or do they just >>>>> use the Generic PHY driver? >>>> >>>> They refuse to probe at all with the Generic PHY driver. I have been >>>> caught off guard a few times now when I had a kernel built with >>>> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case. >>>> >>>>> If the former is the case, do you experience that the PHY driver fails >>>>> to get probed during mdiobus registration if the kernel uses >>>>> fw_devlink=on? >>>> >>>> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first >>>> time I do it. It behaves exactly as you say. >>>> >>>>> >>>>> In my case I am writing a new subdriver for realtek-smi, a DSA driver >>>>> which registers an internal MDIO bus analogously to sja1105, which is >>>>> why I'm asking. I noticed a deferred probe of the PHY driver because the >>>>> supplier (ethernet-switch) is not ready - presumably because all of this >>>>> is happening in the probe of the switch driver. See below: >>>>> >>>>> [ 83.653213] device_add:3270: device: 'SMI-0': device_add >>>>> [ 83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0 >>>>> [ 83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add >>>>> [ 83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch >>>>> [ 83.654291] libphy: SMI slave MII: probed >>>>> ... >>>>> [ 83.659809] device_add:3270: device: 'SMI-0:00': device_add >>>>> [ 83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00 >>>>> [ 83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00 >>>>> [ 83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add >>>>> [ 83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list >>>>> [ 83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list >>>>> [ 83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch >>>>> [ 83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet >>>>> [ 83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready >>>>> [ 83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list >>>> >>>> So it's a circular dependency? Switch cannot finish probing because it >>>> cannot connect to PHY, which cannot probe because switch has not >>>> finished probing, which.... >>> >>> Hi Vladimir/Alvin, >>> >>> If there's a cyclic dependency between two devices, then fw_devlink=on >>> is smart enough to notice that. Once it notices a cycle, it knows that >>> it can't tell which one is the real dependency and which one is the >>> false dependency and so stops enforcing ordering between the devices >>> in the cycle. >>> >>> But fw_devlink doesn't understand all the properties yet. Just most of >>> them and I'm always trying to add more. So when it only understands >>> the property that's causing the false dependency but not the property >>> that causes the real dependency, it can cause issues like this where >>> fw_devlink=on enforces the false dependency and the driver/code >>> enforces the real dependency. These are generally easy to fix -- you >>> just need to teach fw_devlink how to parse more properties. >>> >>> This is just a preliminary analysis since I don't have all the info >>> yet -- so I could be wrong. With that said, I happened to be working >>> on adding fw_devlink support for phy-handle property and I think it >>> should fix your issue with fw_devlink=on. Can you give [1] a shot? >> >> I tried [1] but it did not seem to have any effect. >> >>> >>> If it doesn't fix it, can one of you please point me to an upstream >>> dts (not dtsi) file for a platform in which you see this issue? And >>> ideally also the DT nodes and their drivers that are involved in this >>> cycle? With that info, I should be able to root cause this if the >>> patch above doesn't already fix it. >> >> I'm working with a non-upstream dts - maybe Vladimir is using an >> upstream one? The pattern among the drivers/dts is common between our >> two cases. > > Ideally, I can get a fully upstream example where this issue is > happening so that I can look at the actual code that's hitting this > issue and be sure my analysis is right. Due to NDA issues at work, I am unable to publish the DTS right now. But based on your analysis below, I believe that the problem I am experiencing is exactly for the reasons you describe. > >> >> But for the sake of this discussion, my dts is pretty much the same as >> what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The >> nodes of interest from that dts file are below, and the driver is in >> drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the >> Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part >> of the mdiobus registration, but that never happens. See my previous >> reply for a debug log. > > Your DTS might be similar to this, but the driver code also matters > for me to be sure. Anyway, I took a look at this, but my analysis > below is going to be sketchy because I'm not looking at the actual > code that's reproducing this issue. The driver code I am working with is pretty much the same in the probe path (the driver is modelled on rtl8366rb.c and hooks into realtek-smi-core.c), so it was not a waste of your time to analyze the upstream case. Thanks a lot. > > Assuming this issue actually happens with the example you pointed to > (I don't know this yet), here's what is happening: > > The main problem is that the parent device switch seems to be assuming > it's child/grandchild devices (mdiobus/PHYs) will have probed > successfully as soon as they are added. This assumption is not true > and can be broken for multiple reasons such as: > > 1. The driver for the child devices (PHYs in this case) could be > loaded as a module after the parent (switch) is probed. So when the > devices are added, the PHYs would not be probed. > 2. The child devices could defer probe because one of their suppliers > isn't ready yet. Either because of fw_devlink=on or the framework > itself returning -EPROBE_DEFER. > 3. The child devices could be getting probed asynchronously. So the > device_add() would kick off a thread to probe the child devices in a > separate thread. > > (2) is what is happening in this case. fw_devlink=on sees that > "switch" implements the "switch_intc" and "switch" hasn't finished > probing yet. So it has no way of knowing that switch_intc is actually > ready. And even if switch_intc was registered as part of switch's > probe() by the time the PHYs are added, switch_intc could get > deregistered if the probe fails at a later point. So until probe() > returns 0, fw_devlink can't be fully sure the supplier (switch_intc) > is ready. Which is good in general because you won't have to > forcefully unbind (if that is even handled correctly in the first > place) the consumers of a device if it fails probe() half way through > registering a few services. Right, that makes perfect sense. > > I don't fully understand the networking frameworks, but I think > Vladimir might have a point in his earlier reply [1]. If you can make > the switch driver not assume its child PHYs are ready during the > switch's probe and instead have the switch check if the PHYs are ready > when the switch is "opened" that'd be better. I am pretty new to the DSA subsystem, but it appears to me that DSA is making this assumption: I don't think DSA switches have a concept of "open", as everything is handled in the dsa_register_switch() call in a DSA driver's probe function. And as Vladimir pointed out, this is also where phylink_of_phy_connect() is being called. A quick look at the other DSA drivers suggests that mv88e6xxx may also suffer from case (2) above, so it seems like a more general issue. > > We can come up with hacks that'll delete the dependency that > fw_devlink=on is trying to enforce, but IMHO the proper fix is to have > parent drivers not assume child devices will be probed as soon as > device_add(child) returns. That's not guaranteed at all. You enumerated a number of reasons why, so yes, I agree that it is not necessarily a safe assumption to make. > > Btw, I do know why things work when you do the module load/unload > thing you mention in [2]. That has to do with some forced deletion of > dependencies that happens when device_bind_driver() is called when the > Generic PHY driver is used. The reason for why that's done is kind of > unrelated to the issue at hand, but the comment for > device_links_force_bind() should tell you why. Thanks for clearing that up. Kind regards, Alvin
Hi Vladimir, On 8/19/21 1:22 PM, Vladimir Oltean wrote: > > Sorry for the delay, I wanted to reconfirm what I said (hint, I was wrong). Thanks for the clarification. It also lines up with Saravana's analysis for my case. > > In my case, whatever I do, I cannot get the driver core enforce a device > link between the ethernet-switch and the PHY. > > So I cannot actually see the same issue. What I was seeing was in fact > stupid testing on my part (it was working with the PHY driver as > built-in, it was working, then I made it a module, it broke, I forgot to > switch it back to module, then I thought it's broken while the PHY is > built-in). Do you mean to say that you are hitting the "case (1)" that Saravana described? See below: On 8/19/21 5:28 AM, Saravana Kannan wrote: > The main problem is that the parent device switch seems to be assuming > it's child/grandchild devices (mdiobus/PHYs) will have probed > successfully as soon as they are added. This assumption is not true > and can be broken for multiple reasons such as: > > 1. The driver for the child devices (PHYs in this case) could be > loaded as a module after the parent (switch) is probed. So when the > devices are added, the PHYs would not be probed. > 2. The child devices could defer probe because one of their suppliers > isn't ready yet. Either because of fw_devlink=on or the framework > itself returning -EPROBE_DEFER. > 3. The child devices could be getting probed asynchronously. So the > device_add() would kick off a thread to probe the child devices in a > separate thread. I would think that - in general - it should not matter if the PHY driver is built as a module. Kind regards, Alvin
On Thu, Aug 19, 2021 at 6:35 AM Andrew Lunn <andrew@lunn.ch> wrote: > > > (2) is what is happening in this case. fw_devlink=on sees that > > "switch" implements the "switch_intc" and "switch" hasn't finished > > probing yet. So it has no way of knowing that switch_intc is actually > > ready. And even if switch_intc was registered as part of switch's > > probe() by the time the PHYs are added, switch_intc could get > > deregistered if the probe fails at a later point. So until probe() > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc) > > is ready. Which is good in general because you won't have to > > forcefully unbind (if that is even handled correctly in the first > > place) the consumers of a device if it fails probe() half way through > > registering a few services. I had to read your email a couple of times before I understood it. I think I do now, but apologies if I'm not making sense. > > There are actually a few different circular references with the way > switches work. Take for example: > > &fec1 { > phy-mode = "rmii"; > pinctrl-names = "default"; > pinctrl-0 = <&pinctrl_fec1>; > status = "okay"; > > fixed-link { > speed = <100>; > full-duplex; > }; > > mdio1: mdio { > #address-cells = <1>; > #size-cells = <0>; > clock-frequency = <12500000>; > suppress-preamble; > status = "okay"; > > switch0: switch0@0 { > compatible = "marvell,mv88e6190"; > pinctrl-0 = <&pinctrl_gpio_switch0>; > pinctrl-names = "default"; > reg = <0>; > eeprom-length = <65536>; > interrupt-parent = <&gpio3>; > interrupts = <2 IRQ_TYPE_LEVEL_LOW>; > interrupt-controller; > #interrupt-cells = <2>; > > ports { > #address-cells = <1>; > #size-cells = <0>; > > port@0 { > reg = <0>; > label = "cpu"; > ethernet = <&fec1>; > > fixed-link { > speed = <100>; > full-duplex; > }; > }; > > FEC is an ethernet controller. It has an MDIO bus, and on the bus is > an Ethernet switch. port 0 of the Ethernet switch is connected to the > FEC ethernet controller. > > While the FEC probes, it will at some point register its MDIO bus. At > that point, the MDIO bus is probed, the switch is found, and > registered with the switch core. The switch core looks for the port > with an ethernet property and goes looking for that ethernet > interface. But that this point in time, the FEC probe has only got as > far as registering the MDIO bus. The interface itself is not > registered. So finding the interface fails, and we go into > EPROBE_DEFER for probing the switch. Ok, I understood up to here. Couple of questions: Is this EPROBE_DEFER causing an issue? Wouldn't the switch then probe successfully when it's reattempted? And then things work normally? I don't see what the problem is. > It is pretty hard to solve. An Ethernet interface can be used by the > kernel itself, e.g. NFS root. At the point you call register_netdev() > in the probe function, to register the interface with the core, Are you using "ethernet interface" and "ethernet controller" interchangeably? Looking at some other drivers, it looks like the ethernet controlled (FEC) is what would call register_netdev(). So what's wrong with that happening if switch0 has not probed successfully? > it > needs to be fully ready to go. The networking stack can start using > the interface before register_netdev() even returns. So you cannot > first register the interface and then register the MDIO bus. > > I once looked to see if it was possible to tell the driver core to not > even bother probing a bus as soon as it is registered, go straight to > defer probe handling. Because this is one case we know it cannot > work. But it does not seem possible. fw_devlink doesn't understand the "ethernet" property. If I add that, then in the example you state above, switch0's probe won't even be called until the FEC probe returns. The change is pretty trivial (pasted below) -- can you try it out and tell me if it does what you need/want? -Saravana +++ b/drivers/of/property.c @@ -1292,6 +1292,7 @@ DEFINE_SIMPLE_PROP(resets, "resets", "#reset-cells") DEFINE_SIMPLE_PROP(leds, "leds", NULL) DEFINE_SIMPLE_PROP(backlight, "backlight", NULL) DEFINE_SIMPLE_PROP(phy_handle, "phy-handle", NULL) +DEFINE_SIMPLE_PROP(ethernet, "ethernet", NULL) DEFINE_SUFFIX_PROP(regulators, "-supply", NULL) DEFINE_SUFFIX_PROP(gpio, "-gpio", "#gpio-cells") @@ -1381,6 +1382,7 @@ static const struct supplier_bindings of_supplier_bindings[] = { { .parse_prop = parse_leds, }, { .parse_prop = parse_backlight, }, { .parse_prop = parse_phy_handle, }, + { .parse_prop = parse_ethernet, }, { .parse_prop = parse_gpio_compat, }, { .parse_prop = parse_interrupts, }, { .parse_prop = parse_regulators, },
On Thu, Aug 19, 2021 at 04:52:43PM -0700, Saravana Kannan wrote: > On Thu, Aug 19, 2021 at 6:35 AM Andrew Lunn <andrew@lunn.ch> wrote: > > > > > (2) is what is happening in this case. fw_devlink=on sees that > > > "switch" implements the "switch_intc" and "switch" hasn't finished > > > probing yet. So it has no way of knowing that switch_intc is actually > > > ready. And even if switch_intc was registered as part of switch's > > > probe() by the time the PHYs are added, switch_intc could get > > > deregistered if the probe fails at a later point. So until probe() > > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc) > > > is ready. Which is good in general because you won't have to > > > forcefully unbind (if that is even handled correctly in the first > > > place) the consumers of a device if it fails probe() half way through > > > registering a few services. > > I had to read your email a couple of times before I understood it. I > think I do now, but apologies if I'm not making sense. > > > > > There are actually a few different circular references with the way > > switches work. Take for example: > > > > &fec1 { > > phy-mode = "rmii"; > > pinctrl-names = "default"; > > pinctrl-0 = <&pinctrl_fec1>; > > status = "okay"; > > > > fixed-link { > > speed = <100>; > > full-duplex; > > }; > > > > mdio1: mdio { > > #address-cells = <1>; > > #size-cells = <0>; > > clock-frequency = <12500000>; > > suppress-preamble; > > status = "okay"; > > > > switch0: switch0@0 { > > compatible = "marvell,mv88e6190"; > > pinctrl-0 = <&pinctrl_gpio_switch0>; > > pinctrl-names = "default"; > > reg = <0>; > > eeprom-length = <65536>; > > interrupt-parent = <&gpio3>; > > interrupts = <2 IRQ_TYPE_LEVEL_LOW>; > > interrupt-controller; > > #interrupt-cells = <2>; > > > > ports { > > #address-cells = <1>; > > #size-cells = <0>; > > > > port@0 { > > reg = <0>; > > label = "cpu"; > > ethernet = <&fec1>; > > > > fixed-link { > > speed = <100>; > > full-duplex; > > }; > > }; > > > > FEC is an ethernet controller. It has an MDIO bus, and on the bus is > > an Ethernet switch. port 0 of the Ethernet switch is connected to the > > FEC ethernet controller. > > > > While the FEC probes, it will at some point register its MDIO bus. At > > that point, the MDIO bus is probed, the switch is found, and > > registered with the switch core. The switch core looks for the port > > with an ethernet property and goes looking for that ethernet > > interface. But that this point in time, the FEC probe has only got as > > far as registering the MDIO bus. The interface itself is not > > registered. So finding the interface fails, and we go into > > EPROBE_DEFER for probing the switch. > > Ok, I understood up to here. Couple of questions: > Is this EPROBE_DEFER causing an issue? Wouldn't the switch then > probe successfully when it's reattempted? And then things work > normally? I don't see what the problem is. It's not an issue per se, since it's not a fully circular dependency: the DSA master (the FEC controller) does not have any dependency on the switch beneath it to probe (there's nothing like a phy-handle from the FEC to the switch or to something provided by it). A few EPROBE_DEFER iterations later the switch will finally find its DSA master fully probed via of_find_net_device_by_node. Andrew is wondering how to avoid those extra EPROBE_DEFER iterations. It is weird that the entire functionality of the system depends on those EPROBE_DEFERs, typically you'd expect that EPROBE_DEFER just serializes asynchronous probing of drivers with interdependencies. But in this case it serializes synchronous probing. > > It is pretty hard to solve. An Ethernet interface can be used by the > > kernel itself, e.g. NFS root. At the point you call register_netdev() > > in the probe function, to register the interface with the core, > > Are you using "ethernet interface" and "ethernet controller" > interchangeably? Looking at some other drivers, it looks like the > ethernet controlled (FEC) is what would call register_netdev(). So > what's wrong with that happening if switch0 has not probed > successfully? The "interface" and "controller" terms are not really interchangeable, an interface can also be virtual (stacked interfaces on top of physical ones, like VLAN or DSA) while network controllers are typically physical (unless emulated). But that is not of importance. The context here is that you cannot solve the interdependency by registering the DSA master (FEC) first, then its MDIO bus second (the DSA switch probes on the DSA master's MDIO bus => if you do this, of_find_net_device_by_node from the DSA layer would find its master the first time). The reason you cannot do that is because you need the MDIO bus for really basic stuff: you also have your Ethernet PHY on it, and you need to initialize that in order to send traffic. And you need to be able to send traffic as soon as register_netdev() completes. So since the driver initialization sequence has a single written order regardless of whether DSA switches are attached or not, that order is picked to be the one where traffic works as soon as register_netdev completes. > > it > > needs to be fully ready to go. The networking stack can start using > > the interface before register_netdev() even returns. So you cannot > > first register the interface and then register the MDIO bus. > > > > I once looked to see if it was possible to tell the driver core to not > > even bother probing a bus as soon as it is registered, go straight to > > defer probe handling. Because this is one case we know it cannot > > work. But it does not seem possible. > > fw_devlink doesn't understand the "ethernet" property. If I add that, > then in the example you state above, switch0's probe won't even be > called until the FEC probe returns. The change is pretty trivial > (pasted below) -- can you try it out and tell me if it does what you > need/want? > > -Saravana > > +++ b/drivers/of/property.c > @@ -1292,6 +1292,7 @@ DEFINE_SIMPLE_PROP(resets, "resets", "#reset-cells") > DEFINE_SIMPLE_PROP(leds, "leds", NULL) > DEFINE_SIMPLE_PROP(backlight, "backlight", NULL) > DEFINE_SIMPLE_PROP(phy_handle, "phy-handle", NULL) > +DEFINE_SIMPLE_PROP(ethernet, "ethernet", NULL) > DEFINE_SUFFIX_PROP(regulators, "-supply", NULL) > DEFINE_SUFFIX_PROP(gpio, "-gpio", "#gpio-cells") > > @@ -1381,6 +1382,7 @@ static const struct supplier_bindings > of_supplier_bindings[] = { > { .parse_prop = parse_leds, }, > { .parse_prop = parse_backlight, }, > { .parse_prop = parse_phy_handle, }, > + { .parse_prop = parse_ethernet, }, > { .parse_prop = parse_gpio_compat, }, > { .parse_prop = parse_interrupts, }, > { .parse_prop = parse_regulators, }, I don't have this exact setup to test, so I'll let Andrew do it, but I have a question: DSA sets up a device link to its master in dsa_master_setup. It does this to autoremove itself when the DSA master gets removed, but fundamentally it does it after the entire EPROBE_DEFER shebang discussed above has already happened. If your patch works, we can drop the manually added device link, right? There's also the question of what to do in case of multiple DSA masters (multiple "ethernet" properties). Right now, if you describe two DSA masters in the device tree, DSA will pick the first DSA master and use just that (it doesn't have full support for more than one). But even though the second DSA master is not used for anything, with your change, unbinding it will also unbind the switch, will it not?
On Thu, Aug 19, 2021 at 4:23 AM Vladimir Oltean <vladimir.oltean@nxp.com> wrote: > > On Wed, Aug 18, 2021 at 08:28:44PM -0700, Saravana Kannan wrote: > > On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > > > > > Hi Saravana, > > > > > > On 8/18/21 4:46 AM, Saravana Kannan wrote: > > > > On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote: > > > >> > > > >> Hi Alvin, > > > >> > > > >> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote: > > > >>> I have an observation that's slightly out of the scope of your patch, > > > >>> but I'll post here on the off chance that you find it relevant. > > > >>> Apologies if it's out of place. > > > >>> > > > >>> Do these integrated NXP PHYs use a specific PHY driver, or do they just > > > >>> use the Generic PHY driver? > > > >> > > > >> They refuse to probe at all with the Generic PHY driver. I have been > > > >> caught off guard a few times now when I had a kernel built with > > > >> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case. > > > >> > > > >>> If the former is the case, do you experience that the PHY driver fails > > > >>> to get probed during mdiobus registration if the kernel uses > > > >>> fw_devlink=on? > > > >> > > > >> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first > > > >> time I do it. It behaves exactly as you say. > > Sorry for the delay, I wanted to reconfirm what I said (hint, I was wrong). > > In my case, whatever I do, I cannot get the driver core enforce a device > link between the ethernet-switch and the PHY. > > So I cannot actually see the same issue. What I was seeing was in fact > stupid testing on my part (it was working with the PHY driver as > built-in, it was working, then I made it a module, it broke, I forgot to > switch it back to module, then I thought it's broken while the PHY is > built-in). Sorry, this email is too "messy" for me and I joined this thread halfway through. So if everything is working fine for you with fw_devlink=on, I'd rather not dig into this. Too much stuff to do. I'd be nice if you had put the boot logs, the DT file and the shell command stuff into separate files and attached them. I guess I could do that myself, but that's going to happen only when I get around to it :) > > > > >> > > > >>> > > > >>> In my case I am writing a new subdriver for realtek-smi, a DSA driver > > > >>> which registers an internal MDIO bus analogously to sja1105, which is > > > >>> why I'm asking. I noticed a deferred probe of the PHY driver because the > > > >>> supplier (ethernet-switch) is not ready - presumably because all of this > > > >>> is happening in the probe of the switch driver. See below: > > > >>> > > > >>> [ 83.653213] device_add:3270: device: 'SMI-0': device_add > > > >>> [ 83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0 > > > >>> [ 83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add > > > >>> [ 83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch > > > >>> [ 83.654291] libphy: SMI slave MII: probed > > > >>> ... > > > >>> [ 83.659809] device_add:3270: device: 'SMI-0:00': device_add > > > >>> [ 83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00 > > > >>> [ 83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00 > > > >>> [ 83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add > > > >>> [ 83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list > > > >>> [ 83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list > > > >>> [ 83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch > > > >>> [ 83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet > > > >>> [ 83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready > > > >>> [ 83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list > > > >> > > > >> So it's a circular dependency? Switch cannot finish probing because it > > > >> cannot connect to PHY, which cannot probe because switch has not > > > >> finished probing, which.... > > > > > > > > Hi Vladimir/Alvin, > > > > > > > > If there's a cyclic dependency between two devices, then fw_devlink=on > > > > is smart enough to notice that. Once it notices a cycle, it knows that > > > > it can't tell which one is the real dependency and which one is the > > > > false dependency and so stops enforcing ordering between the devices > > > > in the cycle. > > > > > > > > But fw_devlink doesn't understand all the properties yet. Just most of > > > > them and I'm always trying to add more. So when it only understands > > > > the property that's causing the false dependency but not the property > > > > that causes the real dependency, it can cause issues like this where > > > > fw_devlink=on enforces the false dependency and the driver/code > > > > enforces the real dependency. These are generally easy to fix -- you > > > > just need to teach fw_devlink how to parse more properties. > > > > > > > > This is just a preliminary analysis since I don't have all the info > > > > yet -- so I could be wrong. With that said, I happened to be working > > > > on adding fw_devlink support for phy-handle property and I think it > > > > should fix your issue with fw_devlink=on. Can you give [1] a shot? > > > > > > I tried [1] but it did not seem to have any effect. > > > > > I applied the phy-handle patch, and here are my boot logs. > > [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083] > [ 0.000000] Linux version 5.14.0-rc5+ (tigrisor@skbuf) (aarch64-linux-gnu-gcc (Linaro GCC 7.5-2019.12) 7.5.0, GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706) #440 SMP Thu Aug 19 13:48:11 EEST 2021 > [ 0.000000] Machine model: NXP Layerscape LX2160ABLUEBOX3 > [ 0.000000] fw_devlink_setup: fw_devlink_flags = 0x120 > [ 0.000000] efi: UEFI not found. > [ 0.000000] [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader! Snipped the logs > [ 19.389542] TERM=linux > INIT: version booting > Fuse filesystem already available. > Mounting fuse control filesystem. > Starting udev > [ 19.720832] udevd[480]: starting version 3.2.9 > [ 19.756177] udevd[481]: starting eudev-3.2.9 > [ 19.902508] fsl_dpaa2_eth dpni.3 dpmac10: renamed from eth2 > [ 19.908092] net eth2: renaming to dpmac10 > [ 19.984284] fsl_dpaa2_eth dpni.4 dpmac17: renamed from eth1 > [ 19.989997] net eth1: renaming to dpmac17 > [ 20.020231] fsl_dpaa2_eth dpni.2 dpmac9: renamed from eth3 > [ 20.025849] net eth3: renaming to dpmac9 > [ 20.079989] fsl_dpaa2_eth dpni.0 dpmac5: renamed from eth4 > [ 20.085493] net eth4: renaming to dpmac5 > [ 20.127996] fsl_dpaa2_eth dpni.5 dpmac18: renamed from eth0 > [ 20.133603] net eth0: renaming to dpmac18 > [ 20.166859] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck. > [ 20.206953] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: none. > Starting tcf-agent: OK > > lx2160abluebox3 login: root > root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/ > total 0 > drwxr-xr-x 2 root root 0 Jan 1 1970 . > drwxr-xr-x 4 root root 0 Jan 1 1970 .. > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000:00 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:00 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000:01 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:01 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000:02 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:02 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000:08 -> ../../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000/0x0000000008b96000:08 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b97000:00 -> ../../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000/0x0000000008b97000:00 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b97000:08 -> ../../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000/0x0000000008b97000:08 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c17000:00 -> ../../../devices/platform/soc/8c17000.mdio/mdio_bus/0x0000000008c17000/0x0000000008c17000:00 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c27000:00 -> ../../../devices/platform/soc/8c27000.mdio/mdio_bus/0x0000000008c27000/0x0000000008c27000:00 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c2b000:00 -> ../../../devices/platform/soc/8c2b000.mdio/mdio_bus/0x0000000008c2b000/0x0000000008c2b000:00 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:01 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:01 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:02 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:02 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:03 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:03 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:04 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:04 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:05 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:05 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1:06 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1/spi1.0-base-t1:06 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:01 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:01 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:02 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:02 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:03 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:03 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:04 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:04 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:05 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:05 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1:06 -> ../../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1/spi1.2-base-t1:06 > root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/spi1.0-base-t1\:01 > spi1.0-base-t1:01/ > root@lx2160abluebox3:~# ls -la /sys/bus/mdio_bus/devices/spi1.0-base-t1\:01/ > total 0 > drwxr-xr-x 4 root root 0 Jan 1 1970 . > drwxr-xr-x 10 root root 0 Jan 1 1970 .. > lrwxrwxrwx 1 root root 0 Mar 9 12:35 driver -> ../../../../../../../../../../../../../../bus/mdio_bus/drivers/NXP C45 TJA1103 > lrwxrwxrwx 1 root root 0 Mar 9 12:35 of_node -> ../../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio > @0/ethernet-phy@1 > -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_dev_flags > -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_has_fixups > -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_id > -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_interface > drwxr-xr-x 2 root root 0 Mar 9 12:35 power > drwxr-xr-x 2 root root 0 Mar 9 12:35 statistics > lrwxrwxrwx 1 root root 0 Mar 9 12:35 subsystem -> ../../../../../../../../../../../../../../bus/mdio_bus > -rw-r--r-- 1 root root 4096 Jan 1 1970 uevent > root@lx2160abluebox3:~# > root@lx2160abluebox3:~# ls -la /sys/bus/spi/devices/spi1.0/ > total 0 > drwxr-xr-x 7 root root 0 Jan 1 1970 . > drwxr-xr-x 6 root root 0 Jan 1 1970 .. > lrwxrwxrwx 1 root root 0 Mar 9 12:36 driver -> ../../../../../../../../../../../bus/spi/drivers/sja1105 > -rw-r--r-- 1 root root 4096 Mar 9 12:36 driver_override > drwxr-xr-x 4 root root 0 Jan 1 1970 mdio_bus > -r--r--r-- 1 root root 4096 Mar 9 12:36 modalias > drwxr-xr-x 10 root root 0 Jan 1 1970 net > lrwxrwxrwx 1 root root 0 Mar 9 12:36 of_node -> ../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0 > drwxr-xr-x 2 root root 0 Mar 9 12:36 power > drwxr-xr-x 3 root root 0 Jan 1 1970 ptp > drwxr-xr-x 2 root root 0 Mar 9 12:36 statistics > lrwxrwxrwx 1 root root 0 Jan 1 1970 subsystem -> ../../../../../../../../../../../bus/spi > lrwxrwxrwx 1 root root 0 Mar 9 12:36 supplier:fsl-mc:dpni.4 -> ../../../../../../../../../../virtual/devlink/fsl-mc:dpni.4--spi:spi1.0 > -rw-r--r-- 1 root root 4096 Jan 1 1970 uevent > INIT: Id "S0" respawning too fast: disabled for 5 minutes c > lass/mdio_bus/ > total 0 > drwxr-xr-x 2 root root 0 Jan 1 1970 . > drwxr-xr-x 70 root root 0 Jan 1 1970 .. > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b96000 -> ../../devices/platform/soc/8b96000.mdio/mdio_bus/0x0000000008b96000 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008b97000 -> ../../devices/platform/soc/8b97000.mdio/mdio_bus/0x0000000008b97000 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c17000 -> ../../devices/platform/soc/8c17000.mdio/mdio_bus/0x0000000008c17000 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c27000 -> ../../devices/platform/soc/8c27000.mdio/mdio_bus/0x0000000008c27000 > lrwxrwxrwx 1 root root 0 Jan 1 1970 0x0000000008c2b000 -> ../../devices/platform/soc/8c2b000.mdio/mdio_bus/0x0000000008c2b000 > lrwxrwxrwx 1 root root 0 Jan 1 1970 fixed-0 -> ../../devices/platform/Fixed MDIO bus.0/mdio_bus/fixed-0 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-base-t1 -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-base-t1 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.0-pcs -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.0/mdio_bus/spi1.0-pcs > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-base-t1 -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-base-t1 > lrwxrwxrwx 1 root root 0 Jan 1 1970 spi1.2-pcs -> ../../devices/platform/soc/2000000.i2c/i2c-0/i2c-9/i2c-10/10-0028/spi_master/spi1/spi1.2/mdio_bus/spi1.2-pcs > root@lx2160abluebox3:~# ls -la /sys/class/mdio_bus/spi1.0-base-t1/ > total 0 > drwxr-xr-x 10 root root 0 Jan 1 1970 . > drwxr-xr-x 4 root root 0 Jan 1 1970 .. > lrwxrwxrwx 1 root root 0 Mar 9 12:37 device -> ../../../spi1.0 > lrwxrwxrwx 1 root root 0 Mar 9 12:37 of_node -> ../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio@0 > drwxr-xr-x 2 root root 0 Mar 9 12:37 power > drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:01 > drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:02 > drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:03 > drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:04 > drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:05 > drwxr-xr-x 4 root root 0 Jan 1 1970 spi1.0-base-t1:06 > drwxr-xr-x 2 root root 0 Mar 9 12:37 statistics > lrwxrwxrwx 1 root root 0 Jan 1 1970 subsystem -> ../../../../../../../../../../../../../class/mdio_bus > -rw-r--r-- 1 root root 4096 Jan 1 1970 uevent > -r--r--r-- 1 root root 4096 Mar 9 12:37 waiting_for_supplier > root@lx2160abluebox3:~# ls -la /sys/class/mdio_bus/spi1.0-base-t1/spi1.0-base-t1\:01/ > total 0 > drwxr-xr-x 4 root root 0 Jan 1 1970 . > drwxr-xr-x 10 root root 0 Jan 1 1970 .. > lrwxrwxrwx 1 root root 0 Mar 9 12:35 driver -> ../../../../../../../../../../../../../../bus/mdio_bus/drivers/NXP C45 TJA1103 > lrwxrwxrwx 1 root root 0 Mar 9 12:35 of_node -> ../../../../../../../../../../../../../../firmware/devicetree/base/soc/i2c@2000000/i2c-mux@77/i2c@7/i2c-mux@75/i2c@0/spi@28/ethernet-switch@0/mdios/mdio > @0/ethernet-phy@1 > -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_dev_flags > -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_has_fixups > -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_id > -r--r--r-- 1 root root 4096 Mar 9 12:35 phy_interface > drwxr-xr-x 2 root root 0 Mar 9 12:35 power > drwxr-xr-x 2 root root 0 Mar 9 12:35 statistics > lrwxrwxrwx 1 root root 0 Mar 9 12:35 subsystem -> ../../../../../../../../../../../../../../bus/mdio_bus > -rw-r--r-- 1 root root 4096 Jan 1 1970 uevent > > As you can see, there was no provider/consumer relationship enforced > between the PHY at spi1.0-base-t1\:01 and the ethernet-switch. So the only thing that's "not working" right now is that you are saying the fw_devlink phy-handle patch doesn't seem to have an impact? To make it easy for me, can you give me the name of this node in DT? There are too many MDIOs and SPIs and PHY for me to keep track in my head. Since this isn't explicitly breaking anything, I'll get around to this when I have time. -Saravana
On Thu, Aug 19, 2021 at 6:42 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > Hi Saravana, > > Thanks for your lucid analysis, it's very much appreciated. Please find > my replies inline - but in summary, I don't think there is anything more > I can ask of you right now. You are welcome. > > On 8/19/21 5:28 AM, Saravana Kannan wrote: > > On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > >> > >> Hi Saravana, > >> > >> On 8/18/21 4:46 AM, Saravana Kannan wrote: > >>> On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote: > >>>> > >>>> Hi Alvin, > >>>> > >>>> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote: > >>>>> I have an observation that's slightly out of the scope of your patch, > >>>>> but I'll post here on the off chance that you find it relevant. > >>>>> Apologies if it's out of place. > >>>>> > >>>>> Do these integrated NXP PHYs use a specific PHY driver, or do they just > >>>>> use the Generic PHY driver? > >>>> > >>>> They refuse to probe at all with the Generic PHY driver. I have been > >>>> caught off guard a few times now when I had a kernel built with > >>>> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case. > >>>> > >>>>> If the former is the case, do you experience that the PHY driver fails > >>>>> to get probed during mdiobus registration if the kernel uses > >>>>> fw_devlink=on? > >>>> > >>>> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first > >>>> time I do it. It behaves exactly as you say. > >>>> > >>>>> > >>>>> In my case I am writing a new subdriver for realtek-smi, a DSA driver > >>>>> which registers an internal MDIO bus analogously to sja1105, which is > >>>>> why I'm asking. I noticed a deferred probe of the PHY driver because the > >>>>> supplier (ethernet-switch) is not ready - presumably because all of this > >>>>> is happening in the probe of the switch driver. See below: > >>>>> > >>>>> [ 83.653213] device_add:3270: device: 'SMI-0': device_add > >>>>> [ 83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0 > >>>>> [ 83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add > >>>>> [ 83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch > >>>>> [ 83.654291] libphy: SMI slave MII: probed > >>>>> ... > >>>>> [ 83.659809] device_add:3270: device: 'SMI-0:00': device_add > >>>>> [ 83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00 > >>>>> [ 83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00 > >>>>> [ 83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add > >>>>> [ 83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list > >>>>> [ 83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list > >>>>> [ 83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch > >>>>> [ 83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet > >>>>> [ 83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready > >>>>> [ 83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list > >>>> > >>>> So it's a circular dependency? Switch cannot finish probing because it > >>>> cannot connect to PHY, which cannot probe because switch has not > >>>> finished probing, which.... > >>> > >>> Hi Vladimir/Alvin, > >>> > >>> If there's a cyclic dependency between two devices, then fw_devlink=on > >>> is smart enough to notice that. Once it notices a cycle, it knows that > >>> it can't tell which one is the real dependency and which one is the > >>> false dependency and so stops enforcing ordering between the devices > >>> in the cycle. > >>> > >>> But fw_devlink doesn't understand all the properties yet. Just most of > >>> them and I'm always trying to add more. So when it only understands > >>> the property that's causing the false dependency but not the property > >>> that causes the real dependency, it can cause issues like this where > >>> fw_devlink=on enforces the false dependency and the driver/code > >>> enforces the real dependency. These are generally easy to fix -- you > >>> just need to teach fw_devlink how to parse more properties. > >>> > >>> This is just a preliminary analysis since I don't have all the info > >>> yet -- so I could be wrong. With that said, I happened to be working > >>> on adding fw_devlink support for phy-handle property and I think it > >>> should fix your issue with fw_devlink=on. Can you give [1] a shot? > >> > >> I tried [1] but it did not seem to have any effect. > >> > >>> > >>> If it doesn't fix it, can one of you please point me to an upstream > >>> dts (not dtsi) file for a platform in which you see this issue? And > >>> ideally also the DT nodes and their drivers that are involved in this > >>> cycle? With that info, I should be able to root cause this if the > >>> patch above doesn't already fix it. > >> > >> I'm working with a non-upstream dts - maybe Vladimir is using an > >> upstream one? The pattern among the drivers/dts is common between our > >> two cases. > > > > Ideally, I can get a fully upstream example where this issue is > > happening so that I can look at the actual code that's hitting this > > issue and be sure my analysis is right. > > Due to NDA issues at work, I am unable to publish the DTS right now. But > based on your analysis below, I believe that the problem I am > experiencing is exactly for the reasons you describe. > > > > >> > >> But for the sake of this discussion, my dts is pretty much the same as > >> what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The > >> nodes of interest from that dts file are below, and the driver is in > >> drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the > >> Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part > >> of the mdiobus registration, but that never happens. See my previous > >> reply for a debug log. > > > > Your DTS might be similar to this, but the driver code also matters > > for me to be sure. Anyway, I took a look at this, but my analysis > > below is going to be sketchy because I'm not looking at the actual > > code that's reproducing this issue. > > The driver code I am working with is pretty much the same in the probe > path (the driver is modelled on rtl8366rb.c and hooks into > realtek-smi-core.c), so it was not a waste of your time to analyze the > upstream case. Thanks a lot. Good to know it's useful. > > > > > Assuming this issue actually happens with the example you pointed to > > (I don't know this yet), here's what is happening: > > > > The main problem is that the parent device switch seems to be assuming > > it's child/grandchild devices (mdiobus/PHYs) will have probed > > successfully as soon as they are added. This assumption is not true > > and can be broken for multiple reasons such as: > > > > 1. The driver for the child devices (PHYs in this case) could be > > loaded as a module after the parent (switch) is probed. So when the > > devices are added, the PHYs would not be probed. > > 2. The child devices could defer probe because one of their suppliers > > isn't ready yet. Either because of fw_devlink=on or the framework > > itself returning -EPROBE_DEFER. > > 3. The child devices could be getting probed asynchronously. So the > > device_add() would kick off a thread to probe the child devices in a > > separate thread. > > > > (2) is what is happening in this case. fw_devlink=on sees that > > "switch" implements the "switch_intc" and "switch" hasn't finished > > probing yet. So it has no way of knowing that switch_intc is actually > > ready. And even if switch_intc was registered as part of switch's > > probe() by the time the PHYs are added, switch_intc could get > > deregistered if the probe fails at a later point. So until probe() > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc) > > is ready. Which is good in general because you won't have to > > forcefully unbind (if that is even handled correctly in the first > > place) the consumers of a device if it fails probe() half way through > > registering a few services. > > Right, that makes perfect sense. > > > > > I don't fully understand the networking frameworks, but I think > > Vladimir might have a point in his earlier reply [1]. If you can make > > the switch driver not assume its child PHYs are ready during the > > switch's probe and instead have the switch check if the PHYs are ready > > when the switch is "opened" that'd be better. > > I am pretty new to the DSA subsystem, I know absolutely nothing about DSA. Heck, I don't even know what it stands for. It's just about as informative as "XYZ" to me :) > but it appears to me that DSA is > making this assumption: I don't think DSA switches have a concept of > "open", as everything is handled in the dsa_register_switch() call in a > DSA driver's probe function. And as Vladimir pointed out, this is also > where phylink_of_phy_connect() is being called. Yeah, I noticed that part. > A quick look at the > other DSA drivers suggests that mv88e6xxx may also suffer from case (2) > above, so it seems like a more general issue. > I obviously want fw_devlink=3Don to be used by everyone and not have people use fw_devlink=3Dpermissive. In this scenario, I think fw_devlink=3Don is doing the right thing too. So this is where I'm hoping the network experts can jump in and fix the general DSA issue and I can help with the parts I understand. -Saravana P.S: Alvin, I accidentally replied only to you. So sending it to the list again. If you see this email twice, that's why :) > > > > We can come up with hacks that'll delete the dependency that > > fw_devlink=on is trying to enforce, but IMHO the proper fix is to have > > parent drivers not assume child devices will be probed as soon as > > device_add(child) returns. That's not guaranteed at all. > > You enumerated a number of reasons why, so yes, I agree that it is not > necessarily a safe assumption to make. > > > > > Btw, I do know why things work when you do the module load/unload > > thing you mention in [2]. That has to do with some forced deletion of > > dependencies that happens when device_bind_driver() is called when the > > Generic PHY driver is used. The reason for why that's done is kind of > > unrelated to the issue at hand, but the comment for > > device_links_force_bind() should tell you why. > > Thanks for clearing that up. > > Kind regards, > Alvin
On Thu, Aug 19, 2021 at 5:37 PM Vladimir Oltean <olteanv@gmail.com> wrote: > > On Thu, Aug 19, 2021 at 04:52:43PM -0700, Saravana Kannan wrote: > > On Thu, Aug 19, 2021 at 6:35 AM Andrew Lunn <andrew@lunn.ch> wrote: > > > > > > > (2) is what is happening in this case. fw_devlink=on sees that > > > > "switch" implements the "switch_intc" and "switch" hasn't finished > > > > probing yet. So it has no way of knowing that switch_intc is actually > > > > ready. And even if switch_intc was registered as part of switch's > > > > probe() by the time the PHYs are added, switch_intc could get > > > > deregistered if the probe fails at a later point. So until probe() > > > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc) > > > > is ready. Which is good in general because you won't have to > > > > forcefully unbind (if that is even handled correctly in the first > > > > place) the consumers of a device if it fails probe() half way through > > > > registering a few services. > > > > I had to read your email a couple of times before I understood it. I > > think I do now, but apologies if I'm not making sense. > > > > > > > > There are actually a few different circular references with the way > > > switches work. Take for example: > > > > > > &fec1 { > > > phy-mode = "rmii"; > > > pinctrl-names = "default"; > > > pinctrl-0 = <&pinctrl_fec1>; > > > status = "okay"; > > > > > > fixed-link { > > > speed = <100>; > > > full-duplex; > > > }; > > > > > > mdio1: mdio { > > > #address-cells = <1>; > > > #size-cells = <0>; > > > clock-frequency = <12500000>; > > > suppress-preamble; > > > status = "okay"; > > > > > > switch0: switch0@0 { > > > compatible = "marvell,mv88e6190"; > > > pinctrl-0 = <&pinctrl_gpio_switch0>; > > > pinctrl-names = "default"; > > > reg = <0>; > > > eeprom-length = <65536>; > > > interrupt-parent = <&gpio3>; > > > interrupts = <2 IRQ_TYPE_LEVEL_LOW>; > > > interrupt-controller; > > > #interrupt-cells = <2>; > > > > > > ports { > > > #address-cells = <1>; > > > #size-cells = <0>; > > > > > > port@0 { > > > reg = <0>; > > > label = "cpu"; > > > ethernet = <&fec1>; > > > > > > fixed-link { > > > speed = <100>; > > > full-duplex; > > > }; > > > }; > > > > > > FEC is an ethernet controller. It has an MDIO bus, and on the bus is > > > an Ethernet switch. port 0 of the Ethernet switch is connected to the > > > FEC ethernet controller. > > > > > > While the FEC probes, it will at some point register its MDIO bus. At > > > that point, the MDIO bus is probed, the switch is found, and > > > registered with the switch core. The switch core looks for the port > > > with an ethernet property and goes looking for that ethernet > > > interface. But that this point in time, the FEC probe has only got as > > > far as registering the MDIO bus. The interface itself is not > > > registered. So finding the interface fails, and we go into > > > EPROBE_DEFER for probing the switch. > > > > Ok, I understood up to here. Couple of questions: > > Is this EPROBE_DEFER causing an issue? Wouldn't the switch then > > probe successfully when it's reattempted? And then things work > > normally? I don't see what the problem is. > > It's not an issue per se, since it's not a fully circular dependency: > the DSA master (the FEC controller) does not have any dependency on the > switch beneath it to probe (there's nothing like a phy-handle from the > FEC to the switch or to something provided by it). > > A few EPROBE_DEFER iterations later the switch will finally find its DSA > master fully probed via of_find_net_device_by_node. > > Andrew is wondering how to avoid those extra EPROBE_DEFER iterations. > It is weird that the entire functionality of the system depends on those > EPROBE_DEFERs, typically you'd expect that EPROBE_DEFER just serializes > asynchronous probing of drivers with interdependencies. But in this case > it serializes synchronous probing. > > > > It is pretty hard to solve. An Ethernet interface can be used by the > > > kernel itself, e.g. NFS root. At the point you call register_netdev() > > > in the probe function, to register the interface with the core, > > > > Are you using "ethernet interface" and "ethernet controller" > > interchangeably? Looking at some other drivers, it looks like the > > ethernet controlled (FEC) is what would call register_netdev(). So > > what's wrong with that happening if switch0 has not probed > > successfully? > > The "interface" and "controller" terms are not really interchangeable, > an interface can also be virtual (stacked interfaces on top of physical > ones, like VLAN or DSA) while network controllers are typically physical > (unless emulated). But that is not of importance. > > The context here is that you cannot solve the interdependency by > registering the DSA master (FEC) first, then its MDIO bus second (the > DSA switch probes on the DSA master's MDIO bus => if you do this, > of_find_net_device_by_node from the DSA layer would find its master the > first time). The reason you cannot do that is because you need the MDIO > bus for really basic stuff: you also have your Ethernet PHY on it, and > you need to initialize that in order to send traffic. And you need to be > able to send traffic as soon as register_netdev() completes. > > So since the driver initialization sequence has a single written order > regardless of whether DSA switches are attached or not, that order is > picked to be the one where traffic works as soon as register_netdev completes. > > > > it > > > needs to be fully ready to go. The networking stack can start using > > > the interface before register_netdev() even returns. So you cannot > > > first register the interface and then register the MDIO bus. > > > > > > I once looked to see if it was possible to tell the driver core to not > > > even bother probing a bus as soon as it is registered, go straight to > > > defer probe handling. Because this is one case we know it cannot > > > work. But it does not seem possible. > > > > fw_devlink doesn't understand the "ethernet" property. If I add that, > > then in the example you state above, switch0's probe won't even be > > called until the FEC probe returns. The change is pretty trivial > > (pasted below) -- can you try it out and tell me if it does what you > > need/want? > > > > -Saravana > > > > +++ b/drivers/of/property.c > > @@ -1292,6 +1292,7 @@ DEFINE_SIMPLE_PROP(resets, "resets", "#reset-cells") > > DEFINE_SIMPLE_PROP(leds, "leds", NULL) > > DEFINE_SIMPLE_PROP(backlight, "backlight", NULL) > > DEFINE_SIMPLE_PROP(phy_handle, "phy-handle", NULL) > > +DEFINE_SIMPLE_PROP(ethernet, "ethernet", NULL) > > DEFINE_SUFFIX_PROP(regulators, "-supply", NULL) > > DEFINE_SUFFIX_PROP(gpio, "-gpio", "#gpio-cells") > > > > @@ -1381,6 +1382,7 @@ static const struct supplier_bindings > > of_supplier_bindings[] = { > > { .parse_prop = parse_leds, }, > > { .parse_prop = parse_backlight, }, > > { .parse_prop = parse_phy_handle, }, > > + { .parse_prop = parse_ethernet, }, > > { .parse_prop = parse_gpio_compat, }, > > { .parse_prop = parse_interrupts, }, > > { .parse_prop = parse_regulators, }, > > I don't have this exact setup to test, so I'll let Andrew do it, but I > have a question: DSA sets up a device link to its master in dsa_master_setup. > It does this to autoremove itself when the DSA master gets removed, but > fundamentally it does it after the entire EPROBE_DEFER shebang discussed > above has already happened. If your patch works, we can drop the manually > added device link, right? Yes, you can drop that code if fw_devlink=on works and forms the links correctly (it should and if not it shouldn't be hard to fix -- just needs a bit of time to figure out why). > There's also the question of what to do in case of multiple DSA masters Sigh... I'm out of my depth with all this network specific discussions :( Especially when it comes to Linux network related frameworks. > (multiple "ethernet" properties). Right now, if you describe two DSA > masters in the device tree, DSA will pick the first DSA master and use > just that (it doesn't have full support for more than one). But even > though the second DSA master is not used for anything, with your change, > unbinding it will also unbind the switch, will it not? I think I understand your question. And the answer is "yes". It actually goes one step further. Until both all the DSA masters (devices pointed to by the "ethernet" properties) of a DSA are bound successfully, the DSA will not even be probed in the first place. You can set come command line args to timeout waiting for suppliers with missing drivers, but ideally you shouldn't need to use those (fw_devlink is fairly new and I'm continuing to work on improving it/making it work by default). -Saravana
On Thu, Aug 19, 2021 at 04:52:43PM -0700, Saravana Kannan wrote: > On Thu, Aug 19, 2021 at 6:35 AM Andrew Lunn <andrew@lunn.ch> wrote: > > > > > (2) is what is happening in this case. fw_devlink=on sees that > > > "switch" implements the "switch_intc" and "switch" hasn't finished > > > probing yet. So it has no way of knowing that switch_intc is actually > > > ready. And even if switch_intc was registered as part of switch's > > > probe() by the time the PHYs are added, switch_intc could get > > > deregistered if the probe fails at a later point. So until probe() > > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc) > > > is ready. Which is good in general because you won't have to > > > forcefully unbind (if that is even handled correctly in the first > > > place) the consumers of a device if it fails probe() half way through > > > registering a few services. > > I had to read your email a couple of times before I understood it. I > think I do now, but apologies if I'm not making sense. > > > > > There are actually a few different circular references with the way > > switches work. Take for example: > > > > &fec1 { > > phy-mode = "rmii"; > > pinctrl-names = "default"; > > pinctrl-0 = <&pinctrl_fec1>; > > status = "okay"; > > > > fixed-link { > > speed = <100>; > > full-duplex; > > }; > > > > mdio1: mdio { > > #address-cells = <1>; > > #size-cells = <0>; > > clock-frequency = <12500000>; > > suppress-preamble; > > status = "okay"; > > > > switch0: switch0@0 { > > compatible = "marvell,mv88e6190"; > > pinctrl-0 = <&pinctrl_gpio_switch0>; > > pinctrl-names = "default"; > > reg = <0>; > > eeprom-length = <65536>; > > interrupt-parent = <&gpio3>; > > interrupts = <2 IRQ_TYPE_LEVEL_LOW>; > > interrupt-controller; > > #interrupt-cells = <2>; > > > > ports { > > #address-cells = <1>; > > #size-cells = <0>; > > > > port@0 { > > reg = <0>; > > label = "cpu"; > > ethernet = <&fec1>; > > > > fixed-link { > > speed = <100>; > > full-duplex; > > }; > > }; > > > > FEC is an ethernet controller. It has an MDIO bus, and on the bus is > > an Ethernet switch. port 0 of the Ethernet switch is connected to the > > FEC ethernet controller. > > > > While the FEC probes, it will at some point register its MDIO bus. At > > that point, the MDIO bus is probed, the switch is found, and > > registered with the switch core. The switch core looks for the port > > with an ethernet property and goes looking for that ethernet > > interface. But that this point in time, the FEC probe has only got as > > far as registering the MDIO bus. The interface itself is not > > registered. So finding the interface fails, and we go into > > EPROBE_DEFER for probing the switch. > > Ok, I understood up to here. Couple of questions: > Is this EPROBE_DEFER causing an issue? Wouldn't the switch then > probe successfully when it's reattempted? And then things work > normally? I don't see what the problem is. Everything works on the second time around. So there is no problem, other than we waste time trying to probe the switch, which we know is going to fail. Depending on the setup, this can add 1/2 second to the boot time. > > It is pretty hard to solve. An Ethernet interface can be used by the > > kernel itself, e.g. NFS root. At the point you call register_netdev() > > in the probe function, to register the interface with the core, > > Are you using "ethernet interface" and "ethernet controller" > interchangeably? Yeh, pretty much. > Looking at some other drivers, it looks like the > ethernet controlled (FEC) is what would call register_netdev(). So > what's wrong with that happening if switch0 has not probed > successfully? > > > it > > needs to be fully ready to go. The networking stack can start using > > the interface before register_netdev() even returns. So you cannot > > first register the interface and then register the MDIO bus. > > > > I once looked to see if it was possible to tell the driver core to not > > even bother probing a bus as soon as it is registered, go straight to > > defer probe handling. Because this is one case we know it cannot > > work. But it does not seem possible. > > fw_devlink doesn't understand the "ethernet" property. If I add that, > then in the example you state above, switch0's probe won't even be > called until the FEC probe returns. The change is pretty trivial > (pasted below) -- can you try it out and tell me if it does what you > need/want? Cool. I will try this. Thanks Andrew
On Thu, Aug 19, 2021 at 6:08 PM Saravana Kannan <saravanak@google.com> wrote: > > On Thu, Aug 19, 2021 at 6:42 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > > > Hi Saravana, > > > > Thanks for your lucid analysis, it's very much appreciated. Please find > > my replies inline - but in summary, I don't think there is anything more > > I can ask of you right now. > > You are welcome. > > > > > On 8/19/21 5:28 AM, Saravana Kannan wrote: > > > On Wed, Aug 18, 2021 at 3:18 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > >> > > >> Hi Saravana, > > >> > > >> On 8/18/21 4:46 AM, Saravana Kannan wrote: > > >>> On Tue, Aug 17, 2021 at 3:31 PM Vladimir Oltean <olteanv@gmail.com> wrote: > > >>>> > > >>>> Hi Alvin, > > >>>> > > >>>> On Tue, Aug 17, 2021 at 09:25:28PM +0000, Alvin Šipraga wrote: > > >>>>> I have an observation that's slightly out of the scope of your patch, > > >>>>> but I'll post here on the off chance that you find it relevant. > > >>>>> Apologies if it's out of place. > > >>>>> > > >>>>> Do these integrated NXP PHYs use a specific PHY driver, or do they just > > >>>>> use the Generic PHY driver? > > >>>> > > >>>> They refuse to probe at all with the Generic PHY driver. I have been > > >>>> caught off guard a few times now when I had a kernel built with > > >>>> CONFIG_NXP_C45_TJA11XX_PHY=n and their probing returns -22 in that case. > > >>>> > > >>>>> If the former is the case, do you experience that the PHY driver fails > > >>>>> to get probed during mdiobus registration if the kernel uses > > >>>>> fw_devlink=on? > > >>>> > > >>>> I don't test with "fw_devlink=on" in /proc/cmdline, this is the first > > >>>> time I do it. It behaves exactly as you say. > > >>>> > > >>>>> > > >>>>> In my case I am writing a new subdriver for realtek-smi, a DSA driver > > >>>>> which registers an internal MDIO bus analogously to sja1105, which is > > >>>>> why I'm asking. I noticed a deferred probe of the PHY driver because the > > >>>>> supplier (ethernet-switch) is not ready - presumably because all of this > > >>>>> is happening in the probe of the switch driver. See below: > > >>>>> > > >>>>> [ 83.653213] device_add:3270: device: 'SMI-0': device_add > > >>>>> [ 83.653905] device_pm_add:136: PM: Adding info for No Bus:SMI-0 > > >>>>> [ 83.654055] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0': device_add > > >>>>> [ 83.654224] device_link_add:843: mdio_bus SMI-0: Linked as a sync state only consumer to ethernet-switch > > >>>>> [ 83.654291] libphy: SMI slave MII: probed > > >>>>> ... > > >>>>> [ 83.659809] device_add:3270: device: 'SMI-0:00': device_add > > >>>>> [ 83.659883] bus_add_device:447: bus: 'mdio_bus': add device SMI-0:00 > > >>>>> [ 83.659970] device_pm_add:136: PM: Adding info for mdio_bus:SMI-0:00 > > >>>>> [ 83.660122] device_add:3270: device: 'platform:ethernet-switch--mdio_bus:SMI-0:00': device_add > > >>>>> [ 83.660274] devices_kset_move_last:2701: devices_kset: Moving SMI-0:00 to end of list > > >>>>> [ 83.660282] device_pm_move_last:203: PM: Moving mdio_bus:SMI-0:00 to end of list > > >>>>> [ 83.660293] device_link_add:859: mdio_bus SMI-0:00: Linked as a consumer to ethernet-switch > > >>>>> [ 83.660350] __driver_probe_device:736: bus: 'mdio_bus': __driver_probe_device: matched device SMI-0:00 with driver RTL8365MB-VC Gigabit Ethernet > > >>>>> [ 83.660365] device_links_check_suppliers:1001: mdio_bus SMI-0:00: probe deferral - supplier ethernet-switch not ready > > >>>>> [ 83.660376] driver_deferred_probe_add:138: mdio_bus SMI-0:00: Added to deferred list > > >>>> > > >>>> So it's a circular dependency? Switch cannot finish probing because it > > >>>> cannot connect to PHY, which cannot probe because switch has not > > >>>> finished probing, which.... > > >>> > > >>> Hi Vladimir/Alvin, > > >>> > > >>> If there's a cyclic dependency between two devices, then fw_devlink=on > > >>> is smart enough to notice that. Once it notices a cycle, it knows that > > >>> it can't tell which one is the real dependency and which one is the > > >>> false dependency and so stops enforcing ordering between the devices > > >>> in the cycle. > > >>> > > >>> But fw_devlink doesn't understand all the properties yet. Just most of > > >>> them and I'm always trying to add more. So when it only understands > > >>> the property that's causing the false dependency but not the property > > >>> that causes the real dependency, it can cause issues like this where > > >>> fw_devlink=on enforces the false dependency and the driver/code > > >>> enforces the real dependency. These are generally easy to fix -- you > > >>> just need to teach fw_devlink how to parse more properties. > > >>> > > >>> This is just a preliminary analysis since I don't have all the info > > >>> yet -- so I could be wrong. With that said, I happened to be working > > >>> on adding fw_devlink support for phy-handle property and I think it > > >>> should fix your issue with fw_devlink=on. Can you give [1] a shot? > > >> > > >> I tried [1] but it did not seem to have any effect. > > >> > > >>> > > >>> If it doesn't fix it, can one of you please point me to an upstream > > >>> dts (not dtsi) file for a platform in which you see this issue? And > > >>> ideally also the DT nodes and their drivers that are involved in this > > >>> cycle? With that info, I should be able to root cause this if the > > >>> patch above doesn't already fix it. > > >> > > >> I'm working with a non-upstream dts - maybe Vladimir is using an > > >> upstream one? The pattern among the drivers/dts is common between our > > >> two cases. > > > > > > Ideally, I can get a fully upstream example where this issue is > > > happening so that I can look at the actual code that's hitting this > > > issue and be sure my analysis is right. > > > > Due to NDA issues at work, I am unable to publish the DTS right now. But > > based on your analysis below, I believe that the problem I am > > experiencing is exactly for the reasons you describe. > > > > > > > >> > > >> But for the sake of this discussion, my dts is pretty much the same as > > >> what you will find in arch/arm/boot/dts/gemini-dlink-dir-685.dts. The > > >> nodes of interest from that dts file are below, and the driver is in > > >> drivers/net/ds/{realtek-smi-core.c,rtl8366rb.c}. It's expected that the > > >> Realtek PHY driver in drivers/net/phy/realtek.c will get probed as part > > >> of the mdiobus registration, but that never happens. See my previous > > >> reply for a debug log. > > > > > > Your DTS might be similar to this, but the driver code also matters > > > for me to be sure. Anyway, I took a look at this, but my analysis > > > below is going to be sketchy because I'm not looking at the actual > > > code that's reproducing this issue. > > > > The driver code I am working with is pretty much the same in the probe > > path (the driver is modelled on rtl8366rb.c and hooks into > > realtek-smi-core.c), so it was not a waste of your time to analyze the > > upstream case. Thanks a lot. > > Good to know it's useful. > > > > > > > > > Assuming this issue actually happens with the example you pointed to > > > (I don't know this yet), here's what is happening: > > > > > > The main problem is that the parent device switch seems to be assuming > > > it's child/grandchild devices (mdiobus/PHYs) will have probed > > > successfully as soon as they are added. This assumption is not true > > > and can be broken for multiple reasons such as: > > > > > > 1. The driver for the child devices (PHYs in this case) could be > > > loaded as a module after the parent (switch) is probed. So when the > > > devices are added, the PHYs would not be probed. > > > 2. The child devices could defer probe because one of their suppliers > > > isn't ready yet. Either because of fw_devlink=on or the framework > > > itself returning -EPROBE_DEFER. > > > 3. The child devices could be getting probed asynchronously. So the > > > device_add() would kick off a thread to probe the child devices in a > > > separate thread. > > > > > > (2) is what is happening in this case. fw_devlink=on sees that > > > "switch" implements the "switch_intc" and "switch" hasn't finished > > > probing yet. So it has no way of knowing that switch_intc is actually > > > ready. And even if switch_intc was registered as part of switch's > > > probe() by the time the PHYs are added, switch_intc could get > > > deregistered if the probe fails at a later point. So until probe() > > > returns 0, fw_devlink can't be fully sure the supplier (switch_intc) > > > is ready. Which is good in general because you won't have to > > > forcefully unbind (if that is even handled correctly in the first > > > place) the consumers of a device if it fails probe() half way through > > > registering a few services. > > > > Right, that makes perfect sense. > > > > > > > > I don't fully understand the networking frameworks, but I think > > > Vladimir might have a point in his earlier reply [1]. If you can make > > > the switch driver not assume its child PHYs are ready during the > > > switch's probe and instead have the switch check if the PHYs are ready > > > when the switch is "opened" that'd be better. > > > > I am pretty new to the DSA subsystem, > > I know absolutely nothing about DSA. Heck, I don't even know what it > stands for. It's just about as informative as "XYZ" to me :) > > > but it appears to me that DSA is > > making this assumption: I don't think DSA switches have a concept of > > "open", as everything is handled in the dsa_register_switch() call in a > > DSA driver's probe function. And as Vladimir pointed out, this is also > > where phylink_of_phy_connect() is being called. > > Yeah, I noticed that part. > > > A quick look at the > > other DSA drivers suggests that mv88e6xxx may also suffer from case (2) > > above, so it seems like a more general issue. > > > > I obviously want fw_devlink=on to be used by everyone and not have > people use fw_devlink=permissive. In this scenario, I think > fw_devlink=on is doing the right thing too. So this is where I'm > hoping the network experts can jump in and fix the general DSA issue > and I can help with the parts I understand. Hi Alvin, Can you give this a shot to see if it fixes your issue? It basically delays the registration of dsa_register_switch() until all the consumers of this switch have probed. So it has a couple of caveats: 1. I'm hoping the PHYs are the only consumers of this switch. 2. All of them have to probe successfully before the switch will register itself. 3. If dsa_register_switch() fails, we can't defer the probe (because it already succeeded). But I'm not sure if it's a likely error code. But, for now, I want to see if it fixes your issues. -Saravana +++ b/drivers/net/dsa/realtek-smi-core.c @@ -454,14 +454,16 @@ static int realtek_smi_probe(struct platform_device *pdev) smi->ds->priv = smi; smi->ds->ops = var->ds_ops; - ret = dsa_register_switch(smi->ds); - if (ret) { - dev_err(dev, "unable to register switch ret = %d\n", ret); - return ret; - } return 0; } +static void realtek_smi_sync_state(struct device *dev) +{ + struct realtek_smi *smi = dev_get_drvdata(dev); + if (dsa_register_switch(smi->ds)) + dev_err(dev, "unable to register switch ret = %d\n", ret); +} + static int realtek_smi_remove(struct platform_device *pdev) { struct realtek_smi *smi = dev_get_drvdata(&pdev->dev); @@ -492,6 +494,7 @@ static struct platform_driver realtek_smi_driver = { .driver = { .name = "realtek-smi", .of_match_table = of_match_ptr(realtek_smi_of_match), + .sync_state = realtek_smi_sync_state, }, .probe = realtek_smi_probe, .remove = realtek_smi_remove,
> 3. If dsa_register_switch() fails, we can't defer the probe (because > it already succeeded). But I'm not sure if it's a likely error code. That is not a safe assumption. The tagging driver can be a module, and not be loaded yet. So we defer probing to allow it to load. Ethernet switches can be a big graph of parts, not a nice tree of parts. Andrew
On Fri, Aug 20, 2021 at 10:54 AM Andrew Lunn <andrew@lunn.ch> wrote: > > > 3. If dsa_register_switch() fails, we can't defer the probe (because > > it already succeeded). But I'm not sure if it's a likely error code. > > That is not a safe assumption. The tagging driver can be a module, and I don't know what "tagging driver" means. And yes, this is just a test patch for now. > not be loaded yet. So we defer probing to allow it to load. > > Ethernet switches can be a big graph of parts, not a nice tree of > parts. fw_devlink can handle graphs well. It has cycle detection for a reason :) Look at slide 10 for a complicated example: https://linuxplumbersconf.org/event/7/contributions/787/attachments/529/942/LPC_2020_-_fw_devlink.pdf However, after sending that email I realized that fw_devlink=on would make sure all the dependencies of this device would have probed first before it allows this to probe. So EPROBE_DEFER can't really happen. But for now I just want to see if it fixes the issue Alvin was seeing. -Saravana
Hi Saravana, Thanks for the follow-up. I tested your change and it does the trick: there is no deferral and the PHY driver gets probed first-try during the mdiobus registration during the call to dsa_register_switch(). I tested with the switch, PHY, and tagging drivers all builtin, or all modules, and it worked in both cases. On 8/20/21 6:52 PM, Saravana Kannan wrote: > Hi Alvin, > > Can you give this a shot to see if it fixes your issue? It basically > delays the registration of dsa_register_switch() until all the > consumers of this switch have probed. So it has a couple of caveats: Hm, weren't the only consumers the PHYs themselves? It seems like the main effect of your change is that - by doing the actual dsa_register_switch() call after the switch driver probe - the ethernet-switch (provider) is already probed, thereby allowing the PHY (consumer) to probe immediately. > 1. I'm hoping the PHYs are the only consumers of this switch. In my case that is true, if you count the mdio_bus as well: /sys/devices/platform/ethernet-switch# ls -l consumer\:* lrwxrwxrwx 1 root root 0 Aug 22 16:00 consumer:mdio_bus:SMI-0 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 lrwxrwxrwx 1 root root 0 Aug 22 16:00 consumer:mdio_bus:SMI-0:00 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 lrwxrwxrwx 1 root root 0 Aug 22 16:00 consumer:mdio_bus:SMI-0:01 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 lrwxrwxrwx 1 root root 0 Aug 22 16:00 consumer:mdio_bus:SMI-0:02 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 lrwxrwxrwx 1 root root 0 Aug 22 16:00 consumer:mdio_bus:SMI-0:03 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 > 2. All of them have to probe successfully before the switch will > register itself. Yes. > 3. If dsa_register_switch() fails, we can't defer the probe (because > it already succeeded). But I'm not sure if it's a likely error code. It's of course possible that dsa_register_switch() fails. Assuming fw_devlink is doing its job properly, I think the reason is most likely going to be something specific to the driver, such as a communication timeout with the switch hardware itself. I get the impression that you don't necessarily regard this change as a proper fix, so I'm happy to do further tests if you choose to investigate further. Kind regards, Alvin > > -Saravana > > > +++ b/drivers/net/dsa/realtek-smi-core.c > @@ -454,14 +454,16 @@ static int realtek_smi_probe(struct platform_device *pdev) > smi->ds->priv = smi; > > smi->ds->ops = var->ds_ops; > - ret = dsa_register_switch(smi->ds); > - if (ret) { > - dev_err(dev, "unable to register switch ret = %d\n", ret); > - return ret; > - } > return 0; > } > > +static void realtek_smi_sync_state(struct device *dev) > +{ > + struct realtek_smi *smi = dev_get_drvdata(dev); > + if (dsa_register_switch(smi->ds)) > + dev_err(dev, "unable to register switch ret = %d\n", ret); > +} > + > static int realtek_smi_remove(struct platform_device *pdev) > { > struct realtek_smi *smi = dev_get_drvdata(&pdev->dev); > @@ -492,6 +494,7 @@ static struct platform_driver realtek_smi_driver = { > .driver = { > .name = "realtek-smi", > .of_match_table = of_match_ptr(realtek_smi_of_match), > + .sync_state = realtek_smi_sync_state, > }, > .probe = realtek_smi_probe, > .remove = realtek_smi_remove, >
On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > Hi Saravana, > > Thanks for the follow-up. I tested your change and it does the trick: > there is no deferral and the PHY driver gets probed first-try during the > mdiobus registration during the call to dsa_register_switch(). I'm fairly certain the mdiobus registration happens before dsa_register_switch(). It's in the probe call path of the DSA. The connecting of the PHYs with the DSA is what happens when dsa_register_switch() is called. > I tested > with the switch, PHY, and tagging drivers all builtin, or all modules, > and it worked in both cases. > > On 8/20/21 6:52 PM, Saravana Kannan wrote: > > Hi Alvin, > > > > Can you give this a shot to see if it fixes your issue? It basically > > delays the registration of dsa_register_switch() until all the > > consumers of this switch have probed. So it has a couple of caveats: > > Hm, weren't the only consumers the PHYs themselves? It seems like the > main effect of your change is that - by doing the actual > dsa_register_switch() call after the switch driver probe - the > ethernet-switch (provider) is already probed, thereby allowing the PHY > (consumer) to probe immediately. Correct-ish -- if you modify this to account for what I said above. > > > 1. I'm hoping the PHYs are the only consumers of this switch. > > In my case that is true, if you count the mdio_bus as well: > > /sys/devices/platform/ethernet-switch# ls -l consumer\:* > lrwxrwxrwx 1 root root 0 Aug 22 16:00 > consumer:mdio_bus:SMI-0 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 > lrwxrwxrwx 1 root root 0 Aug 22 16:00 > consumer:mdio_bus:SMI-0:00 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 > lrwxrwxrwx 1 root root 0 Aug 22 16:00 > consumer:mdio_bus:SMI-0:01 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 > lrwxrwxrwx 1 root root 0 Aug 22 16:00 > consumer:mdio_bus:SMI-0:02 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 > lrwxrwxrwx 1 root root 0 Aug 22 16:00 > consumer:mdio_bus:SMI-0:03 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 Hmm... mdio_bus being a consumer should prevent the sync_state() from being called on "ethernet-switch". What's the value of the "status" and "sync_state_only" files inside that mdio_bus folder? > > 2. All of them have to probe successfully before the switch will > > register itself. > > Yes. Right, it's a yes in your case. But will it be a yes for all instances of "realtek,rtl8366rb"? > > 3. If dsa_register_switch() fails, we can't defer the probe (because > > it already succeeded). But I'm not sure if it's a likely error code. > > It's of course possible that dsa_register_switch() fails. Assuming > fw_devlink is doing its job properly, I think the reason is most likely > going to be something specific to the driver, such as a communication > timeout with the switch hardware itself. But what if someone sets fw_devlink=permissive? Is it okay to break this? There are ways to make this work for fw_devlink=permissive and =on -- you check for each and decide where to call dsa_register_switch() based on that. > I get the impression that you don't necessarily regard this change as a > proper fix, so I'm happy to do further tests if you choose to > investigate further. I thought about this in the background the past few days. I think there are a couple of options: 1. We (community/Andrew) agree that this driver would only work with fw_devlink=on and we can confirm that the other upstream uses of "realtek,rtl8366rb" won't have any unprobed consumers problem and switch to using my patch. Benefit is that it's a trivial and quick change that gets things working again. 2. The "realtek,rtl8366rb" driver needs to be fixed to use a "component device". A component device is a logical device that represents a group of other devices. It's only initialized after all these devices have probed successfully. The actual switch should be a component device and it should call dsa_register_switch() in it's "bind" (equivalent of probe). That way you can explicitly control what devices need to be probed instead of depending on sync_state() that have a bunch of caveats. Alvin, do you want to take up (2)? -Saravana
> I thought about this in the background the past few days. I think > there are a couple of options: > 1. We (community/Andrew) agree that this driver would only work with > fw_devlink=on and we can confirm that the other upstream uses of > "realtek,rtl8366rb" won't have any unprobed consumers problem and > switch to using my patch. Benefit is that it's a trivial and quick > change that gets things working again. I don't think realtek,rtl8366rb is doing anything particularly unusual. It is not the only switch driver with an MDIO bus driver with its internal PHYs on it. > 2. The "realtek,rtl8366rb" driver needs to be fixed to use a > "component device". Again, i don't think "realtek,rtl8366rb is doing anything unusual, compared to the other DSA drivers. If you are suggesting it needs to make use of the component driver, you might also be suggesting that all the switch drivers need to be component devices. I don't fully understand the details here, but it might be, you are also suggesting some Ethernet drivers need modifying to use the component framework? And that is not going to fly. This has all worked until now, things might need a few iterations with deferral, but we get there in the end. Maybe we need to back out the phy-handle patch? It does appear to be causing regressions. Andrew
On Mon, Aug 23, 2021 at 1:44 PM Andrew Lunn <andrew@lunn.ch> wrote: > > > I thought about this in the background the past few days. I think > > there are a couple of options: > > 1. We (community/Andrew) agree that this driver would only work with > > fw_devlink=on and we can confirm that the other upstream uses of > > "realtek,rtl8366rb" won't have any unprobed consumers problem and > > switch to using my patch. Benefit is that it's a trivial and quick > > change that gets things working again. > > I don't think realtek,rtl8366rb is doing anything particularly > unusual. It is not the only switch driver with an MDIO bus driver with > its internal PHYs on it. But realtek,rtl8366rb is doing some unusual things though: 1. Expecting its child devices would get probed as soon as they are added in its own probe(). 2. The child device in turn depends on the parent node to have probed (the interrupt dependency in this case). There is absolutely no guarantee from the driver core that (1) would happen. In fact, it's more likely for this to not happen. (2) isn't fully wrong, but it's kinda weird when combined with (1) because it causes a quasi cyclic dependency. > > > 2. The "realtek,rtl8366rb" driver needs to be fixed to use a > > "component device". > > Again, i don't think "realtek,rtl8366rb is doing anything unusual, > compared to the other DSA drivers. If you are suggesting it needs to > make use of the component driver, you might also be suggesting that > all the switch drivers need to be component devices. I'm saying any driver that does both (1) and (2) above needs to start using the component devices. Otherwise, it is using a broken driver model AND it'll be caught/enforced by fw_devlink=on. Even just doing (1) is wrong, but fw_devlink=on doesn't catch/enforce it. > I don't fully > understand the details here, but it might be, you are also suggesting > some Ethernet drivers need modifying to use the component framework? I'm not sure if I'm saying that and more likely I'm NOT saying that (see further below). Based on link [1] I think it's actually the opposite of point (1) above. IIUC, in the case of FEC, you don't want the child to probe before the parent is done (because the child depends on the parent) and fw_devlink=on actually enforces that. What I'm pointing out is that since dsa_register_switch() is assuming the PHYs are ready/attaches to PHYs when it's called, if ALL the following conditions are met, a DSA switch driver needs to use a component driver model: 1. Switches that have child PHYs that dsa_register_switch() will attach to when called. 2. The child PHYs depend on services provided by the parent (eg: interrupt controller) I don't know much about dsa_register_switch(), but if the DSA framework could NOT attempt to attach to PHYs as soon as the switch is registered, that might solve the problem too without needing to use the component driver model as that'll break the cyclic dependency. > And that is not going to fly. > > This has all worked until now, things might need a few iterations with > deferral, but we get there in the end. A few deferred probes is totally fine and fw_devlink doesn't have any issues with that. In fact, fw_devlink would cut down on some unnecessary deferred probes [1]. I think you are mixing two different cases. This case is not the same as [1]. In the case of this realtek switch driver, it's saying the deferred probe of its child PHYs is NOT allowed. And I'm saying that's not a valid assumption and the component device model seems like one good way to handle this situation. > Maybe we need to back out the > phy-handle patch? It does appear to be causing regressions. I don't mind if you want to do that (fixing the issue Marek reported is easy). But that doesn't mean this realtek driver isn't wrong and needs to be fixed. It's just that fw_devlink=on is catching this more clearly. [1] - https://lore.kernel.org/netdev/CAGETcx9=AyEfjX_-adgRuX=8a0MkLnj8sy2KJGhxpNCinJu4yA@mail.gmail.com/ -Saravana
Hi Saravana, Sorry for the delayed response. On 8/23/21 8:50 PM, Saravana Kannan wrote: > On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: >> >> Hi Saravana, >> >> Thanks for the follow-up. I tested your change and it does the trick: >> there is no deferral and the PHY driver gets probed first-try during the >> mdiobus registration during the call to dsa_register_switch(). > > I'm fairly certain the mdiobus registration happens before > dsa_register_switch(). It's in the probe call path of the DSA. The > connecting of the PHYs with the DSA is what happens when > dsa_register_switch() is called. Hm, are you sure about this? My understanding is that mdiobus registration happens as follows: dsa_register_switch() -> ... -> rtl836{6,5mb}_setup() # see [1] for RFC patch with rtl8365mb -> realtek_smi_setup_mdio() -> of_mdiobus_register() As it stands, dsa_register_switch() is currently called from realtek_smi_probe(). Your patch just moves this call to realtek_smi_sync_state(), but per the above, the mdiobus registration happens inside dsa_register_switch(), meaning it doesn't happen in the probe call path. Or am I missing something? I'm happy to be wrong :-) [1] https://lore.kernel.org/netdev/20210822193145.1312668-1-alvin@pqrs.dk/T/ > >> I tested >> with the switch, PHY, and tagging drivers all builtin, or all modules, >> and it worked in both cases. >> >> On 8/20/21 6:52 PM, Saravana Kannan wrote: >>> Hi Alvin, >>> >>> Can you give this a shot to see if it fixes your issue? It basically >>> delays the registration of dsa_register_switch() until all the >>> consumers of this switch have probed. So it has a couple of caveats: >> >> Hm, weren't the only consumers the PHYs themselves? It seems like the >> main effect of your change is that - by doing the actual >> dsa_register_switch() call after the switch driver probe - the >> ethernet-switch (provider) is already probed, thereby allowing the PHY >> (consumer) to probe immediately. > > Correct-ish -- if you modify this to account for what I said above. > >> >>> 1. I'm hoping the PHYs are the only consumers of this switch. >> >> In my case that is true, if you count the mdio_bus as well: >> >> /sys/devices/platform/ethernet-switch# ls -l consumer\:* >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >> consumer:mdio_bus:SMI-0 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >> consumer:mdio_bus:SMI-0:00 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >> consumer:mdio_bus:SMI-0:01 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >> consumer:mdio_bus:SMI-0:02 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >> consumer:mdio_bus:SMI-0:03 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 > > Hmm... mdio_bus being a consumer should prevent the sync_state() from > being called on "ethernet-switch". What's the value of the "status" > and "sync_state_only" files inside that mdio_bus folder? Without your patch: /sys/devices/platform/ethernet-switch# ls -l consumer\:* lrwxrwxrwx 1 root root 0 Aug 25 13:42 consumer:mdio_bus:SMI-0 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 /sys/devices/platform/ethernet-switch# cat consumer\:*/status available /sys/devices/platform/ethernet-switch# cat consumer\:*/sync_state_only 1 With your patch: 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# ls -l consumer\:* lrwxrwxrwx 1 root root 0 Aug 25 15:03 consumer:mdio_bus:SMI-0 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 lrwxrwxrwx 1 root root 0 Aug 25 15:03 consumer:mdio_bus:SMI-0:00 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 lrwxrwxrwx 1 root root 0 Aug 25 15:03 consumer:mdio_bus:SMI-0:01 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 lrwxrwxrwx 1 root root 0 Aug 25 15:03 consumer:mdio_bus:SMI-0:02 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 lrwxrwxrwx 1 root root 0 Aug 25 15:03 consumer:mdio_bus:SMI-0:03 -> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat consumer\:*/status available active active active active 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat consumer\:*/sync_state_only 1 0 0 0 0 Hope that helps you understand what's going on better. BTW, I noticed that when I build realtek-smi as a module with your patch, my kernel crashes if I unload the module. I haven't debugged this because it was just a test, nor did I get a stacktrace. LMK if you want more info. > >>> 2. All of them have to probe successfully before the switch will >>> register itself. >> >> Yes. > > Right, it's a yes in your case. But will it be a yes for all instances > of "realtek,rtl8366rb"? > >>> 3. If dsa_register_switch() fails, we can't defer the probe (because >>> it already succeeded). But I'm not sure if it's a likely error code. >> >> It's of course possible that dsa_register_switch() fails. Assuming >> fw_devlink is doing its job properly, I think the reason is most likely >> going to be something specific to the driver, such as a communication >> timeout with the switch hardware itself. > > But what if someone sets fw_devlink=permissive? Is it okay to break > this? There are ways to make this work for fw_devlink=permissive and > =on -- you check for each and decide where to call > dsa_register_switch() based on that. I am new to DSA myself so I think I am unqualified to answer whether it's OK to break things or not. > >> I get the impression that you don't necessarily regard this change as a >> proper fix, so I'm happy to do further tests if you choose to >> investigate further. > > I thought about this in the background the past few days. I think > there are a couple of options: > 1. We (community/Andrew) agree that this driver would only work with > fw_devlink=on and we can confirm that the other upstream uses of > "realtek,rtl8366rb" won't have any unprobed consumers problem and > switch to using my patch. Benefit is that it's a trivial and quick > change that gets things working again. > 2. The "realtek,rtl8366rb" driver needs to be fixed to use a > "component device". A component device is a logical device that > represents a group of other devices. It's only initialized after all > these devices have probed successfully. The actual switch should be a > component device and it should call dsa_register_switch() in it's > "bind" (equivalent of probe). That way you can explicitly control what > devices need to be probed instead of depending on sync_state() that > have a bunch of caveats. > > Alvin, do you want to take up (2)? I can give it a shot, but first: - It seems Andrew may also need some convincing that this is the right approach. - Are you sure that this will solve the problem? See what I wrote upstairs in this email. - I have never written - nor can I recall reading - a component driver, so I would appreciate if you could point me to an upstream example that you think is illustrative. There's one more issue: I do not have an RTL8366 to test on - rather, I encountered this problem in realtek-smi while writing a new subdriver for it to support the RTL8365MB. So any proposed fix may be perceived as speculative if I cannot test on the '66 hardware. In that case this may have to wait until the '65MB subdriver is accepted. Many ifs and maybes, but don't take it to mean I'm not interested in helping. On the contrary - I would like to fix this bug since I am affected by it! Kind regards, Alvin > > -Saravana >
On Wed, Aug 25, 2021 at 6:40 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > Hi Saravana, > > Sorry for the delayed response. > > On 8/23/21 8:50 PM, Saravana Kannan wrote: > > On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > >> > >> Hi Saravana, > >> > >> Thanks for the follow-up. I tested your change and it does the trick: > >> there is no deferral and the PHY driver gets probed first-try during the > >> mdiobus registration during the call to dsa_register_switch(). > > > > I'm fairly certain the mdiobus registration happens before > > dsa_register_switch(). It's in the probe call path of the DSA. The > > connecting of the PHYs with the DSA is what happens when > > dsa_register_switch() is called. > > Hm, are you sure about this? My understanding is that mdiobus > registration happens as follows: > > dsa_register_switch() > -> ... > -> rtl836{6,5mb}_setup() # see [1] for RFC patch with rtl8365mb > -> realtek_smi_setup_mdio() > -> of_mdiobus_register() My bad, you are definitely right! Thanks for correcting my understanding. > As it stands, dsa_register_switch() is currently called from > realtek_smi_probe(). Your patch just moves this call to > realtek_smi_sync_state(), but per the above, the mdiobus registration > happens inside dsa_register_switch(), meaning it doesn't happen in the > probe call path. Or am I missing something? I'm happy to be wrong :-) Ok, my sync_state() hack is definitely not a solution anymore. It's just a terrible hack. > > [1] https://lore.kernel.org/netdev/20210822193145.1312668-1-alvin@pqrs.dk/T/ > > > > >> I tested > >> with the switch, PHY, and tagging drivers all builtin, or all modules, > >> and it worked in both cases. > >> > >> On 8/20/21 6:52 PM, Saravana Kannan wrote: > >>> Hi Alvin, > >>> > >>> Can you give this a shot to see if it fixes your issue? It basically > >>> delays the registration of dsa_register_switch() until all the > >>> consumers of this switch have probed. So it has a couple of caveats: > >> > >> Hm, weren't the only consumers the PHYs themselves? It seems like the > >> main effect of your change is that - by doing the actual > >> dsa_register_switch() call after the switch driver probe - the > >> ethernet-switch (provider) is already probed, thereby allowing the PHY > >> (consumer) to probe immediately. > > > > Correct-ish -- if you modify this to account for what I said above. > > > >> > >>> 1. I'm hoping the PHYs are the only consumers of this switch. > >> > >> In my case that is true, if you count the mdio_bus as well: > >> > >> /sys/devices/platform/ethernet-switch# ls -l consumer\:* > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > >> consumer:mdio_bus:SMI-0 -> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > >> consumer:mdio_bus:SMI-0:00 -> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > >> consumer:mdio_bus:SMI-0:01 -> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > >> consumer:mdio_bus:SMI-0:02 -> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > >> consumer:mdio_bus:SMI-0:03 -> > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 > > > > Hmm... mdio_bus being a consumer should prevent the sync_state() from > > being called on "ethernet-switch". What's the value of the "status" > > and "sync_state_only" files inside that mdio_bus folder? > > Without your patch: > > /sys/devices/platform/ethernet-switch# ls -l consumer\:* > lrwxrwxrwx 1 root root 0 Aug 25 13:42 > consumer:mdio_bus:SMI-0 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 > /sys/devices/platform/ethernet-switch# cat consumer\:*/status > available > /sys/devices/platform/ethernet-switch# cat consumer\:*/sync_state_only > 1 > > > With your patch: > > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# ls -l consumer\:* > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > consumer:mdio_bus:SMI-0 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > consumer:mdio_bus:SMI-0:00 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > consumer:mdio_bus:SMI-0:01 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > consumer:mdio_bus:SMI-0:02 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > consumer:mdio_bus:SMI-0:03 -> > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat > consumer\:*/status > available > active > active > active > active > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat > consumer\:*/sync_state_only > 1 > 0 > 0 > 0 > 0 > > Hope that helps you understand what's going on better. > > BTW, I noticed that when I build realtek-smi as a module with your > patch, my kernel crashes if I unload the module. I haven't debugged this > because it was just a test, nor did I get a stacktrace. LMK if you want > more info. Yeah, don't bother. It should never merge. > > > >>> 2. All of them have to probe successfully before the switch will > >>> register itself. > >> > >> Yes. > > > > Right, it's a yes in your case. But will it be a yes for all instances > > of "realtek,rtl8366rb"? > > > >>> 3. If dsa_register_switch() fails, we can't defer the probe (because > >>> it already succeeded). But I'm not sure if it's a likely error code. > >> > >> It's of course possible that dsa_register_switch() fails. Assuming > >> fw_devlink is doing its job properly, I think the reason is most likely > >> going to be something specific to the driver, such as a communication > >> timeout with the switch hardware itself. > > > > But what if someone sets fw_devlink=permissive? Is it okay to break > > this? There are ways to make this work for fw_devlink=permissive and > > =on -- you check for each and decide where to call > > dsa_register_switch() based on that. > > I am new to DSA myself so I think I am unqualified to answer whether > it's OK to break things or not. > > > > >> I get the impression that you don't necessarily regard this change as a > >> proper fix, so I'm happy to do further tests if you choose to > >> investigate further. > > > > I thought about this in the background the past few days. I think > > there are a couple of options: > > 1. We (community/Andrew) agree that this driver would only work with > > fw_devlink=on and we can confirm that the other upstream uses of > > "realtek,rtl8366rb" won't have any unprobed consumers problem and > > switch to using my patch. Benefit is that it's a trivial and quick > > change that gets things working again. > > 2. The "realtek,rtl8366rb" driver needs to be fixed to use a > > "component device". A component device is a logical device that > > represents a group of other devices. It's only initialized after all > > these devices have probed successfully. The actual switch should be a > > component device and it should call dsa_register_switch() in it's > > "bind" (equivalent of probe). That way you can explicitly control what > > devices need to be probed instead of depending on sync_state() that > > have a bunch of caveats. > > > > Alvin, do you want to take up (2)? > > I can give it a shot, but first: > > - It seems Andrew may also need some convincing that this is the > right approach. Agreed. Let's wait to see what Andrew thinks of my last response to him. > - Are you sure that this will solve the problem? See what I wrote > upstairs in this email. Yeah, it would solve the problem with a few changes: 1. The IRQ registration and mdio bus registration would get moved to realtek_smi_probe() which probes "realtek,rtl8366rb". 2. The component device needs to be set up to be "made up of" realtek,rtl8366rb and all the PHYs. So it'll wait for all of them to finish probing before it's initialized. PHYs will initialize now because realtek,rtl8366rb probe would finish without problems. 3. The component device init would call dsa_register_switch() which kinda makes sense because the rtl8366rb and all the PHYs combined together is what makes up the logical DSA switch. If (2) and (3) could be make part of the framework itself, with (1) like fix ups done to drivers with have these cyclic dependency issues, then we could fix bad driver usage model (assuming device_add() will probe the device before it returns). > - I have never written - nor can I recall reading - a component > driver, so I would appreciate if you could point me to an upstream > example that you think is illustrative. The APIs you'll end up using are those in drivers/base/component.c. You can also grep for component_master_ops and component_ops. drivers/gpu/drm/msm/msm_drv.c might be one place to start. Btw, the component API itself could afford some clean up and there was a series [1], but it looks like Stephen has been busy and hasn't gotten around to it? [1] - https://lore.kernel.org/lkml/20210520002519.3538432-1-swboyd@chromium.org/ Having said all that, I might be able to give an API to tell fw_devlink to ignore a specific child node as a supplier. So you'd call it to ask fw_devlink to ignore the interrupt controller as a supplier. I still maintain the current model of this driver is definitely broken, but I'd rather just unblock this right now that revert phy-handle support because of a few whacky corner cases like this. I'll try to send those out this week. > There's one more issue: I do not have an RTL8366 to test on - rather, I > encountered this problem in realtek-smi while writing a new subdriver > for it to support the RTL8365MB. So any proposed fix may be perceived as > speculative if I cannot test on the '66 hardware. If you know it fixes the issue on a very similar downstream hardware/driver, I think it's okay to send fixes. And someone else might be able to test it out for you. > In that case this may > have to wait until the '65MB subdriver is accepted. Many ifs and maybes, > but don't take it to mean I'm not interested in helping. On the contrary > - I would like to fix this bug since I am affected by it! Thanks. -Saravana
On Wed, Aug 25, 2021 at 10:33 PM Saravana Kannan <saravanak@google.com> wrote: > > On Wed, Aug 25, 2021 at 6:40 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > > > Hi Saravana, > > > > Sorry for the delayed response. > > > > On 8/23/21 8:50 PM, Saravana Kannan wrote: > > > On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: > > >> > > >> Hi Saravana, > > >> > > >> Thanks for the follow-up. I tested your change and it does the trick: > > >> there is no deferral and the PHY driver gets probed first-try during the > > >> mdiobus registration during the call to dsa_register_switch(). > > > > > > I'm fairly certain the mdiobus registration happens before > > > dsa_register_switch(). It's in the probe call path of the DSA. The > > > connecting of the PHYs with the DSA is what happens when > > > dsa_register_switch() is called. > > > > Hm, are you sure about this? My understanding is that mdiobus > > registration happens as follows: > > > > dsa_register_switch() > > -> ... > > -> rtl836{6,5mb}_setup() # see [1] for RFC patch with rtl8365mb > > -> realtek_smi_setup_mdio() > > -> of_mdiobus_register() > > My bad, you are definitely right! Thanks for correcting my understanding. > > > As it stands, dsa_register_switch() is currently called from > > realtek_smi_probe(). Your patch just moves this call to > > realtek_smi_sync_state(), but per the above, the mdiobus registration > > happens inside dsa_register_switch(), meaning it doesn't happen in the > > probe call path. Or am I missing something? I'm happy to be wrong :-) > > Ok, my sync_state() hack is definitely not a solution anymore. It's > just a terrible hack. > > > > > [1] https://lore.kernel.org/netdev/20210822193145.1312668-1-alvin@pqrs.dk/T/ > > > > > > > >> I tested > > >> with the switch, PHY, and tagging drivers all builtin, or all modules, > > >> and it worked in both cases. > > >> > > >> On 8/20/21 6:52 PM, Saravana Kannan wrote: > > >>> Hi Alvin, > > >>> > > >>> Can you give this a shot to see if it fixes your issue? It basically > > >>> delays the registration of dsa_register_switch() until all the > > >>> consumers of this switch have probed. So it has a couple of caveats: > > >> > > >> Hm, weren't the only consumers the PHYs themselves? It seems like the > > >> main effect of your change is that - by doing the actual > > >> dsa_register_switch() call after the switch driver probe - the > > >> ethernet-switch (provider) is already probed, thereby allowing the PHY > > >> (consumer) to probe immediately. > > > > > > Correct-ish -- if you modify this to account for what I said above. > > > > > >> > > >>> 1. I'm hoping the PHYs are the only consumers of this switch. > > >> > > >> In my case that is true, if you count the mdio_bus as well: > > >> > > >> /sys/devices/platform/ethernet-switch# ls -l consumer\:* > > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > > >> consumer:mdio_bus:SMI-0 -> > > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 > > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > > >> consumer:mdio_bus:SMI-0:00 -> > > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 > > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > > >> consumer:mdio_bus:SMI-0:01 -> > > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 > > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > > >> consumer:mdio_bus:SMI-0:02 -> > > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 > > >> lrwxrwxrwx 1 root root 0 Aug 22 16:00 > > >> consumer:mdio_bus:SMI-0:03 -> > > >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 > > > > > > Hmm... mdio_bus being a consumer should prevent the sync_state() from > > > being called on "ethernet-switch". What's the value of the "status" > > > and "sync_state_only" files inside that mdio_bus folder? > > > > Without your patch: > > > > /sys/devices/platform/ethernet-switch# ls -l consumer\:* > > lrwxrwxrwx 1 root root 0 Aug 25 13:42 > > consumer:mdio_bus:SMI-0 -> > > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 > > /sys/devices/platform/ethernet-switch# cat consumer\:*/status > > available > > /sys/devices/platform/ethernet-switch# cat consumer\:*/sync_state_only > > 1 > > > > > > With your patch: > > > > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# ls -l consumer\:* > > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > > consumer:mdio_bus:SMI-0 -> > > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 > > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > > consumer:mdio_bus:SMI-0:00 -> > > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 > > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > > consumer:mdio_bus:SMI-0:01 -> > > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 > > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > > consumer:mdio_bus:SMI-0:02 -> > > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 > > lrwxrwxrwx 1 root root 0 Aug 25 15:03 > > consumer:mdio_bus:SMI-0:03 -> > > ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 > > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# > > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# > > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat > > consumer\:*/status > > available > > active > > active > > active > > active > > 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat > > consumer\:*/sync_state_only > > 1 > > 0 > > 0 > > 0 > > 0 > > > > Hope that helps you understand what's going on better. > > > > BTW, I noticed that when I build realtek-smi as a module with your > > patch, my kernel crashes if I unload the module. I haven't debugged this > > because it was just a test, nor did I get a stacktrace. LMK if you want > > more info. > > Yeah, don't bother. It should never merge. > > > > > > >>> 2. All of them have to probe successfully before the switch will > > >>> register itself. > > >> > > >> Yes. > > > > > > Right, it's a yes in your case. But will it be a yes for all instances > > > of "realtek,rtl8366rb"? > > > > > >>> 3. If dsa_register_switch() fails, we can't defer the probe (because > > >>> it already succeeded). But I'm not sure if it's a likely error code. > > >> > > >> It's of course possible that dsa_register_switch() fails. Assuming > > >> fw_devlink is doing its job properly, I think the reason is most likely > > >> going to be something specific to the driver, such as a communication > > >> timeout with the switch hardware itself. > > > > > > But what if someone sets fw_devlink=permissive? Is it okay to break > > > this? There are ways to make this work for fw_devlink=permissive and > > > =on -- you check for each and decide where to call > > > dsa_register_switch() based on that. > > > > I am new to DSA myself so I think I am unqualified to answer whether > > it's OK to break things or not. > > > > > > > >> I get the impression that you don't necessarily regard this change as a > > >> proper fix, so I'm happy to do further tests if you choose to > > >> investigate further. > > > > > > I thought about this in the background the past few days. I think > > > there are a couple of options: > > > 1. We (community/Andrew) agree that this driver would only work with > > > fw_devlink=on and we can confirm that the other upstream uses of > > > "realtek,rtl8366rb" won't have any unprobed consumers problem and > > > switch to using my patch. Benefit is that it's a trivial and quick > > > change that gets things working again. > > > 2. The "realtek,rtl8366rb" driver needs to be fixed to use a > > > "component device". A component device is a logical device that > > > represents a group of other devices. It's only initialized after all > > > these devices have probed successfully. The actual switch should be a > > > component device and it should call dsa_register_switch() in it's > > > "bind" (equivalent of probe). That way you can explicitly control what > > > devices need to be probed instead of depending on sync_state() that > > > have a bunch of caveats. > > > > > > Alvin, do you want to take up (2)? > > > > I can give it a shot, but first: > > > > - It seems Andrew may also need some convincing that this is the > > right approach. > > Agreed. Let's wait to see what Andrew thinks of my last response to him. > > > - Are you sure that this will solve the problem? See what I wrote > > upstairs in this email. > > Yeah, it would solve the problem with a few changes: > 1. The IRQ registration and mdio bus registration would get moved to > realtek_smi_probe() which probes "realtek,rtl8366rb". > 2. The component device needs to be set up to be "made up of" > realtek,rtl8366rb and all the PHYs. So it'll wait for all of them to > finish probing before it's initialized. PHYs will initialize now > because realtek,rtl8366rb probe would finish without problems. > 3. The component device init would call dsa_register_switch() which > kinda makes sense because the rtl8366rb and all the PHYs combined > together is what makes up the logical DSA switch. > > If (2) and (3) could be make part of the framework itself, with (1) > like fix ups done to drivers with have these cyclic dependency issues, > then we could fix bad driver usage model (assuming device_add() will > probe the device before it returns). > > > - I have never written - nor can I recall reading - a component > > driver, so I would appreciate if you could point me to an upstream > > example that you think is illustrative. > > The APIs you'll end up using are those in drivers/base/component.c. > You can also grep for component_master_ops and component_ops. > drivers/gpu/drm/msm/msm_drv.c might be one place to start. > > Btw, the component API itself could afford some clean up and there was > a series [1], but it looks like Stephen has been busy and hasn't > gotten around to it? > [1] - https://lore.kernel.org/lkml/20210520002519.3538432-1-swboyd@chromium.org/ > > Having said all that, I might be able to give an API to tell > fw_devlink to ignore a specific child node as a supplier. So you'd > call it to ask fw_devlink to ignore the interrupt controller as a > supplier. I still maintain the current model of this driver is > definitely broken, but I'd rather just unblock this right now that > revert phy-handle support because of a few whacky corner cases like > this. I'll try to send those out this week. Not exactly what I described, but here's an attempt. Can you give it a shot? https://lore.kernel.org/lkml/20210826074526.825517-1-saravanak@google.com/ -Saravana > > > There's one more issue: I do not have an RTL8366 to test on - rather, I > > encountered this problem in realtek-smi while writing a new subdriver > > for it to support the RTL8365MB. So any proposed fix may be perceived as > > speculative if I cannot test on the '66 hardware. > > If you know it fixes the issue on a very similar downstream > hardware/driver, I think it's okay to send fixes. And someone else > might be able to test it out for you. > > > In that case this may > > have to wait until the '65MB subdriver is accepted. Many ifs and maybes, > > but don't take it to mean I'm not interested in helping. On the contrary > > - I would like to fix this bug since I am affected by it! > > Thanks. > > -Saravana
On 8/26/21 7:33 AM, Saravana Kannan wrote: > On Wed, Aug 25, 2021 at 6:40 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: >> >> Hi Saravana, >> >> Sorry for the delayed response. >> >> On 8/23/21 8:50 PM, Saravana Kannan wrote: >>> On Sun, Aug 22, 2021 at 7:19 AM Alvin Šipraga <ALSI@bang-olufsen.dk> wrote: >>>> >>>> Hi Saravana, >>>> >>>> Thanks for the follow-up. I tested your change and it does the trick: >>>> there is no deferral and the PHY driver gets probed first-try during the >>>> mdiobus registration during the call to dsa_register_switch(). >>> >>> I'm fairly certain the mdiobus registration happens before >>> dsa_register_switch(). It's in the probe call path of the DSA. The >>> connecting of the PHYs with the DSA is what happens when >>> dsa_register_switch() is called. >> >> Hm, are you sure about this? My understanding is that mdiobus >> registration happens as follows: >> >> dsa_register_switch() >> -> ... >> -> rtl836{6,5mb}_setup() # see [1] for RFC patch with rtl8365mb >> -> realtek_smi_setup_mdio() >> -> of_mdiobus_register() > > My bad, you are definitely right! Thanks for correcting my understanding. > >> As it stands, dsa_register_switch() is currently called from >> realtek_smi_probe(). Your patch just moves this call to >> realtek_smi_sync_state(), but per the above, the mdiobus registration >> happens inside dsa_register_switch(), meaning it doesn't happen in the >> probe call path. Or am I missing something? I'm happy to be wrong :-) > > Ok, my sync_state() hack is definitely not a solution anymore. It's > just a terrible hack. > >> >> [1] https://lore.kernel.org/netdev/20210822193145.1312668-1-alvin@pqrs.dk/T/>>> >>> >>>> I tested >>>> with the switch, PHY, and tagging drivers all builtin, or all modules, >>>> and it worked in both cases. >>>> >>>> On 8/20/21 6:52 PM, Saravana Kannan wrote: >>>>> Hi Alvin, >>>>> >>>>> Can you give this a shot to see if it fixes your issue? It basically >>>>> delays the registration of dsa_register_switch() until all the >>>>> consumers of this switch have probed. So it has a couple of caveats: >>>> >>>> Hm, weren't the only consumers the PHYs themselves? It seems like the >>>> main effect of your change is that - by doing the actual >>>> dsa_register_switch() call after the switch driver probe - the >>>> ethernet-switch (provider) is already probed, thereby allowing the PHY >>>> (consumer) to probe immediately. >>> >>> Correct-ish -- if you modify this to account for what I said above. >>> >>>> >>>>> 1. I'm hoping the PHYs are the only consumers of this switch. >>>> >>>> In my case that is true, if you count the mdio_bus as well: >>>> >>>> /sys/devices/platform/ethernet-switch# ls -l consumer\:* >>>> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >>>> consumer:mdio_bus:SMI-0 -> >>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 >>>> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >>>> consumer:mdio_bus:SMI-0:00 -> >>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 >>>> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >>>> consumer:mdio_bus:SMI-0:01 -> >>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 >>>> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >>>> consumer:mdio_bus:SMI-0:02 -> >>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 >>>> lrwxrwxrwx 1 root root 0 Aug 22 16:00 >>>> consumer:mdio_bus:SMI-0:03 -> >>>> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 >>> >>> Hmm... mdio_bus being a consumer should prevent the sync_state() from >>> being called on "ethernet-switch". What's the value of the "status" >>> and "sync_state_only" files inside that mdio_bus folder? >> >> Without your patch: >> >> /sys/devices/platform/ethernet-switch# ls -l consumer\:* >> lrwxrwxrwx 1 root root 0 Aug 25 13:42 >> consumer:mdio_bus:SMI-0 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 >> /sys/devices/platform/ethernet-switch# cat consumer\:*/status >> available >> /sys/devices/platform/ethernet-switch# cat consumer\:*/sync_state_only >> 1 >> >> >> With your patch: >> >> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# ls -l consumer\:* >> lrwxrwxrwx 1 root root 0 Aug 25 15:03 >> consumer:mdio_bus:SMI-0 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0 >> lrwxrwxrwx 1 root root 0 Aug 25 15:03 >> consumer:mdio_bus:SMI-0:00 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:00 >> lrwxrwxrwx 1 root root 0 Aug 25 15:03 >> consumer:mdio_bus:SMI-0:01 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:01 >> lrwxrwxrwx 1 root root 0 Aug 25 15:03 >> consumer:mdio_bus:SMI-0:02 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:02 >> lrwxrwxrwx 1 root root 0 Aug 25 15:03 >> consumer:mdio_bus:SMI-0:03 -> >> ../../virtual/devlink/platform:ethernet-switch--mdio_bus:SMI-0:03 >> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# >> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# >> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat >> consumer\:*/status >> available >> active >> active >> active >> active >> 0.0.0.0@CA44-:/sys/devices/platform/ethernet-switch# cat >> consumer\:*/sync_state_only >> 1 >> 0 >> 0 >> 0 >> 0 >> >> Hope that helps you understand what's going on better. >> >> BTW, I noticed that when I build realtek-smi as a module with your >> patch, my kernel crashes if I unload the module. I haven't debugged this >> because it was just a test, nor did I get a stacktrace. LMK if you want >> more info. > > Yeah, don't bother. It should never merge. > >>> >>>>> 2. All of them have to probe successfully before the switch will >>>>> register itself. >>>> >>>> Yes. >>> >>> Right, it's a yes in your case. But will it be a yes for all instances >>> of "realtek,rtl8366rb"? >>> >>>>> 3. If dsa_register_switch() fails, we can't defer the probe (because >>>>> it already succeeded). But I'm not sure if it's a likely error code. >>>> >>>> It's of course possible that dsa_register_switch() fails. Assuming >>>> fw_devlink is doing its job properly, I think the reason is most likely >>>> going to be something specific to the driver, such as a communication >>>> timeout with the switch hardware itself. >>> >>> But what if someone sets fw_devlink=permissive? Is it okay to break >>> this? There are ways to make this work for fw_devlink=permissive and >>> =on -- you check for each and decide where to call >>> dsa_register_switch() based on that. >> >> I am new to DSA myself so I think I am unqualified to answer whether >> it's OK to break things or not. >> >>> >>>> I get the impression that you don't necessarily regard this change as a >>>> proper fix, so I'm happy to do further tests if you choose to >>>> investigate further. >>> >>> I thought about this in the background the past few days. I think >>> there are a couple of options: >>> 1. We (community/Andrew) agree that this driver would only work with >>> fw_devlink=on and we can confirm that the other upstream uses of >>> "realtek,rtl8366rb" won't have any unprobed consumers problem and >>> switch to using my patch. Benefit is that it's a trivial and quick >>> change that gets things working again. >>> 2. The "realtek,rtl8366rb" driver needs to be fixed to use a >>> "component device". A component device is a logical device that >>> represents a group of other devices. It's only initialized after all >>> these devices have probed successfully. The actual switch should be a >>> component device and it should call dsa_register_switch() in it's >>> "bind" (equivalent of probe). That way you can explicitly control what >>> devices need to be probed instead of depending on sync_state() that >>> have a bunch of caveats. >>> >>> Alvin, do you want to take up (2)? >> >> I can give it a shot, but first: >> >> - It seems Andrew may also need some convincing that this is the >> right approach. > > Agreed. Let's wait to see what Andrew thinks of my last response to him. > >> - Are you sure that this will solve the problem? See what I wrote >> upstairs in this email. > > Yeah, it would solve the problem with a few changes: > 1. The IRQ registration and mdio bus registration would get moved to > realtek_smi_probe() which probes "realtek,rtl8366rb". > 2. The component device needs to be set up to be "made up of" > realtek,rtl8366rb and all the PHYs. So it'll wait for all of them to > finish probing before it's initialized. PHYs will initialize now > because realtek,rtl8366rb probe would finish without problems. > 3. The component device init would call dsa_register_switch() which > kinda makes sense because the rtl8366rb and all the PHYs combined > together is what makes up the logical DSA switch. > > If (2) and (3) could be make part of the framework itself, with (1) > like fix ups done to drivers with have these cyclic dependency issues, > then we could fix bad driver usage model (assuming device_add() will > probe the device before it returns). > >> - I have never written - nor can I recall reading - a component >> driver, so I would appreciate if you could point me to an upstream >> example that you think is illustrative. > > The APIs you'll end up using are those in drivers/base/component.c. > You can also grep for component_master_ops and component_ops. > drivers/gpu/drm/msm/msm_drv.c might be one place to start. Thanks for your pointers, I think your sketch makes sense to me. If we get some sort of approval from the DSA maintainers then I can give it a shot. Just one question though - it seems like the component master ought to have its own compatible string too? How is that going to work without breaking the existing dt-bindings for this switch? Kind regards, Alvin > > Btw, the component API itself could afford some clean up and there was > a series [1], but it looks like Stephen has been busy and hasn't > gotten around to it? > [1] - https://lore.kernel.org/lkml/20210520002519.3538432-1-swboyd@chromium.org/>> > Having said all that, I might be able to give an API to tell > fw_devlink to ignore a specific child node as a supplier. So you'd > call it to ask fw_devlink to ignore the interrupt controller as a > supplier. I still maintain the current model of this driver is > definitely broken, but I'd rather just unblock this right now that > revert phy-handle support because of a few whacky corner cases like > this. I'll try to send those out this week. > >> There's one more issue: I do not have an RTL8366 to test on - rather, I >> encountered this problem in realtek-smi while writing a new subdriver >> for it to support the RTL8365MB. So any proposed fix may be perceived as >> speculative if I cannot test on the '66 hardware. > > If you know it fixes the issue on a very similar downstream > hardware/driver, I think it's okay to send fixes. And someone else > might be able to test it out for you. > >> In that case this may >> have to wait until the '65MB subdriver is accepted. Many ifs and maybes, >> but don't take it to mean I'm not interested in helping. On the contrary >> - I would like to fix this bug since I am affected by it! > > Thanks. > > -Saravana >
diff --git a/drivers/net/dsa/sja1105/sja1105_mdio.c b/drivers/net/dsa/sja1105/sja1105_mdio.c index 19aea8fb76f6..705d3900e43a 100644 --- a/drivers/net/dsa/sja1105/sja1105_mdio.c +++ b/drivers/net/dsa/sja1105/sja1105_mdio.c @@ -284,8 +284,7 @@ static int sja1105_mdiobus_base_tx_register(struct sja1105_private *priv, struct mii_bus *bus; int rc = 0; - np = of_find_compatible_node(mdio_node, NULL, - "nxp,sja1110-base-tx-mdio"); + np = of_get_compatible_child(mdio_node, "nxp,sja1110-base-tx-mdio"); if (!np) return 0; @@ -339,8 +338,7 @@ static int sja1105_mdiobus_base_t1_register(struct sja1105_private *priv, struct mii_bus *bus; int rc = 0; - np = of_find_compatible_node(mdio_node, NULL, - "nxp,sja1110-base-t1-mdio"); + np = of_get_compatible_child(mdio_node, "nxp,sja1110-base-t1-mdio"); if (!np) return 0;
It seems that of_find_compatible_node has a weird calling convention in which it calls of_node_put() on the "from" node argument, instead of leaving that up to the caller. This comes from the fact that of_find_compatible_node with a non-NULL "from" argument it only supposed to be used as the iterator function of for_each_compatible_node(). OF iterator functions call of_node_get on the next OF node and of_node_put() on the previous one. When of_find_compatible_node calls of_node_put, it actually never expects the refcount to drop to zero, because the call is done under the atomic devtree_lock context, and when the refcount drops to zero it triggers a kobject and a sysfs file deletion, which assume blocking context. So any driver call to of_find_compatible_node is probably buggy because an unexpected of_node_put() takes place. What should be done is to use the of_get_compatible_child() function. Fixes: 5a8f09748ee7 ("net: dsa: sja1105: register the MDIO buses for 100base-T1 and 100base-TX") Link: https://lore.kernel.org/netdev/20210814010139.kzryimmp4rizlznt@skbuf/ Suggested-by: Frank Rowand <frowand.list@gmail.com> Suggested-by: Rob Herring <robh+dt@kernel.org> Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com> --- drivers/net/dsa/sja1105/sja1105_mdio.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)