| Message ID | 20201016090833.1892-2-thunder.leizhen@huawei.com |
|---|---|
| State | Accepted |
| Commit | 2013a4b684b6eb614ee5c9a3c07b0ae6f5ca96d9 |
| Headers | show |
| Series | eliminate two common errors reported by any yaml on arm64 | expand |
On 2020/10/17 3:27, Florian Fainelli wrote: > On 10/16/20 11:23 AM, Arnd Bergmann wrote: >> On Fri, Oct 16, 2020 at 6:48 PM Florian Fainelli <f.fainelli@gmail.com> wrote: >>> On 10/16/20 4:01 AM, Arnd Bergmann wrote: >>>> On Fri, Oct 16, 2020 at 11:09 AM Zhen Lei <thunder.leizhen@huawei.com> wrote: >>>>> >>>>> Suggested-by: Arnd Bergmann <arnd@arndb.de> >>>>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> >>>> >>>> Acked-by: Arnd Bergmann <arnd@arndb.de> >>>> >>>> I see that at least the 'bcd' and 'xhci' devices in fact try to >>>> use 64-bit DMA. It would be good to test this on actual >>>> hardware to ensure that it works correctly when this is enabled. >>>> >>>> Ideally avoiding the swiotlb bounce buffering should only >>>> make it faster here, but there are many chips on which >>>> 64-bit DMA is broken in some form. >>> >>> Is this change really an improvement though? This 'usb' pseudo bus node >>> could just keep being defined with #address-cells = <1> and #size-cells >>> = <1> so as to satisfy the 'reg' definition however we could just adjust >>> dma-ranges to indicate full 64-bit addressing capability. Would not that >>> work? >> >> When #address-cells is '1', you cannot specify dma-ranges that >> go beyond a 32-bit address range. > > Would not it be enough to remove the 'dma-ranges' property though? Sorry > for being slow here. Remove the 'dma-ranges' property should also work. After all, it is equivalent to the original empty dma-ranges scheme. In addition, since the IOMMU nodes are defined, it should be enabled. Therefore, Arnd's concern about the scenario where the IOMMU is disabled may not be triggered. All roads lead to Rome. Which solution should be chose depends on individual preferences. For me, either solution is fine. The third solution is to define a non-empty dma-ranges property. However, because stingray-usb.dtsi is included in multiple files, it may not be appropriate. >
Hi, everybody: How do we deal with this problem? I updated the kernel to the latest and the problem still persists. make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j24 dtbs 2>err.txt vim err.txt arch/arm64/boot/dts/qcom/ipq6018.dtsi:185.3-14: Warning (dma_ranges_format): /soc:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) arch/arm64/boot/dts/qcom/ipq6018.dtsi:185.3-14: Warning (dma_ranges_format): /soc:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) On 2020/10/26 10:21, Leizhen (ThunderTown) wrote: > > > On 2020/10/23 15:17, Arnd Bergmann wrote: >> On Sun, Oct 18, 2020 at 4:10 AM Leizhen (ThunderTown) >> <thunder.leizhen@huawei.com> wrote: >>> On 2020/10/17 3:27, Florian Fainelli wrote: >>>> On 10/16/20 11:23 AM, Arnd Bergmann wrote: >>>>> On Fri, Oct 16, 2020 at 6:48 PM Florian Fainelli <f.fainelli@gmail.com> wrote: >>>>>> On 10/16/20 4:01 AM, Arnd Bergmann wrote: >>>>>>> On Fri, Oct 16, 2020 at 11:09 AM Zhen Lei <thunder.leizhen@huawei.com> wrote: >>>>>>>> >>>>>>>> Suggested-by: Arnd Bergmann <arnd@arndb.de> >>>>>>>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> >>>>>>> >>>>>>> Acked-by: Arnd Bergmann <arnd@arndb.de> >>>>>>> >>>>>>> I see that at least the 'bcd' and 'xhci' devices in fact try to >>>>>>> use 64-bit DMA. It would be good to test this on actual >>>>>>> hardware to ensure that it works correctly when this is enabled. >>>>>>> >>>>>>> Ideally avoiding the swiotlb bounce buffering should only >>>>>>> make it faster here, but there are many chips on which >>>>>>> 64-bit DMA is broken in some form. >>>>>> >>>>>> Is this change really an improvement though? This 'usb' pseudo bus node >>>>>> could just keep being defined with #address-cells = <1> and #size-cells >>>>>> = <1> so as to satisfy the 'reg' definition however we could just adjust >>>>>> dma-ranges to indicate full 64-bit addressing capability. Would not that >>>>>> work? >>>>> >>>>> When #address-cells is '1', you cannot specify dma-ranges that >>>>> go beyond a 32-bit address range. >>>> >>>> Would not it be enough to remove the 'dma-ranges' property though? Sorry >>>> for being slow here. >>> >>> Remove the 'dma-ranges' property should also work. After all, it is equivalent >>> to the original empty dma-ranges scheme. In addition, since the IOMMU nodes are >>> defined, it should be enabled. >> >> Are you sure? I was expecting the IOMMU not to get used here since >> the devices do contain list an 'iommus' property. > > OK，If the SMMU maybe disabled, then your proposal is necessary. > >> >> Arnd >> >> . >>
On 11/8/20 10:18 PM, Leizhen (ThunderTown) wrote: > Hi, everybody: > How do we deal with this problem? I updated the kernel to the latest and the problem still persists. > > make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j24 dtbs 2>err.txt > vim err.txt > > arch/arm64/boot/dts/qcom/ipq6018.dtsi:185.3-14: Warning (dma_ranges_format): /soc:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) > arch/arm64/boot/dts/qcom/ipq6018.dtsi:185.3-14: Warning (dma_ranges_format): /soc:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) I was hoping that Ray or Scott could give this a try, and TBH, I am still not a big fan of having to mangle the 'reg' property to be compatible with #address-cells = <2> and #size-cells = <2>, I would have preferred omitting the 'dma-ranges' property entirely. We have plenty of time to get this patch applied for v5.11. -- Florian
On Mon, Nov 9, 2020 at 6:30 PM Florian Fainelli <f.fainelli@gmail.com> wrote: > > On 11/8/20 10:18 PM, Leizhen (ThunderTown) wrote: > > Hi, everybody: > > How do we deal with this problem? I updated the kernel to the latest and the problem still persists. > > > > make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j24 dtbs 2>err.txt > > vim err.txt > > > > arch/arm64/boot/dts/qcom/ipq6018.dtsi:185.3-14: Warning (dma_ranges_format): /soc:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) > > arch/arm64/boot/dts/qcom/ipq6018.dtsi:185.3-14: Warning (dma_ranges_format): /soc:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) > > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) > > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) > > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) > > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) > > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #address-cells (1) differs from / (2) > > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but its #size-cells (1) differs from / (2) > > I was hoping that Ray or Scott could give this a try, and TBH, I am > still not a big fan of having to mangle the 'reg' property to be > compatible with #address-cells = <2> and #size-cells = <2>, I would have > preferred omitting the 'dma-ranges' property entirely. > > We have plenty of time to get this patch applied for v5.11. I would much prefer to get the warning fixed for v5.10, once we know what the hardware can or cannot actually do, as it is one of the warnings that shows up in every kernelci build. Arnd
On Fri, 16 Oct 2020 17:08:32 +0800, Zhen Lei <thunder.leizhen@huawei.com> wrote: > The scripts/dtc/checks.c requires that the node have empty "dma-ranges" > property must have the same "#address-cells" and "#size-cells" values as > the parent node. Otherwise, the following warnings is reported: > > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ > (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ > its #address-cells (1) differs from / (2) > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ > (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ > its #size-cells (1) differs from / (2) > > Arnd Bergmann figured out why it's necessary: > Also note that the #address-cells=<1> means that any device under > this bus is assumed to only support 32-bit addressing, and DMA will > have to go through a slow swiotlb in the absence of an IOMMU. > > Suggested-by: Arnd Bergmann <arnd@arndb.de> > Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> > --- Applied to devicetree-arm64/next, thanks! -- Florian
On Sat, Nov 28, 2020 at 5:53 AM Florian Fainelli <f.fainelli@gmail.com> wrote: > > On Fri, 16 Oct 2020 17:08:32 +0800, Zhen Lei <thunder.leizhen@huawei.com> wrote: > > The scripts/dtc/checks.c requires that the node have empty "dma-ranges" > > property must have the same "#address-cells" and "#size-cells" values as > > the parent node. Otherwise, the following warnings is reported: > > > > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ > > (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ > > its #address-cells (1) differs from / (2) > > arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ > > (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ > > its #size-cells (1) differs from / (2) > > > > Arnd Bergmann figured out why it's necessary: > > Also note that the #address-cells=<1> means that any device under > > this bus is assumed to only support 32-bit addressing, and DMA will > > have to go through a slow swiotlb in the absence of an IOMMU. > > > > Suggested-by: Arnd Bergmann <arnd@arndb.de> > > Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> > > --- > > Applied to devicetree-arm64/next, thanks! The notification may have gone missing, but I had merged it into v5.10-fixes already, and as of today, it's in mainline, so you can drop it from your next branch, or just leave it in if you want to avoid taking things out of your tree. Arnd
+ Bharat On 11/28/2020 1:58 AM, Arnd Bergmann wrote: > On Sat, Nov 28, 2020 at 5:53 AM Florian Fainelli <f.fainelli@gmail.com> wrote: >> >> On Fri, 16 Oct 2020 17:08:32 +0800, Zhen Lei <thunder.leizhen@huawei.com> wrote: >>> The scripts/dtc/checks.c requires that the node have empty "dma-ranges" >>> property must have the same "#address-cells" and "#size-cells" values as >>> the parent node. Otherwise, the following warnings is reported: >>> >>> arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ >>> (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ >>> its #address-cells (1) differs from / (2) >>> arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ >>> (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ >>> its #size-cells (1) differs from / (2) >>> >>> Arnd Bergmann figured out why it's necessary: >>> Also note that the #address-cells=<1> means that any device under >>> this bus is assumed to only support 32-bit addressing, and DMA will >>> have to go through a slow swiotlb in the absence of an IOMMU. >>> >>> Suggested-by: Arnd Bergmann <arnd@arndb.de> >>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> >>> --- >> >> Applied to devicetree-arm64/next, thanks! > > The notification may have gone missing, but I had merged it into v5.10-fixes > already, and as of today, it's in mainline, so you can drop it from your > next branch, or just leave it in if you want to avoid taking things out of > your tree. > > Arnd > It looks like this patch might have caused a regression on Stingray USB. Bharat, could you please confirm? The fix would be to properly define the dma-ranges to be 32-bit (0x0 ~ 0xffffffff) since IOMMU is disabled on this device and the device's DMA engine is on a 32-bit bus. Thanks, Ray -- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
On Mon, Dec 14, 2020 at 8:09 PM Ray Jui <ray.jui@broadcom.com> wrote: > On 11/28/2020 1:58 AM, Arnd Bergmann wrote: > > On Sat, Nov 28, 2020 at 5:53 AM Florian Fainelli <f.fainelli@gmail.com> wrote: > >> > >> On Fri, 16 Oct 2020 17:08:32 +0800, Zhen Lei <thunder.leizhen@huawei.com> wrote: > >>> The scripts/dtc/checks.c requires that the node have empty "dma-ranges" > >>> property must have the same "#address-cells" and "#size-cells" values as > >>> the parent node. Otherwise, the following warnings is reported: > >>> > >>> arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ > >>> (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ > >>> its #address-cells (1) differs from / (2) > >>> arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ > >>> (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ > >>> its #size-cells (1) differs from / (2) > >>> > >>> Arnd Bergmann figured out why it's necessary: > >>> Also note that the #address-cells=<1> means that any device under > >>> this bus is assumed to only support 32-bit addressing, and DMA will > >>> have to go through a slow swiotlb in the absence of an IOMMU. > >>> > >>> Suggested-by: Arnd Bergmann <arnd@arndb.de> > >>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> > >>> --- > >> > >> Applied to devicetree-arm64/next, thanks! > > > > The notification may have gone missing, but I had merged it into v5.10-fixes > > already, and as of today, it's in mainline, so you can drop it from your > > next branch, or just leave it in if you want to avoid taking things out of > > your tree. > > It looks like this patch might have caused a regression on Stingray USB. > Bharat, could you please confirm? Well, this is what I had asked about originally, I assumed that Florian had asked someone with access to the datasheet. > The fix would be to properly define the dma-ranges to be 32-bit (0x0 ~ > 0xffffffff) since IOMMU is disabled on this device and the device's DMA > engine is on a 32-bit bus. That's not how dma-ranges work, they tell you what the capabilities of the bus are, while the capabilities of the device are identified by the properties of that device. The device claims to be compatible with "generic-xhci", so the driver asks for a 64-bit mask to be set according to the xhci specification. If this device is not xhci compliant, then it should not ask for a 64-bit mask. However, if this is a 64-bit capable bus master on a 32-bit bus, then the dma-ranges property should list the capabilities of the bus, so the kernel can force the driver to fall back to 32-bit addressing. Arnd
On 12/14/2020 11:46 AM, Arnd Bergmann wrote: > On Mon, Dec 14, 2020 at 8:09 PM Ray Jui <ray.jui@broadcom.com> wrote: >> On 11/28/2020 1:58 AM, Arnd Bergmann wrote: >>> On Sat, Nov 28, 2020 at 5:53 AM Florian Fainelli <f.fainelli@gmail.com> wrote: >>>> >>>> On Fri, 16 Oct 2020 17:08:32 +0800, Zhen Lei <thunder.leizhen@huawei.com> wrote: >>>>> The scripts/dtc/checks.c requires that the node have empty "dma-ranges" >>>>> property must have the same "#address-cells" and "#size-cells" values as >>>>> the parent node. Otherwise, the following warnings is reported: >>>>> >>>>> arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ >>>>> (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ >>>>> its #address-cells (1) differs from / (2) >>>>> arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ >>>>> (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ >>>>> its #size-cells (1) differs from / (2) >>>>> >>>>> Arnd Bergmann figured out why it's necessary: >>>>> Also note that the #address-cells=<1> means that any device under >>>>> this bus is assumed to only support 32-bit addressing, and DMA will >>>>> have to go through a slow swiotlb in the absence of an IOMMU. >>>>> >>>>> Suggested-by: Arnd Bergmann <arnd@arndb.de> >>>>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> >>>>> --- >>>> >>>> Applied to devicetree-arm64/next, thanks! >>> >>> The notification may have gone missing, but I had merged it into v5.10-fixes >>> already, and as of today, it's in mainline, so you can drop it from your >>> next branch, or just leave it in if you want to avoid taking things out of >>> your tree. >> >> It looks like this patch might have caused a regression on Stingray USB. >> Bharat, could you please confirm? > > Well, this is what I had asked about originally, I assumed that > Florian had asked someone with access to the datasheet. It looks like we had some bad communication here, the notification email indicating that you had applied the patch did not make it through, and I failed to make you aware that I was waiting for some testing from Scott and Ray.
On 12/15/2020 5:19 AM, Bharat Gooty wrote: > Since the IOMMU is disabled and DMA engine is on 32-bit bus, We can not > give the complete DDR for the USB DMA. > So restricting the usable DAM size to 4GB. Thanks, can you make this a proper patch submission along with a Fixes: tag that is: Fixes: 2013a4b684b6 ("arm64: dts: broadcom: clear the warnings caused by empty dma-ranges") -- Florian
On Tue, Dec 15, 2020 at 4:41 PM Florian Fainelli <f.fainelli@gmail.com> wrote: > > On 12/15/2020 5:19 AM, Bharat Gooty wrote: > > Since the IOMMU is disabled and DMA engine is on 32-bit bus, We can not > > give the complete DDR for the USB DMA. > > So restricting the usable DAM size to 4GB. > > Thanks, can you make this a proper patch submission along with a Fixes: > tag that is: > > Fixes: 2013a4b684b6 ("arm64: dts: broadcom: clear the warnings caused by > empty dma-ranges") Yes, that would be helpful, though I would appreciate a better description that explains what is actually going on: is it the device or the bus that has the 32-bit limitation, and if it is indeed a bug in the device, why do you pretend that this is a 64-bit device on a 32-bit bus instead (this could also use a comment in the dts file)? Arnd
Hi Arnd, On 2020-12-15 7:49 a.m., Arnd Bergmann wrote: > On Tue, Dec 15, 2020 at 4:41 PM Florian Fainelli <f.fainelli@gmail.com> wrote: >> >> On 12/15/2020 5:19 AM, Bharat Gooty wrote: >>> Since the IOMMU is disabled and DMA engine is on 32-bit bus, We can not >>> give the complete DDR for the USB DMA. >>> So restricting the usable DAM size to 4GB. >> >> Thanks, can you make this a proper patch submission along with a Fixes: >> tag that is: >> >> Fixes: 2013a4b684b6 ("arm64: dts: broadcom: clear the warnings caused by >> empty dma-ranges") > > Yes, that would be helpful, though I would appreciate a better description > that explains what is actually going on: is it the device or the bus that > has the 32-bit limitation, and if it is indeed a bug in the device, why do > you pretend that this is a 64-bit device on a 32-bit bus instead (this > could also use a comment in the dts file)? > > Arnd > Sorry for the delay in reply. Bharat finally got time to do some investigation to confirm the following: These USB controllers indeed can address 64-bit. However, on the bus internally, only 40-bits are connected to the interconnect of CCN. As a result, the 'dma-ranges' should be modified to address 40-bit in size. We also have a somewhat related question, is it true that since v5.10, defining of 'dma-ranges' on the bus node where its child device node has implication of IOMMU usage (through 'iommus' or 'iommu-map') is now mandatory? My understanding is that the 'dma-ranges' in this scheme will define the IOVA address to system address mapping required by all devices on that bus. Please help to confirm if my understanding is correct. Thanks, Ray
On Tue, Jan 12, 2021 at 7:28 PM Ray Jui <ray.jui@broadcom.com> wrote: > On 2020-12-15 7:49 a.m., Arnd Bergmann wrote: > > On Tue, Dec 15, 2020 at 4:41 PM Florian Fainelli <f.fainelli@gmail.com> wrote: > >> > >> On 12/15/2020 5:19 AM, Bharat Gooty wrote: > >>> Since the IOMMU is disabled and DMA engine is on 32-bit bus, We can not > >>> give the complete DDR for the USB DMA. > >>> So restricting the usable DAM size to 4GB. > >> > >> Thanks, can you make this a proper patch submission along with a Fixes: > >> tag that is: > >> > >> Fixes: 2013a4b684b6 ("arm64: dts: broadcom: clear the warnings caused by > >> empty dma-ranges") > > > > Yes, that would be helpful, though I would appreciate a better description > > that explains what is actually going on: is it the device or the bus that > > has the 32-bit limitation, and if it is indeed a bug in the device, why do > > you pretend that this is a 64-bit device on a 32-bit bus instead (this > > could also use a comment in the dts file)? > > > > Arnd > > > > Sorry for the delay in reply. Bharat finally got time to do some > investigation to confirm the following: > > These USB controllers indeed can address 64-bit. However, on the bus > internally, only 40-bits are connected to the interconnect of CCN. As a > result, the 'dma-ranges' should be modified to address 40-bit in size. a 40-bit range is effectively the same as a 64-bit range though, so I think you'll still need a driver quirk if the device cannot use the whole 40 bit addressing but is limited to 32 bits because of a bug in its dma engine. > We also have a somewhat related question, is it true that since v5.10, > defining of 'dma-ranges' on the bus node where its child device node has > implication of IOMMU usage (through 'iommus' or 'iommu-map') is now > mandatory? My understanding is that the 'dma-ranges' in this scheme will > define the IOVA address to system address mapping required by all > devices on that bus. Please help to confirm if my understanding is correct I have not actually seen that change. In general, you should always have a dma-ranges property in the parent of a DMA master, I think we just never enforced that and fall back to a 32-bit mask by default. Arnd
On 1/12/2021 12:40 PM, Arnd Bergmann wrote: > On Tue, Jan 12, 2021 at 7:28 PM Ray Jui <ray.jui@broadcom.com> wrote: >> On 2020-12-15 7:49 a.m., Arnd Bergmann wrote: >>> On Tue, Dec 15, 2020 at 4:41 PM Florian Fainelli <f.fainelli@gmail.com> wrote: >>>> >>>> On 12/15/2020 5:19 AM, Bharat Gooty wrote: >>>>> Since the IOMMU is disabled and DMA engine is on 32-bit bus, We can not >>>>> give the complete DDR for the USB DMA. >>>>> So restricting the usable DAM size to 4GB. >>>> >>>> Thanks, can you make this a proper patch submission along with a Fixes: >>>> tag that is: >>>> >>>> Fixes: 2013a4b684b6 ("arm64: dts: broadcom: clear the warnings caused by >>>> empty dma-ranges") >>> >>> Yes, that would be helpful, though I would appreciate a better description >>> that explains what is actually going on: is it the device or the bus that >>> has the 32-bit limitation, and if it is indeed a bug in the device, why do >>> you pretend that this is a 64-bit device on a 32-bit bus instead (this >>> could also use a comment in the dts file)? >>> >>> Arnd >>> >> >> Sorry for the delay in reply. Bharat finally got time to do some >> investigation to confirm the following: >> >> These USB controllers indeed can address 64-bit. However, on the bus >> internally, only 40-bits are connected to the interconnect of CCN. As a >> result, the 'dma-ranges' should be modified to address 40-bit in size. > > a 40-bit range is effectively the same as a 64-bit range though, so I > think you'll still need a driver quirk if the device cannot use the whole 40 > bit addressing but is limited to 32 bits because of a bug in its dma engine. > Bharat can correct me if I'm wrong, but I don't think we have a bug in the USB DMA engine that causes it can only address 32-bit. I believe we can set dma-ranges size to 40-bit here. The dma-range property is though required to be specified, instead of leaving it as empty, with the use of IOMMU. That seems to be a v5.10 specific behavior as I described below. Thanks, Ray >> We also have a somewhat related question, is it true that since v5.10, >> defining of 'dma-ranges' on the bus node where its child device node has >> implication of IOMMU usage (through 'iommus' or 'iommu-map') is now >> mandatory? My understanding is that the 'dma-ranges' in this scheme will >> define the IOVA address to system address mapping required by all >> devices on that bus. Please help to confirm if my understanding is correct > > I have not actually seen that change. In general, you should always > have a dma-ranges property in the parent of a DMA master, I think > we just never enforced that and fall back to a 32-bit mask by default. > > Arnd >
Hello Ray, I had cross checked with Design and integration team. Yes we can set the "dma-rages" to 40 bit DMA ranges. Tested, it is working. Thanks, -Bharat -----Original Message----- From: Ray Jui <ray.jui@broadcom.com> Sent: Wednesday, January 13, 2021 2:28 AM To: Arnd Bergmann <arnd@kernel.org> Cc: Florian Fainelli <f.fainelli@gmail.com>; Bharat Gooty <bharat.gooty@broadcom.com>; devicetree <devicetree@vger.kernel.org>; linux-kernel <linux-kernel@vger.kernel.org>; Arnd Bergmann <arnd@arndb.de>; Scott Branden <sbranden@broadcom.com>; Ray Jui <rjui@broadcom.com>; linux-arm-msm <linux-arm-msm@vger.kernel.org>; Andy Gross <agross@kernel.org>; Bjorn Andersson <bjorn.andersson@linaro.org>; Rob Herring <robh+dt@kernel.org>; bcm-kernel-feedback-list <bcm-kernel-feedback-list@broadcom.com>; Zhen Lei <thunder.leizhen@huawei.com>; linux-arm-kernel <linux-arm-kernel@lists.infradead.org> Subject: Re: [PATCH v2 1/2] arm64: dts: broadcom: clear the warnings caused by empty dma-ranges On 1/12/2021 12:40 PM, Arnd Bergmann wrote: > On Tue, Jan 12, 2021 at 7:28 PM Ray Jui <ray.jui@broadcom.com> wrote: >> On 2020-12-15 7:49 a.m., Arnd Bergmann wrote: >>> On Tue, Dec 15, 2020 at 4:41 PM Florian Fainelli <f.fainelli@gmail.com> >>> wrote: >>>> >>>> On 12/15/2020 5:19 AM, Bharat Gooty wrote: >>>>> Since the IOMMU is disabled and DMA engine is on 32-bit bus, We can >>>>> not >>>>> give the complete DDR for the USB DMA. >>>>> So restricting the usable DAM size to 4GB. >>>> >>>> Thanks, can you make this a proper patch submission along with a Fixes: >>>> tag that is: >>>> >>>> Fixes: 2013a4b684b6 ("arm64: dts: broadcom: clear the warnings caused >>>> by >>>> empty dma-ranges") >>> >>> Yes, that would be helpful, though I would appreciate a better >>> description >>> that explains what is actually going on: is it the device or the bus >>> that >>> has the 32-bit limitation, and if it is indeed a bug in the device, why >>> do >>> you pretend that this is a 64-bit device on a 32-bit bus instead (this >>> could also use a comment in the dts file)? >>> >>> Arnd >>> >> >> Sorry for the delay in reply. Bharat finally got time to do some >> investigation to confirm the following: >> >> These USB controllers indeed can address 64-bit. However, on the bus >> internally, only 40-bits are connected to the interconnect of CCN. As a >> result, the 'dma-ranges' should be modified to address 40-bit in size. > > a 40-bit range is effectively the same as a 64-bit range though, so I > think you'll still need a driver quirk if the device cannot use the whole > 40 > bit addressing but is limited to 32 bits because of a bug in its dma > engine. > Bharat can correct me if I'm wrong, but I don't think we have a bug in the USB DMA engine that causes it can only address 32-bit. I believe we can set dma-ranges size to 40-bit here. The dma-range property is though required to be specified, instead of leaving it as empty, with the use of IOMMU. That seems to be a v5.10 specific behavior as I described below. Thanks, Ray >> We also have a somewhat related question, is it true that since v5.10, >> defining of 'dma-ranges' on the bus node where its child device node has >> implication of IOMMU usage (through 'iommus' or 'iommu-map') is now >> mandatory? My understanding is that the 'dma-ranges' in this scheme will >> define the IOVA address to system address mapping required by all >> devices on that bus. Please help to confirm if my understanding is >> correct > > I have not actually seen that change. In general, you should always > have a dma-ranges property in the parent of a DMA master, I think > we just never enforced that and fall back to a 32-bit mask by default. > > Arnd >
On Wed, Jan 13, 2021 at 4:42 AM Bharat Gooty <bharat.gooty@broadcom.com> wrote: > > Hello Ray, > > I had cross checked with Design and integration team. > Yes we can set the "dma-rages" to 40 bit DMA ranges. Tested, it is working. > > -----Original Message----- > From: Ray Jui <ray.jui@broadcom.com> > > Bharat can correct me if I'm wrong, but I don't think we have a bug in > the USB DMA engine that causes it can only address 32-bit. I believe we > can set dma-ranges size to 40-bit here. > > The dma-range property is though required to be specified, instead of > leaving it as empty, with the use of IOMMU. That seems to be a v5.10 > specific behavior as I described below. Ok, thanks for double-checking. I had misremembered the version that actually went into the as the one that used 64-bit dma-ranges and thought that was what broke, rather than the version without dma-ranges. If any of you want to send me that bugfix directly, or have Florian pick it up through his fixes branch, I'll make sure we get it into v5.11. Arnd
On 1/13/21 12:05 AM, Arnd Bergmann wrote: > On Wed, Jan 13, 2021 at 4:42 AM Bharat Gooty <bharat.gooty@broadcom.com> wrote: >> >> Hello Ray, >> >> I had cross checked with Design and integration team. >> Yes we can set the "dma-rages" to 40 bit DMA ranges. Tested, it is working. >> >> -----Original Message----- >> From: Ray Jui <ray.jui@broadcom.com> >> >> Bharat can correct me if I'm wrong, but I don't think we have a bug in >> the USB DMA engine that causes it can only address 32-bit. I believe we >> can set dma-ranges size to 40-bit here. >> >> The dma-range property is though required to be specified, instead of >> leaving it as empty, with the use of IOMMU. That seems to be a v5.10 >> specific behavior as I described below. > > Ok, thanks for double-checking. I had misremembered the version > that actually went into the as the one that used 64-bit dma-ranges > and thought that was what broke, rather than the version without > dma-ranges. > > If any of you want to send me that bugfix directly, or have Florian > pick it up through his fixes branch, I'll make sure we get it into v5.11. I have another change for v5.11 that I would like to send, so please do send a bugfix when you get a chance and we can lump those two changes together, say, by the end of the week? -- Florian
Hi Bharat, On 1/13/2021 8:55 AM, Florian Fainelli wrote: > On 1/13/21 12:05 AM, Arnd Bergmann wrote: >> On Wed, Jan 13, 2021 at 4:42 AM Bharat Gooty <bharat.gooty@broadcom.com> wrote: >>> >>> Hello Ray, >>> >>> I had cross checked with Design and integration team. >>> Yes we can set the "dma-rages" to 40 bit DMA ranges. Tested, it is working. >>> >>> -----Original Message----- >>> From: Ray Jui <ray.jui@broadcom.com> >>> >>> Bharat can correct me if I'm wrong, but I don't think we have a bug in >>> the USB DMA engine that causes it can only address 32-bit. I believe we >>> can set dma-ranges size to 40-bit here. >>> >>> The dma-range property is though required to be specified, instead of >>> leaving it as empty, with the use of IOMMU. That seems to be a v5.10 >>> specific behavior as I described below. >> >> Ok, thanks for double-checking. I had misremembered the version >> that actually went into the as the one that used 64-bit dma-ranges >> and thought that was what broke, rather than the version without >> dma-ranges. >> >> If any of you want to send me that bugfix directly, or have Florian >> pick it up through his fixes branch, I'll make sure we get it into v5.11. > > I have another change for v5.11 that I would like to send, so please do > send a bugfix when you get a chance and we can lump those two changes > together, say, by the end of the week? > Would you be able to help to send this out by the end of this week as Florian advised? Your previous patch with the extension to 40-bit in size plus a fixes-by tag should do! Thanks! Ray
diff --git a/arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi b/arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi index 55259f973b5a9e4..aef8f2b00778d71 100644 --- a/arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi +++ b/arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi @@ -5,20 +5,20 @@ usb { compatible = "simple-bus"; dma-ranges; - #address-cells = <1>; - #size-cells = <1>; - ranges = <0x0 0x0 0x68500000 0x00400000>; + #address-cells = <2>; + #size-cells = <2>; + ranges = <0x0 0x0 0x0 0x68500000 0x0 0x00400000>; usbphy0: usb-phy@0 { compatible = "brcm,sr-usb-combo-phy"; - reg = <0x00000000 0x100>; + reg = <0x0 0x00000000 0x0 0x100>; #phy-cells = <1>; status = "disabled"; }; xhci0: usb@1000 { compatible = "generic-xhci"; - reg = <0x00001000 0x1000>; + reg = <0x0 0x00001000 0x0 0x1000>; interrupts = <GIC_SPI 256 IRQ_TYPE_LEVEL_HIGH>; phys = <&usbphy0 1>, <&usbphy0 0>; phy-names = "phy0", "phy1"; @@ -28,7 +28,7 @@ bdc0: usb@2000 { compatible = "brcm,bdc-v0.16"; - reg = <0x00002000 0x1000>; + reg = <0x0 0x00002000 0x0 0x1000>; interrupts = <GIC_SPI 259 IRQ_TYPE_LEVEL_HIGH>; phys = <&usbphy0 0>, <&usbphy0 1>; phy-names = "phy0", "phy1"; @@ -38,21 +38,21 @@ usbphy1: usb-phy@10000 { compatible = "brcm,sr-usb-combo-phy"; - reg = <0x00010000 0x100>; + reg = <0x0 0x00010000 0x0 0x100>; #phy-cells = <1>; status = "disabled"; }; usbphy2: usb-phy@20000 { compatible = "brcm,sr-usb-hs-phy"; - reg = <0x00020000 0x100>; + reg = <0x0 0x00020000 0x0 0x100>; #phy-cells = <0>; status = "disabled"; }; xhci1: usb@11000 { compatible = "generic-xhci"; - reg = <0x00011000 0x1000>; + reg = <0x0 0x00011000 0x0 0x1000>; interrupts = <GIC_SPI 263 IRQ_TYPE_LEVEL_HIGH>; phys = <&usbphy1 1>, <&usbphy2>, <&usbphy1 0>; phy-names = "phy0", "phy1", "phy2"; @@ -62,7 +62,7 @@ bdc1: usb@21000 { compatible = "brcm,bdc-v0.16"; - reg = <0x00021000 0x1000>; + reg = <0x0 0x00021000 0x0 0x1000>; interrupts = <GIC_SPI 266 IRQ_TYPE_LEVEL_HIGH>; phys = <&usbphy2>; phy-names = "phy0";
The scripts/dtc/checks.c requires that the node have empty "dma-ranges" property must have the same "#address-cells" and "#size-cells" values as the parent node. Otherwise, the following warnings is reported: arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ its #address-cells (1) differs from / (2) arch/arm64/boot/dts/broadcom/stingray/stingray-usb.dtsi:7.3-14: Warning \ (dma_ranges_format): /usb:dma-ranges: empty "dma-ranges" property but \ its #size-cells (1) differs from / (2) Arnd Bergmann figured out why it's necessary: Also note that the #address-cells=<1> means that any device under this bus is assumed to only support 32-bit addressing, and DMA will have to go through a slow swiotlb in the absence of an IOMMU. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com> --- .../boot/dts/broadcom/stingray/stingray-usb.dtsi | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)