[RFC] ARM: sa1100/assabet: move dmabounce hack to ohci driver

From: Arnd Bergmann <arnd@arndb.de>

The sa1111 platform is one of the two remaining users of the old Arm
specific "dmabounce" code, which is an earlier implementation of the
generic swiotlb.

Linus Walleij submitted a patch that removes dmabounce support from
the ixp4xx, and I had a look at the other user, which is the sa1111
companion chip.

Looking at how dmabounce is used, I could narrow it down to one driver
one one machine:

 - dmabounce is only initialized on assabet and pfs168, but not on
   any other sa1100 or pxa platform using sa1111.

 - pfs168 is not supported in mainline Linux.

 - only the OHCI and audio devices on sa1111 support DMA

 - There is no audio driver for this hardware

In the OHCI code, I noticed that two other platforms already have
a local bounce buffer support in the form of the "local_mem"
allocator. Specifically, TMIO and SM501 use this on a few other ARM
boards with 16KB or 128KB of local SRAM that can be accessed from the
OHCI and from the CPU.

While this is not the same problem as on sa1111, I could not find a
reason why we can't re-use the existing implementation but replace the
physical SRAM address mapping with a locally allocated DMA buffer.

There are two main downsides:

 - rather than using a dynamically sized pool, this buffer needs
   to be allocated at probe time using a fixed size. Without
   having any idea of what it should be, I picked a size of
   64KB, which is between what the other two OHCI front-ends use
   in their SRAM. If anyone has a better idea what that size
   is reasonable, this can be trivially changed.

 - Previously, only USB transfers to the second memory bank
   on Assabet needed to go through the bounce buffer, now all
   of them do, which may impact runtime performance, depending
   on what type of device is attached.

On the upside, the local_mem support uses write-combining
buffers, which should be a bit faster for transfers to the device
compared to normal uncached coherent memory as used in dmabounce.

Cc: Linus Walleij <linus.walleij@linaro.org>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Laurentiu Tudor <laurentiu.tudor@nxp.com>
Cc: Alan Stern <stern@rowland.harvard.edu>
Cc: linux-usb@vger.kernel.org
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
---
I don't have this hardware, so the patch is not tested at all.
---
 arch/arm/common/Kconfig        |  1 -
 arch/arm/common/sa1111.c       | 64 ----------------------------------
 drivers/usb/core/hcd.c         | 17 +++++++--
 drivers/usb/host/ohci-sa1111.c | 16 +++++++++
 4 files changed, 30 insertions(+), 68 deletions(-)

On Tue, Feb 1, 2022 at 4:31 PM Alan Stern <stern@rowland.harvard.edu> wrote:

> > diff --git a/drivers/usb/core/hcd.c b/drivers/usb/core/hcd.c
> > index 3c7c64ff3c0a..5f2fa46c7958 100644
> > --- a/drivers/usb/core/hcd.c
> > +++ b/drivers/usb/core/hcd.c
> > @@ -1260,7 +1260,8 @@ void usb_hcd_unlink_urb_from_ep(struct usb_hcd *hcd, struct urb *urb)
> >  EXPORT_SYMBOL_GPL(usb_hcd_unlink_urb_from_ep);
> >
> >  /*
> > - * Some usb host controllers can only perform dma using a small SRAM area.
> > + * Some usb host controllers can only perform dma using a small SRAM area,
> > + * or that have restrictions in addressable DRAM.
>
> s/that //
> s/in/on/

Fixed now.

> Otherwise the USB parts of this look okay to me.  I don't have suitable
> hardware to test either.  (I wonder if anyone is still using this
> platform...)

I assumed Russell was still using the Assabet, but his last upstream
commits for sa1100 are from 2016 (merged in 2019), so that may have
changed in the meantime.

> Acked-by: Alan Stern <stern@rowland.harvard.edu>

Thanks!

       Arnd

On Tue, Feb 01, 2022 at 05:10:38PM +0000, Robin Murphy wrote:
> On 2022-02-01 15:02, Arnd Bergmann wrote:
> > From: Arnd Bergmann <arnd@arndb.de>
> > 
> > The sa1111 platform is one of the two remaining users of the old Arm
> > specific "dmabounce" code, which is an earlier implementation of the
> > generic swiotlb.
> > 
> > Linus Walleij submitted a patch that removes dmabounce support from
> > the ixp4xx, and I had a look at the other user, which is the sa1111
> > companion chip.
> > 
> > Looking at how dmabounce is used, I could narrow it down to one driver
> > one one machine:
> > 
> >   - dmabounce is only initialized on assabet and pfs168, but not on
> >     any other sa1100 or pxa platform using sa1111.
> 
> Hmm, my reading of it was different. AFAICS it should affect all platforms
> with CONFIG_ARCH_SA1100 + CONFIG_SA1111 - the bus notifier from
> sa1111_init() will initialise dmabounce for everything where
> sa1111_configure_smc() has punched a hole in the DMA mask to handle the
> addressing erratum. sa1111_needs_bounce() looks to be a further
> consideration for platforms where DMA *additionally* cannot target an entire
> bank of memory at all.

Correct. The SA1111 companion can only access one SDRAM bank, whereas
the SA1110 SoC can address up to four SDRAM banks. On platforms where
there is only one bank of SDRAM, there is no issue. However, on the
Assabet there is one SDRAM bank, and on the Neponset daughter board
with the SA1111, there is a second bank. As explained in the commentry,
the SA1111 can be hardware-configured via resistive jumpers to access
either bank, but we only support the factory-shipped configuration,
which is bank 0 (the lowest addressable bank.)

The SA1111 also has an issue that one of its address lines doesn't
behave correctly, and depending on the SDRAM columns/rows, this
punches multiple holes in the SDRAM address space it can access,
which is what the sa1111_dma_mask[] array is about, and we end up
with every alternate megabyte of physical address space being
inaccessible.

The DMA mask, along with the logic in dmabounce (which truely uses the
DMA mask as, erm, a *mask* rather than the misnamed *limit* that it
has been) know about both of these issues.

On Tue, Feb 1, 2022 at 6:48 PM Russell King (Oracle)
<linux@armlinux.org.uk> wrote:
> On Tue, Feb 01, 2022 at 05:10:38PM +0000, Robin Murphy wrote:
> >
> > Hmm, my reading of it was different. AFAICS it should affect all platforms
> > with CONFIG_ARCH_SA1100 + CONFIG_SA1111 - the bus notifier from
> > sa1111_init() will initialise dmabounce for everything where
> > sa1111_configure_smc() has punched a hole in the DMA mask to handle the
> > addressing erratum. sa1111_needs_bounce() looks to be a further
> > consideration for platforms where DMA *additionally* cannot target an entire
> > bank of memory at all.
>
> Correct. The SA1111 companion can only access one SDRAM bank, whereas
> the SA1110 SoC can address up to four SDRAM banks. On platforms where
> there is only one bank of SDRAM, there is no issue. However, on the
> Assabet there is one SDRAM bank, and on the Neponset daughter board
> with the SA1111, there is a second bank. As explained in the commentry,
> the SA1111 can be hardware-configured via resistive jumpers to access
> either bank, but we only support the factory-shipped configuration,
> which is bank 0 (the lowest addressable bank.)

Ok, so this is the part that I think my patch gets right.

> The SA1111 also has an issue that one of its address lines doesn't
> behave correctly, and depending on the SDRAM columns/rows, this
> punches multiple holes in the SDRAM address space it can access,
> which is what the sa1111_dma_mask[] array is about, and we end up
> with every alternate megabyte of physical address space being
> inaccessible.
>
> The DMA mask, along with the logic in dmabounce (which truely uses the
> DMA mask as, erm, a *mask* rather than the misnamed *limit* that it
> has been) know about both of these issues.

while this part would not work if dma_alloc_flags() ends up getting
memory that is not accessible. At the minimum I need to drop the
machine_is_assabet() check and always allocate a safe buffer to
back hcd->local_mem regardless of the machine.

After reading through the dmabounce code again, my interpretation
is that the safe buffer it uses for bounces ultimately relies on
dma_alloc_coherent() allocating physical pages using GFP_DMA,
which in turn is sized to 1MB on the machines that need it.

If I'm not missing something else, using dmam_alloc_flags() in the
local_mem code should work with the same address restrictions, so
I hope I only need to update the changelog text plus the trivial change
below.

         Arnd

@@ -207,6 +207,14 @@ static int ohci_hcd_sa1111_probe(struct sa1111_dev *dev)
        }

        /*
+        * According to the "Intel StrongARM SA-1111 Microprocessor Companion
+        * Chip Specification Update" (June 2000), erratum #7, there is a
+        * significant bug in the SA1111 SDRAM shared memory controller.  If
+        * an access to a region of memory above 1MB relative to the bank base,
+        * it is important that address bit 10 _NOT_ be asserted. Depending
+        * on the configuration of the RAM, bit 10 may correspond to one
+        * of several different (processor-relative) address bits.
+        *
         * Section 4.6 of the "Intel StrongARM SA-1111 Development Module
         * User's Guide" mentions that jumpers R51 and R52 control the
         * target of SA-1111 DMA (either SDRAM bank 0 on Assabet, or
@@ -214,13 +222,14 @@ static int ohci_hcd_sa1111_probe(struct sa1111_dev *dev)
         * Assabet, so any address in bank 1 is necessarily invalid.
         *
         * As a workaround, use a bounce buffer in addressable memory
-        * as local_mem.
+        * as local_mem, relying on ZONE_DMA to provide an area that
+        * fits within the above constraints.
+        *
+        * SZ_64K is an estimate for what size this might need.
         */
-       if (machine_is_assabet()) {
-               ret = usb_hcd_setup_local_mem(hcd, 0, 0, SZ_64K);
-               if (ret)
-                       goto out_err1;
-       }
+       ret = usb_hcd_setup_local_mem(hcd, 0, 0, SZ_64K);
+       if (ret)
+               goto out_err1;

        if (!request_mem_region(hcd->rsrc_start, hcd->rsrc_len, hcd_name)) {
                dev_dbg(&dev->dev, "request_mem_region failed\n");

On Wed, Feb 2, 2022 at 12:10 AM Arnd Bergmann <arnd@kernel.org> wrote:
>
> while this part would not work if dma_alloc_flags() ends up getting
> memory that is not accessible. At the minimum I need to drop the
> machine_is_assabet() check and always allocate a safe buffer to
> back hcd->local_mem regardless of the machine.

I double-checked this to ensure we don't have to check for sa1100
vs pxa instead, and I found that all three sa1100 platforms with sa1111
(assabet/neponset, badge4, jornada720) enable DMA, but the one
pxa machine (lubbock) has the DMA mask set to 0 and won't
be able to use OHCI anyway.

       Arnd