diff mbox

How to specify IOMMU'able devices in DT (was: [RFC 0/5] ARM: dma-mapping: New dma_map_ops to control IOVA more precisely)

Message ID 20120924120415.8e6929a34c422185a98d3f82@nvidia.com
State New
Headers show

Commit Message

Hiroshi Doyu Sept. 24, 2012, 9:04 a.m. UTC 
On Fri, 21 Sep 2012 20:16:00 +0200
Krishna Reddy <vdumpa@nvidia.com> wrote:

> > > The device(H/W controller) need to access few special memory
> > > blocks(IOVA==PA) and DRAM as well.
> >
> > OK, so only /some/ of the VA space is VA==PA, and some is remapped; that's a
> > little different that what you originally implied above.
> >
> > BTW, which HW module is this; AVP/COP or something else. This sounds like an
> > odd requirement.
>
> This is not specific to ARM7. There are protected memory regions on Tegra that
> can be accessed by some controllers like display, 2D, 3D, VDE, HDA. These are
> DRAM regions configured as protected by BootRom. These memory regions
> are not exposed to and not managed by OS page allocator. The H/W controller
>  accesses to these regions still to go through IOMMU.
> The IOMMU view for all the H/W controllers is not uniform on Tegra.
> Some Controllers see entire 4GB IOVA space. i.e all accesses go though IOMMU.
> Some controllers see the IOVA Space that don't overlap with MMIO space.  i.e
> The MMIO address access bypass IOMMU and directly go to MMIO space.
> Tegra IOMMU can support multiple address spaces as well. To hide controller
> Specific behavior, the drivers should take care of one to one mapping and
> remove inaccessible iova spaces in their address space's based platform device info.

The above is also related to another issue,
    how to specify IOMMU'able devices in DT.

As mentioned above, some IOVA mapping may be unique to some devices,
and the number of IOMMU'able device are quite many nowadays, a few
dozen in Tegra30 now. Basically they are seen as just normal platform
devices from CPU even if they belong to different busses in H/W. IOW, their
IOMMU'ability just depend on a platfrom bus from _S/W_ POV. Doing each
registration(create a map & attach device) in board files isn't so
nice. Currently we register them at "platform_device_add()" at once
with just a HACK(*1), but this could/should be done based on the info
passed from DT. For tegra, those parameter could be, "ASID" and
"address range"(start, size, alignment). For example in DT:

deviceA {
                      "start"     "size"   "align"
          iommu = <0x12340000 0x00400000 0x0000000>;   # exclusively specify "start" or "align"
          iommu = <0x00000000 0x00400000 0x0010000>;
          iommu = <0x12340000 0x00040000 0x12380000 0x00040000>; # "start", "size" could be repeated...
	  asid = 3; # if needed

or
          dma_range = <0x12340000 0x00400000 0x0000000>; # if iommu is considered as one implementation of dma.....
};

Is there any way to specify each IOMMU'able _platform device_ and
specify its map in DT?

The above ASID may be specific to Tegra, though. If we can specify the
above info in DT and the info is passed to kernel, some platform
common code would register them as IOMMU'able device automatically. It
would be really covenient if this is done in platform_device/IOMMU
common code. If the above attribute is implemented specific to
Tegra/platform, we have to call attach_device quite many times
somewhere in device initializations.

Any comment would be really appreciated.

*1:
From dd4dd6532d705c7bba0914b54c819d8d735c2fad Mon Sep 17 00:00:00 2001
From: Hiroshi Doyu <hdoyu@nvidia.com>
Date: Thu, 22 Mar 2012 16:06:27 +0200
Subject: [RFC 1/1] platform: IOMMU'able platform_device w/
 PLATFORM_ENABLE_IOMMU

Introduced a new kernel config option, PLATFORM_ENABLE_IOMMU. With
this, all platform devices can be converted to be IOMMU'able if
platform_bus has non-null dma_iommu_map, where H/Ws always see its IO
virtual address and virt_to_phys() doesn't work from H/W POV.

Signed-off-by: Hiroshi Doyu <hdoyu@nvidia.com>
---
 arch/arm/mm/dma-mapping.c |    7 +++++++
 drivers/base/Kconfig      |    4 ++++
 drivers/base/platform.c   |   17 +++++++++++++++--
 drivers/iommu/Kconfig     |    2 +-
 include/linux/device.h    |    5 ++++-
 5 files changed, 31 insertions(+), 4 deletions(-)

Comments

James Bottomley Sept. 24, 2012, 9:28 a.m. UTC | #1 
On Mon, 2012-09-24 at 12:04 +0300, Hiroshi Doyu wrote:
> diff --git a/drivers/base/platform.c b/drivers/base/platform.c
> index a1a7225..9eae3be 100644
> --- a/drivers/base/platform.c
> +++ b/drivers/base/platform.c
> @@ -21,6 +21,8 @@
>  #include <linux/slab.h>
>  #include <linux/pm_runtime.h>
> 
> +#include <asm/dma-iommu.h>
> +
>  #include "base.h"
> 
>  #define to_platform_driver(drv)        (container_of((drv), struct
> platform_driver, \
> @@ -305,8 +307,19 @@ int platform_device_add(struct platform_device
> *pdev)
>                  dev_name(&pdev->dev), dev_name(pdev->dev.parent));
> 
>         ret = device_add(&pdev->dev);
> -       if (ret == 0)
> -               return ret;
> +       if (ret)
> +               goto failed;
> +
> +#ifdef CONFIG_PLATFORM_ENABLE_IOMMU
> +       if (platform_bus_type.map && !pdev->dev.archdata.mapping) {
> +               ret = arm_iommu_attach_device(&pdev->dev,
> +                                             platform_bus_type.map);
> +               if (ret)
> +                       goto failed;

This is horrible ... you're adding an architecture specific callback
into our generic code; that's really a no-no.  If the concept of
CONFIG_PLATFORM_ENABE_IOMMU is useful to more than just arm, then this
could become a generic callback.

James
Hiroshi Doyu Sept. 24, 2012, 9:44 a.m. UTC | #2 
Hi James,

On Mon, 24 Sep 2012 11:28:01 +0200
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> On Mon, 2012-09-24 at 12:04 +0300, Hiroshi Doyu wrote:
> > diff --git a/drivers/base/platform.c b/drivers/base/platform.c
> > index a1a7225..9eae3be 100644
> > --- a/drivers/base/platform.c
> > +++ b/drivers/base/platform.c
> > @@ -21,6 +21,8 @@
> >  #include <linux/slab.h>
> >  #include <linux/pm_runtime.h>
> > 
> > +#include <asm/dma-iommu.h>
> > +
> >  #include "base.h"
> > 
> >  #define to_platform_driver(drv)        (container_of((drv), struct
> > platform_driver, \
> > @@ -305,8 +307,19 @@ int platform_device_add(struct platform_device
> > *pdev)
> >                  dev_name(&pdev->dev), dev_name(pdev->dev.parent));
> > 
> >         ret = device_add(&pdev->dev);
> > -       if (ret == 0)
> > -               return ret;
> > +       if (ret)
> > +               goto failed;
> > +
> > +#ifdef CONFIG_PLATFORM_ENABLE_IOMMU
> > +       if (platform_bus_type.map && !pdev->dev.archdata.mapping) {
> > +               ret = arm_iommu_attach_device(&pdev->dev,
> > +                                             platform_bus_type.map);
> > +               if (ret)
> > +                       goto failed;
> 
> This is horrible ... you're adding an architecture specific callback
> into our generic code; that's really a no-no.  If the concept of
> CONFIG_PLATFORM_ENABE_IOMMU is useful to more than just arm, then this
> could become a generic callback.

As mentioned in the original, this is a heck to explain what is
needed. I am looking for some generic solution for how to specify
IOMMU info for each platform devices. I'm guessing that some other SoC
may have the similar requirements on the above. As you mentioned, this
solution should be a generic, not arch specific.
Marek Szyprowski Sept. 24, 2012, 11:14 a.m. UTC | #3 
Hello,

On Monday, September 24, 2012 11:45 AM Hiroshi Doyu wrote:

> On Mon, 24 Sep 2012 11:28:01 +0200
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> > On Mon, 2012-09-24 at 12:04 +0300, Hiroshi Doyu wrote:
> > > diff --git a/drivers/base/platform.c b/drivers/base/platform.c
> > > index a1a7225..9eae3be 100644
> > > --- a/drivers/base/platform.c
> > > +++ b/drivers/base/platform.c
> > > @@ -21,6 +21,8 @@
> > >  #include <linux/slab.h>
> > >  #include <linux/pm_runtime.h>
> > >
> > > +#include <asm/dma-iommu.h>
> > > +
> > >  #include "base.h"
> > >
> > >  #define to_platform_driver(drv)        (container_of((drv), struct
> > > platform_driver, \
> > > @@ -305,8 +307,19 @@ int platform_device_add(struct platform_device
> > > *pdev)
> > >                  dev_name(&pdev->dev), dev_name(pdev->dev.parent));
> > >
> > >         ret = device_add(&pdev->dev);
> > > -       if (ret == 0)
> > > -               return ret;
> > > +       if (ret)
> > > +               goto failed;
> > > +
> > > +#ifdef CONFIG_PLATFORM_ENABLE_IOMMU
> > > +       if (platform_bus_type.map && !pdev->dev.archdata.mapping) {
> > > +               ret = arm_iommu_attach_device(&pdev->dev,
> > > +                                             platform_bus_type.map);
> > > +               if (ret)
> > > +                       goto failed;
> >
> > This is horrible ... you're adding an architecture specific callback
> > into our generic code; that's really a no-no.  If the concept of
> > CONFIG_PLATFORM_ENABE_IOMMU is useful to more than just arm, then this
> > could become a generic callback.
> 
> As mentioned in the original, this is a heck to explain what is
> needed. I am looking for some generic solution for how to specify
> IOMMU info for each platform devices. I'm guessing that some other SoC
> may have the similar requirements on the above. As you mentioned, this
> solution should be a generic, not arch specific.

Please read more about bus notifiers. IMHO a good example is provided in 
the following thread:
http://www.mail-archive.com/linux-samsung-soc@vger.kernel.org/msg12238.html

Best regards
Hiroshi Doyu Sept. 24, 2012, 11:50 a.m. UTC | #4 
Hi Marek,

Marek Szyprowski <m.szyprowski@samsung.com> wrote @ Mon, 24 Sep 2012 13:14:51 +0200:

> Hello,
> 
> On Monday, September 24, 2012 11:45 AM Hiroshi Doyu wrote:
> 
> > On Mon, 24 Sep 2012 11:28:01 +0200
> > James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> > 
> > > On Mon, 2012-09-24 at 12:04 +0300, Hiroshi Doyu wrote:
> > > > diff --git a/drivers/base/platform.c b/drivers/base/platform.c
> > > > index a1a7225..9eae3be 100644
> > > > --- a/drivers/base/platform.c
> > > > +++ b/drivers/base/platform.c
> > > > @@ -21,6 +21,8 @@
> > > >  #include <linux/slab.h>
> > > >  #include <linux/pm_runtime.h>
> > > >
> > > > +#include <asm/dma-iommu.h>
> > > > +
> > > >  #include "base.h"
> > > >
> > > >  #define to_platform_driver(drv)        (container_of((drv), struct
> > > > platform_driver, \
> > > > @@ -305,8 +307,19 @@ int platform_device_add(struct platform_device
> > > > *pdev)
> > > >                  dev_name(&pdev->dev), dev_name(pdev->dev.parent));
> > > >
> > > >         ret = device_add(&pdev->dev);
> > > > -       if (ret == 0)
> > > > -               return ret;
> > > > +       if (ret)
> > > > +               goto failed;
> > > > +
> > > > +#ifdef CONFIG_PLATFORM_ENABLE_IOMMU
> > > > +       if (platform_bus_type.map && !pdev->dev.archdata.mapping) {
> > > > +               ret = arm_iommu_attach_device(&pdev->dev,
> > > > +                                             platform_bus_type.map);
> > > > +               if (ret)
> > > > +                       goto failed;
> > >
> > > This is horrible ... you're adding an architecture specific callback
> > > into our generic code; that's really a no-no.  If the concept of
> > > CONFIG_PLATFORM_ENABE_IOMMU is useful to more than just arm, then this
> > > could become a generic callback.
> > 
> > As mentioned in the original, this is a heck to explain what is
> > needed. I am looking for some generic solution for how to specify
> > IOMMU info for each platform devices. I'm guessing that some other SoC
> > may have the similar requirements on the above. As you mentioned, this
> > solution should be a generic, not arch specific.
> 
> Please read more about bus notifiers. IMHO a good example is provided in 
> the following thread:
> http://www.mail-archive.com/linux-samsung-soc@vger.kernel.org/msg12238.html

This bus notifier seems enough flexible to afford the variation of
IOMMU map info, like Tegra ASID, which could be platform-specific, and
the other could be common too. There's already iommu_bus_notifier
too. I'll try to implement something base on this.

Thanks for the good info.
diff mbox

Patch

diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 242289f..28ca7c2 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -1899,6 +1899,13 @@  arm_iommu_create_mapping(struct bus_type *bus, dma_addr_t base, size_t size,
 	mapping->order = order;
 	spin_lock_init(&mapping->lock);

+#ifdef CONFIG_PLATFORM_ENABLE_IOMMU
+	if (WARN_ON(bus->map))
+		goto err3;
+
+	bus->map = mapping;
+#endif
+
 	mapping->domain = iommu_domain_alloc(bus);
 	if (!mapping->domain)
 		goto err3;
diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index 3df339c..0f45311 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -308,4 +308,8 @@  config CMA_AREAS

 endif

+config PLATFORM_ENABLE_IOMMU
+        bool "Make platform devices iommuable"
+	depends on IOMMU_API
+
 endmenu
diff --git a/drivers/base/platform.c b/drivers/base/platform.c
index a1a7225..9eae3be 100644
--- a/drivers/base/platform.c
+++ b/drivers/base/platform.c
@@ -21,6 +21,8 @@ 
 #include <linux/slab.h>
 #include <linux/pm_runtime.h>

+#include <asm/dma-iommu.h>
+
 #include "base.h"

 #define to_platform_driver(drv)	(container_of((drv), struct platform_driver, \
@@ -305,8 +307,19 @@  int platform_device_add(struct platform_device *pdev)
 		 dev_name(&pdev->dev), dev_name(pdev->dev.parent));

 	ret = device_add(&pdev->dev);
-	if (ret == 0)
-		return ret;
+	if (ret)
+		goto failed;
+
+#ifdef CONFIG_PLATFORM_ENABLE_IOMMU
+	if (platform_bus_type.map && !pdev->dev.archdata.mapping) {
+		ret = arm_iommu_attach_device(&pdev->dev,
+					      platform_bus_type.map);
+		if (ret)
+			goto failed;
+	}
+#endif
+
+	return 0;

  failed:
 	while (--i >= 0) {
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index b736809..8b7eca1 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -164,7 +164,7 @@  config TEGRA_IOMMU_SMMU

 config TEGRA_IOMMU_SMMU_LINEAR
 	bool "Physical RAM IOVA Liner Mapping Support"
-	depends on TEGRA_IOMMU_SMMU
+	depends on TEGRA_IOMMU_SMMU && !PLATFORM_ENABLE_IOMMU
 	default y
 	help
 	  Enables support for linear mapping between physical address
diff --git a/include/linux/device.h b/include/linux/device.h
index e339929..3dcb501 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -35,6 +35,7 @@  struct subsys_private;
 struct bus_type;
 struct device_node;
 struct iommu_ops;
+struct dma_iommu_mapping;

 struct bus_attribute {
 	struct attribute	attr;
@@ -106,7 +107,9 @@  struct bus_type {
 	const struct dev_pm_ops *pm;

 	struct iommu_ops *iommu_ops;
-
+#ifdef CONFIG_PLATFORM_ENABLE_IOMMU
+	struct dma_iommu_mapping *map;
+#endif
 	struct subsys_private *p;
 };