[Xen-devel,v4,26/33] xen/passthrough: Extend XEN_DOMCTL_*assign_device to support DT device

Message ID 1426793399-6283-27-git-send-email-julien.grall@linaro.org
State New
Headers show

Commit Message

Julien Grall March 19, 2015, 7:29 p.m.
A device node is described by a path. It will be used to retrieved the
node in the device tree and assign the related device to the domain.

Only non-PCI protected by an IOMMU can be assigned to a guest.

Also document the behavior of XEN_DOMCTL_deassign_device in the public
headers which differ between non-PCI and PCI.

Signed-off-by: Julien Grall <julien.grall@linaro.org>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>

---
    Changes in v4:
        - Add XSM bits
        - Return -ENODEV rather than -ENOSYS
        - Move the if (...) into the ifdef (see iommu.c)
        - Document the behavior of XEN_DOMCTL_deassign_device
        - Use PCI_BUS and PCI_DEVFN2 when it's possible
        - iommu_dt_device_is_assigned now returns 0 when the device is
        not protected

    Changes in v2:
        - Use a different number for XEN_DOMCTL_assign_dt_device
---
 tools/libxc/include/xenctrl.h         |  10 ++++
 tools/libxc/xc_domain.c               |  95 ++++++++++++++++++++++++++++--
 xen/drivers/passthrough/device_tree.c | 108 +++++++++++++++++++++++++++++++++-
 xen/drivers/passthrough/iommu.c       |   9 ++-
 xen/drivers/passthrough/pci.c         |  47 ++++++++++-----
 xen/include/public/domctl.h           |  24 +++++++-
 xen/include/xen/iommu.h               |   3 +
 7 files changed, 271 insertions(+), 25 deletions(-)

Comments

Julien Grall March 31, 2015, 12:30 p.m. | #1
Hi Ian,

On 31/03/15 12:24, Ian Campbell wrote:
> On Thu, 2015-03-19 at 19:29 +0000, Julien Grall wrote:
>> +int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
>> +                       XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
>> +{
>> +    int ret;
>> +    struct dt_device_node *dev;
>> +
>> +    /* TODO: How to deal with XSM? */
> 
> Looks like you do in the following code? Old comment?

Old comment. I forgot to drop it.

>> +    /* TODO: Do we need to check is_dying? Mostly to protect against
>> +     * hypercall trying to passthrough a device while we are
>> +     * dying.
> 
> iommu_do_pci_domctl does in specific casses (i.e. assign device). I
> guess you should follow that lead.

I'm not sure to fully understand when is_dying should be used or not.

Looking to the PCI code, the is_dying has been added when we add code to
deal with page.

I would be inclined to say it's only necessary when deadling with page.
Can someone confirm me?

Otherwise, I don't why is_dying should be check here and not in other call.

> 
>> diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
>> index 7f90150..a7cb272 100644
>> --- a/xen/include/public/domctl.h
>> +++ b/xen/include/public/domctl.h
>> @@ -475,12 +475,30 @@ typedef struct xen_domctl_sendtrigger xen_domctl_sendtrigger_t;
>>  DEFINE_XEN_GUEST_HANDLE(xen_domctl_sendtrigger_t);
>>  
>>
>> -/* Assign PCI device to HVM guest. Sets up IOMMU structures. */
>> +/* Assign a device to a guest. Sets up IOMMU structures. */
>>  /* XEN_DOMCTL_assign_device */
>>  /* XEN_DOMCTL_test_assign_device */
>> -/* XEN_DOMCTL_deassign_device */
>> +/*
>> + * XEN_DOMCTL_deassign_device: The behavior of this DOMCTL differs
>> + * between the different type of device:
>> + *  - PCI device (XEN_DOMCTL_DEV_PCI) will be reassigned to DOM0
>> + *  - non-PCI device (XEN_DOMCTL_DEV_PCI) will left unassigned. DOM0
> 
> Did you miss a ! before XEN_DOMCTL, or did you mean to say DT?

I intended to say XEN_DOMCTL_DT. I will fix it.

> From an ease of review PoV it would have been nice to add dev and u.pci
> first since that would be mechanical, but nevermind now.

Sorry, I was too lazy to do it.

Regards,

Patch

diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index b6212bf..4648cb0 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2055,6 +2055,16 @@  int xc_deassign_device(xc_interface *xch,
                      uint32_t domid,
                      uint32_t machine_sbdf);
 
+int xc_assign_dt_device(xc_interface *xch,
+                        uint32_t domid,
+                        char *path);
+int xc_test_assign_dt_device(xc_interface *xch,
+                             uint32_t domid,
+                             char *path);
+int xc_deassign_dt_device(xc_interface *xch,
+                          uint32_t domid,
+                          char *path);
+
 int xc_domain_memory_mapping(xc_interface *xch,
                              uint32_t domid,
                              unsigned long first_gfn,
diff --git a/tools/libxc/xc_domain.c b/tools/libxc/xc_domain.c
index 8243b70..924a180 100644
--- a/tools/libxc/xc_domain.c
+++ b/tools/libxc/xc_domain.c
@@ -1633,7 +1633,8 @@  int xc_assign_device(
 
     domctl.cmd = XEN_DOMCTL_assign_device;
     domctl.domain = domid;
-    domctl.u.assign_device.machine_sbdf = machine_sbdf;
+    domctl.u.assign_device.dev = XEN_DOMCTL_DEV_PCI;
+    domctl.u.assign_device.u.pci.machine_sbdf = machine_sbdf;
 
     return do_domctl(xch, &domctl);
 }
@@ -1682,7 +1683,8 @@  int xc_test_assign_device(
 
     domctl.cmd = XEN_DOMCTL_test_assign_device;
     domctl.domain = domid;
-    domctl.u.assign_device.machine_sbdf = machine_sbdf;
+    domctl.u.assign_device.dev = XEN_DOMCTL_DEV_PCI;
+    domctl.u.assign_device.u.pci.machine_sbdf = machine_sbdf;
 
     return do_domctl(xch, &domctl);
 }
@@ -1696,11 +1698,96 @@  int xc_deassign_device(
 
     domctl.cmd = XEN_DOMCTL_deassign_device;
     domctl.domain = domid;
-    domctl.u.assign_device.machine_sbdf = machine_sbdf;
- 
+    domctl.u.assign_device.dev = XEN_DOMCTL_DEV_PCI;
+    domctl.u.assign_device.u.pci.machine_sbdf = machine_sbdf;
+
     return do_domctl(xch, &domctl);
 }
 
+int xc_assign_dt_device(
+    xc_interface *xch,
+    uint32_t domid,
+    char *path)
+{
+    int rc;
+    size_t size = strlen(path);
+    DECLARE_DOMCTL;
+    DECLARE_HYPERCALL_BOUNCE(path, size, XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+    if ( xc_hypercall_bounce_pre(xch, path) )
+        return -1;
+
+    domctl.cmd = XEN_DOMCTL_assign_device;
+    domctl.domain = (domid_t)domid;
+
+    domctl.u.assign_device.dev = XEN_DOMCTL_DEV_DT;
+    domctl.u.assign_device.u.dt.size = size;
+    set_xen_guest_handle(domctl.u.assign_device.u.dt.path, path);
+
+    rc = do_domctl(xch, &domctl);
+
+    xc_hypercall_bounce_post(xch, path);
+
+    return rc;
+}
+
+int xc_test_assign_dt_device(
+    xc_interface *xch,
+    uint32_t domid,
+    char *path)
+{
+    int rc;
+    size_t size = strlen(path);
+    DECLARE_DOMCTL;
+    DECLARE_HYPERCALL_BOUNCE(path, size, XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+    if ( xc_hypercall_bounce_pre(xch, path) )
+        return -1;
+
+    domctl.cmd = XEN_DOMCTL_test_assign_device;
+    domctl.domain = (domid_t)domid;
+
+    domctl.u.assign_device.dev = XEN_DOMCTL_DEV_DT;
+    domctl.u.assign_device.u.dt.size = size;
+    set_xen_guest_handle(domctl.u.assign_device.u.dt.path, path);
+
+    rc = do_domctl(xch, &domctl);
+
+    xc_hypercall_bounce_post(xch, path);
+
+    return rc;
+}
+
+int xc_deassign_dt_device(
+    xc_interface *xch,
+    uint32_t domid,
+    char *path)
+{
+    int rc;
+    size_t size = strlen(path);
+    DECLARE_DOMCTL;
+    DECLARE_HYPERCALL_BOUNCE(path, size, XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+    if ( xc_hypercall_bounce_pre(xch, path) )
+        return -1;
+
+    domctl.cmd = XEN_DOMCTL_deassign_device;
+    domctl.domain = (domid_t)domid;
+
+    domctl.u.assign_device.dev = XEN_DOMCTL_DEV_DT;
+    domctl.u.assign_device.u.dt.size = size;
+    set_xen_guest_handle(domctl.u.assign_device.u.dt.path, path);
+
+    rc = do_domctl(xch, &domctl);
+
+    xc_hypercall_bounce_post(xch, path);
+
+    return rc;
+}
+
+
+
+
 int xc_domain_update_msi_irq(
     xc_interface *xch,
     uint32_t domid,
diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index 0ec4103..c37efeb 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -1,9 +1,6 @@ 
 /*
  * Code to passthrough a device tree node to a guest
  *
- * TODO: This contains only the necessary code to protected device passed to
- * dom0. It will need some updates when device passthrough will is added.
- *
  * Julien Grall <julien.grall@linaro.org>
  * Copyright (c) 2014 Linaro Limited.
  *
@@ -20,8 +17,10 @@ 
 
 #include <xen/lib.h>
 #include <xen/sched.h>
+#include <xen/guest_access.h>
 #include <xen/iommu.h>
 #include <xen/device_tree.h>
+#include <xsm/xsm.h>
 
 static spinlock_t dtdevs_lock = SPIN_LOCK_UNLOCKED;
 
@@ -93,6 +92,20 @@  fail:
     return rc;
 }
 
+static bool_t iommu_dt_device_is_assigned(const struct dt_device_node *dev)
+{
+    bool_t assigned = 0;
+
+    if ( !dt_device_is_protected(dev) )
+        return 0;
+
+    spin_lock(&dtdevs_lock);
+    assigned = !list_empty(&dev->domain_list);
+    spin_unlock(&dtdevs_lock);
+
+    return assigned;
+}
+
 int iommu_dt_domain_init(struct domain *d)
 {
     struct hvm_iommu *hd = domain_hvm_iommu(d);
@@ -121,3 +134,92 @@  int iommu_release_dt_devices(struct domain *d)
 
     return 0;
 }
+
+int iommu_do_dt_domctl(struct xen_domctl *domctl, struct domain *d,
+                       XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
+{
+    int ret;
+    struct dt_device_node *dev;
+
+    /* TODO: How to deal with XSM? */
+    /* TODO: Do we need to check is_dying? Mostly to protect against
+     * hypercall trying to passthrough a device while we are
+     * dying.
+     */
+
+    switch ( domctl->cmd )
+    {
+    case XEN_DOMCTL_assign_device:
+        ret = -ENODEV;
+        if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_DT )
+            break;
+
+        ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
+                                    domctl->u.assign_device.u.dt.size,
+                                    &dev);
+        if ( ret )
+            break;
+
+        ret = xsm_assign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev));
+        if ( ret )
+            break;
+
+        ret = iommu_assign_dt_device(d, dev);
+
+        if ( ret )
+            printk(XENLOG_G_ERR "XEN_DOMCTL_assign_dt_device: assign \"%s\""
+                   " to dom%u failed (%d)\n",
+                   dt_node_full_name(dev), d->domain_id, ret);
+        break;
+
+    case XEN_DOMCTL_deassign_device:
+        ret = -ENODEV;
+        if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_DT )
+            break;
+
+        ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
+                                    domctl->u.assign_device.u.dt.size,
+                                    &dev);
+        if ( ret )
+            break;
+
+        ret = xsm_deassign_dtdevice(XSM_HOOK, d, dt_node_full_name(dev));
+
+        ret = iommu_deassign_dt_device(d, dev);
+
+        if ( ret )
+            printk(XENLOG_G_ERR "XEN_DOMCTL_assign_dt_device: assign \"%s\""
+                   " to dom%u failed (%d)\n",
+                   dt_node_full_name(dev), d->domain_id, ret);
+        break;
+
+    case XEN_DOMCTL_test_assign_device:
+        ret = -ENODEV;
+        if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_DT )
+            break;
+
+        ret = dt_find_node_by_gpath(domctl->u.assign_device.u.dt.path,
+                                    domctl->u.assign_device.u.dt.size,
+                                    &dev);
+        if ( ret )
+            break;
+
+        ret = xsm_test_assign_dtdevice(XSM_HOOK, dt_node_full_name(dev));
+        if ( ret )
+            break;
+
+        if ( iommu_dt_device_is_assigned(dev) )
+        {
+            printk(XENLOG_G_ERR "%s already assigned.\n",
+                   dt_node_full_name(dev));
+            ret = -EINVAL;
+        }
+        break;
+
+    default:
+        ret = -ENOSYS;
+        break;
+    }
+
+    return ret;
+}
diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index faddd50..a5d0831 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -335,7 +335,7 @@  int iommu_do_domctl(
     struct xen_domctl *domctl, struct domain *d,
     XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
 {
-    int ret = -ENOSYS;
+    int ret = -ENODEV;
 
     if ( !iommu_enabled )
         return -ENOSYS;
@@ -344,6 +344,13 @@  int iommu_do_domctl(
     ret = iommu_do_pci_domctl(domctl, d, u_domctl);
 #endif
 
+#ifdef HAS_DEVICE_TREE
+    if ( ret != -ENODEV)
+        return ret;
+
+    ret = iommu_do_dt_domctl(domctl, d, u_domctl);
+#endif
+
     return ret;
 }
 
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 18b74f4..5ddfd8d 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -1495,6 +1495,7 @@  int iommu_do_pci_domctl(
     u16 seg;
     u8 bus, devfn;
     int ret = 0;
+    uint32_t machine_sbdf;
 
     switch ( domctl->cmd )
     {
@@ -1508,8 +1509,8 @@  int iommu_do_pci_domctl(
             break;
 
         seg = domctl->u.get_device_group.machine_sbdf >> 16;
-        bus = (domctl->u.get_device_group.machine_sbdf >> 8) & 0xff;
-        devfn = domctl->u.get_device_group.machine_sbdf & 0xff;
+        bus = PCI_BUS(domctl->u.get_device_group.machine_sbdf);
+        devfn = PCI_DEVFN2(domctl->u.get_device_group.machine_sbdf);
         max_sdevs = domctl->u.get_device_group.max_sdevs;
         sdevs = domctl->u.get_device_group.sdev_array;
 
@@ -1531,13 +1532,19 @@  int iommu_do_pci_domctl(
     break;
 
     case XEN_DOMCTL_test_assign_device:
-        ret = xsm_test_assign_device(XSM_HOOK, domctl->u.assign_device.machine_sbdf);
+        ret = -ENODEV;
+        if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_PCI )
+            break;
+
+        machine_sbdf = domctl->u.assign_device.u.pci.machine_sbdf;
+
+        ret = xsm_test_assign_device(XSM_HOOK, machine_sbdf);
         if ( ret )
             break;
 
-        seg = domctl->u.assign_device.machine_sbdf >> 16;
-        bus = (domctl->u.assign_device.machine_sbdf >> 8) & 0xff;
-        devfn = domctl->u.assign_device.machine_sbdf & 0xff;
+        seg = machine_sbdf >> 16;
+        bus = PCI_BUS(machine_sbdf);
+        devfn = PCI_DEVFN2(machine_sbdf);
 
         if ( device_assigned(seg, bus, devfn) )
         {
@@ -1549,19 +1556,25 @@  int iommu_do_pci_domctl(
         break;
 
     case XEN_DOMCTL_assign_device:
+        ret = -ENODEV;
+        if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_PCI )
+            break;
+
         if ( unlikely(d->is_dying) )
         {
             ret = -EINVAL;
             break;
         }
 
-        ret = xsm_assign_device(XSM_HOOK, d, domctl->u.assign_device.machine_sbdf);
+        machine_sbdf = domctl->u.assign_device.u.pci.machine_sbdf;
+
+        ret = xsm_assign_device(XSM_HOOK, d, machine_sbdf);
         if ( ret )
             break;
 
-        seg = domctl->u.assign_device.machine_sbdf >> 16;
-        bus = (domctl->u.assign_device.machine_sbdf >> 8) & 0xff;
-        devfn = domctl->u.assign_device.machine_sbdf & 0xff;
+        seg = machine_sbdf >> 16;
+        bus = PCI_BUS(machine_sbdf);
+        devfn = PCI_DEVFN2(machine_sbdf);
 
         ret = device_assigned(seg, bus, devfn) ?:
               assign_device(d, seg, bus, devfn);
@@ -1577,13 +1590,19 @@  int iommu_do_pci_domctl(
         break;
 
     case XEN_DOMCTL_deassign_device:
-        ret = xsm_deassign_device(XSM_HOOK, d, domctl->u.assign_device.machine_sbdf);
+        ret = -ENODEV;
+        if ( domctl->u.assign_device.dev != XEN_DOMCTL_DEV_PCI )
+            break;
+
+        machine_sbdf = domctl->u.assign_device.u.pci.machine_sbdf;
+
+        ret = xsm_deassign_device(XSM_HOOK, d, machine_sbdf);
         if ( ret )
             break;
 
-        seg = domctl->u.assign_device.machine_sbdf >> 16;
-        bus = (domctl->u.assign_device.machine_sbdf >> 8) & 0xff;
-        devfn = domctl->u.assign_device.machine_sbdf & 0xff;
+        seg = machine_sbdf >> 16;
+        bus = PCI_BUS(machine_sbdf);
+        devfn = PCI_DEVFN2(machine_sbdf);
 
         spin_lock(&pcidevs_lock);
         ret = deassign_device(d, seg, bus, devfn);
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index 7f90150..a7cb272 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -475,12 +475,30 @@  typedef struct xen_domctl_sendtrigger xen_domctl_sendtrigger_t;
 DEFINE_XEN_GUEST_HANDLE(xen_domctl_sendtrigger_t);
 
 
-/* Assign PCI device to HVM guest. Sets up IOMMU structures. */
+/* Assign a device to a guest. Sets up IOMMU structures. */
 /* XEN_DOMCTL_assign_device */
 /* XEN_DOMCTL_test_assign_device */
-/* XEN_DOMCTL_deassign_device */
+/*
+ * XEN_DOMCTL_deassign_device: The behavior of this DOMCTL differs
+ * between the different type of device:
+ *  - PCI device (XEN_DOMCTL_DEV_PCI) will be reassigned to DOM0
+ *  - non-PCI device (XEN_DOMCTL_DEV_PCI) will left unassigned. DOM0
+ *  will have to call XEN_DOMCTL_assign_device in order to use the
+ *  device.
+ */
+#define XEN_DOMCTL_DEV_PCI      0
+#define XEN_DOMCTL_DEV_DT       1
 struct xen_domctl_assign_device {
-    uint32_t  machine_sbdf;   /* machine PCI ID of assigned device */
+    uint32_t dev;   /* XEN_DOMCTL_DEV_* */
+    union {
+        struct {
+            uint32_t machine_sbdf;   /* machine PCI ID of assigned device */
+        } pci;
+        struct {
+            uint32_t size; /* Length of the path */
+            XEN_GUEST_HANDLE_64(char) path; /* path to the device tree node */
+        } dt;
+    } u;
 };
 typedef struct xen_domctl_assign_device xen_domctl_assign_device_t;
 DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index d9c9ede..b30bf41 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -119,6 +119,9 @@  int iommu_deassign_dt_device(struct domain *d, struct dt_device_node *dev);
 int iommu_dt_domain_init(struct domain *d);
 int iommu_release_dt_devices(struct domain *d);
 
+int iommu_do_dt_domctl(struct xen_domctl *, struct domain *,
+                       XEN_GUEST_HANDLE_PARAM(xen_domctl_t));
+
 #endif /* HAS_DEVICE_TREE */
 
 struct page_info;