From patchwork Wed Apr 9 15:33:08 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Auger Eric X-Patchwork-Id: 28102 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ob0-f197.google.com (mail-ob0-f197.google.com [209.85.214.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id C2B2F202DD for ; Wed, 9 Apr 2014 15:55:55 +0000 (UTC) Received: by mail-ob0-f197.google.com with SMTP id wp18sf11642801obc.8 for ; Wed, 09 Apr 2014 08:55:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:date :message-id:in-reply-to:references:cc:subject:precedence:list-id :list-unsubscribe:list-archive:list-post:list-help:list-subscribe :errors-to:sender:x-original-sender :x-original-authentication-results:mailing-list; bh=TRX79E1Rs5kJkPYVXXcb3kjdSBvv3GU6q07B9bxMiLE=; b=Fr6eW7rGYpyXe3z3fnqLOT4wpMwTvoSjA5XhRAXqRmLY8rkLtLqb+jL7oKhzKlnUlF XxFSExkPCxFbn8W3BRS2EV3P2Ed3jPLZYxxvNQE7tc9Qj1ya7aAWbnmM/ZojYXiBWTuD 5wqzLLBozReqciGb9IxKr91ekzQJ6yoHdm42+NfsKzJd8jtWjIGhUDpcVu29GbmKagaB 0hXF7rFVMiLQTW/cM8ocRE9ywbQFs3gpia+gfqUZKJiQYVDQkFbcQPZttr0ei4+4Mq4O L/rIJc4CXgZLR5pscZTo/0GvDYP3H7srMXhRuaXFb5wux3shsgRjxjqjbFBXyCs+e3K4 9PsQ== X-Gm-Message-State: ALoCoQkYecRUPE0KNPhCksMW3PRcn5TtFgW6pAGVHylxokL0OecN1RboGBR7NwpWqRCFn9QaIUGt X-Received: by 10.50.70.66 with SMTP id k2mr2866939igu.7.1397058955317; Wed, 09 Apr 2014 08:55:55 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.102.103 with SMTP id v94ls745819qge.0.gmail; Wed, 09 Apr 2014 08:55:55 -0700 (PDT) X-Received: by 10.220.139.198 with SMTP id f6mr1166429vcu.47.1397058955104; Wed, 09 Apr 2014 08:55:55 -0700 (PDT) Received: from mail-vc0-f170.google.com (mail-vc0-f170.google.com [209.85.220.170]) by mx.google.com with ESMTPS id xt10si206157veb.58.2014.04.09.08.55.55 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 09 Apr 2014 08:55:55 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.170 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.170; Received: by mail-vc0-f170.google.com with SMTP id hu19so2336761vcb.1 for ; Wed, 09 Apr 2014 08:55:55 -0700 (PDT) X-Received: by 10.58.132.228 with SMTP id ox4mr1056534veb.54.1397058955012; Wed, 09 Apr 2014 08:55:55 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.12.8 with SMTP id v8csp341198vcv; Wed, 9 Apr 2014 08:55:54 -0700 (PDT) X-Received: by 10.224.3.5 with SMTP id 5mr13701963qal.45.1397058954468; Wed, 09 Apr 2014 08:55:54 -0700 (PDT) Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id t9si600190qct.35.2014.04.09.08.55.54 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 09 Apr 2014 08:55:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Received: from localhost ([::1]:46754 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WXuXW-0002N9-4S for patch@linaro.org; Wed, 09 Apr 2014 11:35:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46271) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WXuVw-000093-VS for qemu-devel@nongnu.org; Wed, 09 Apr 2014 11:34:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WXuVq-0000GE-VY for qemu-devel@nongnu.org; Wed, 09 Apr 2014 11:34:12 -0400 Received: from mail-wi0-f171.google.com ([209.85.212.171]:63475) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WXuVq-0000G7-M5 for qemu-devel@nongnu.org; Wed, 09 Apr 2014 11:34:06 -0400 Received: by mail-wi0-f171.google.com with SMTP id q5so9124245wiv.10 for ; Wed, 09 Apr 2014 08:34:06 -0700 (PDT) X-Received: by 10.180.23.99 with SMTP id l3mr37911085wif.47.1397057645903; Wed, 09 Apr 2014 08:34:05 -0700 (PDT) Received: from midway01-04-00.lavalab ([88.98.47.97]) by mx.google.com with ESMTPSA id ga10sm2170061wjb.23.2014.04.09.08.34.05 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 09 Apr 2014 08:34:05 -0700 (PDT) From: Eric Auger To: eric.auger@st.com, christoffer.dall@linaro.org, qemu-devel@nongnu.org Date: Wed, 9 Apr 2014 16:33:08 +0100 Message-Id: <1397057589-11779-6-git-send-email-eric.auger@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1397057589-11779-1-git-send-email-eric.auger@linaro.org> References: <1397057589-11779-1-git-send-email-eric.auger@linaro.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.85.212.171 Cc: peter.maydell@linaro.org, kim.phillips@freescale.com, Eric Auger , agraf@suse.de, stuart.yoder@freescale.com, alex.williamson@redhat.com, christophe.barnichon@st.com, a.motakis@virtualopensystems.com, kvmarm@lists.cs.columbia.edu Subject: [Qemu-devel] [RFC v2 5/6] virt: Assign a VFIO platform device to a virt VM in QEMU command line X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: eric.auger@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.170 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 This patch aims at allowing the end-user to specify the device he wants to directly assign to his virt VM in the QEMU command line. The QEMU platform device becomes generic. Current choice is to reuse the "-device" option. For example when assigning Calxeda Midway xgmac device this option is used: -device vfio-platform,vfio_device="fff51000.ethernet",\ compat="calxeda/hb-xgmac",mmap-timeout-ms=1000 where - fff51000.ethernet is the name of the device in /sys/bus/platform/devices/ - calxeda/hb-xgma is the compatibility where the standard coma separator is replaced by "/" since coma is specifically used by QEMU command line parser - mmap-timeout-ms is minimal amount of time (ms) during which the IP register space stays MMIO mapped after an IRQ triggers in order to trap the end of interrupt (EOI). This is an optional parameter (default value set to 1100 ms). virt machine was modified to interpret this line and automatically - map the device at a chosen guest physical address in [0xa004000, 0x10000000], - map the device IRQs after 48, - create the associated guest device tree with the provided compatibility. The "-device" option underlying implementation is not standard which can be argued. Indeed normaly it induces the call to the QEMU device realize function once after the virtual machine init execution. In this case QDEV mappings and device tree creation must happen. Since virt is the place where the whole memory and IRQ mapping is known and device tree is created, it was chosen to interpret the option line there. This means the realize function is called twice, once in virt.c and once after machine init return. The second call returns immediatly since the QEMU device is recognized as already existing. Another way to implement this would be to create a new option in QEMU. Acknowledgements: - a single compatibility currently is supported - IRQ properties set in the device tree should be refined - More generally devices with more complex device tree nodes must be studied and are not currently handled - cases where multiple VFIO devices are assigned could not be tested Signed-off-by: Eric Auger --- hw/arm/virt.c | 178 +++++++++++++++++++++++++++++++++++++++++++---------- hw/vfio/platform.c | 43 ++++++++++--- 2 files changed, 181 insertions(+), 40 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 31ae7d2..1fb66ef 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -40,6 +40,17 @@ #include "exec/address-spaces.h" #include "qemu/bitops.h" #include "qemu/error-report.h" +#include "monitor/qdev.h" +#include "qemu/config-file.h" + +/* + * this function is implemented in vfio/platform.c + * it returns the name, compatibility, IRQ number and register set size. + * the function only is implemented for VFIO platform devices + */ +void vfio_get_props(SysBusDevice *s, char **pname, char **pcompat, + int *pnum_irqs, size_t *psize); + #define NUM_VIRTIO_TRANSPORTS 32 @@ -65,7 +76,7 @@ enum { VIRT_GIC_CPU, VIRT_UART, VIRT_MMIO, - VIRT_ETHERNET, + VIRT_VFIO, }; typedef struct MemMapEntry { @@ -79,7 +90,10 @@ typedef struct VirtBoardInfo { const char *qdevname; const char *gic_compatible; const MemMapEntry *memmap; + qemu_irq pic[NUM_IRQS]; const int *irqmap; + hwaddr avail_vfio_base; + int avail_vfio_irq; int smp_cpus; void *fdt; int fdt_size; @@ -105,16 +119,16 @@ static const MemMapEntry a15memmap[] = { [VIRT_GIC_CPU] = { 0x8002000, 0x1000 }, [VIRT_UART] = { 0x9000000, 0x1000 }, [VIRT_MMIO] = { 0xa000000, 0x200 }, + [VIRT_VFIO] = { 0xa004000, 0x0 }, /* size is dynamically populated */ /* ...repeating for a total of NUM_VIRTIO_TRANSPORTS, each of that size */ /* 0x10000000 .. 0x40000000 reserved for PCI */ - [VIRT_MEM] = { 0x40000000, 1ULL * 1024 * 1024 * 1024 }, - [VIRT_ETHERNET] = { 0xfff41000, 0x1000 }, + [VIRT_MEM] = { 0x40000000, 30ULL * 1024 * 1024 * 1024 }, }; static const int a15irqmap[] = { [VIRT_UART] = 1, [VIRT_MMIO] = 16, /* ...to 16 + NUM_VIRTIO_TRANSPORTS - 1 */ - [VIRT_ETHERNET] = 77, + [VIRT_VFIO] = 48, }; static VirtBoardInfo machines[] = { @@ -266,7 +280,7 @@ static void fdt_add_gic_node(const VirtBoardInfo *vbi) qemu_fdt_setprop_cell(vbi->fdt, "/intc", "phandle", gic_phandle); } -static void create_uart(const VirtBoardInfo *vbi, qemu_irq *pic) +static void create_uart(const VirtBoardInfo *vbi) { char *nodename; hwaddr base = vbi->memmap[VIRT_UART].base; @@ -275,7 +289,7 @@ static void create_uart(const VirtBoardInfo *vbi, qemu_irq *pic) const char compat[] = "arm,pl011\0arm,primecell"; const char clocknames[] = "uartclk\0apb_pclk"; - sysbus_create_simple("pl011", base, pic[irq]); + sysbus_create_simple("pl011", base, vbi->pic[irq]); nodename = g_strdup_printf("/pl011@%" PRIx64, base); qemu_fdt_add_subnode(vbi->fdt, nodename); @@ -294,34 +308,133 @@ static void create_uart(const VirtBoardInfo *vbi, qemu_irq *pic) g_free(nodename); } -static void create_ethernet(const VirtBoardInfo *vbi, qemu_irq *pic) +/* + * Function called for each vfio-platform device option found in the + * qemu user command line: + * -device vfio-platform,vfio-device="",compat"" + * for instance can be fff51000.ethernet (device unbound from + * original driver and bound to vfio driver) + * for instance can be calxeda/hb-xgmac + * note "/" replaces normal ",". Indeed "," would be interpreted by QEMU as + * a separator + */ + +static int vfio_init_func(QemuOpts *opts, void *opaque) { + const char *driver; + DeviceState *dev; + SysBusDevice *s; + VirtBoardInfo *vbi = (VirtBoardInfo *)opaque; + driver = qemu_opt_get(opts, "driver"); + + /* index the first IRQ should be mapped */ + int irq_start = vbi->avail_vfio_irq; + char *nodename; - hwaddr base = vbi->memmap[VIRT_ETHERNET].base; - hwaddr size = vbi->memmap[VIRT_ETHERNET].size; - const char compat[] = "calxeda,hb-xgmac"; - int irqm = vbi->irqmap[VIRT_ETHERNET]; - int irqp = irqm+1; - int irqlp = irqm+2; - sysbus_create_varargs("vfio-platform", base, - pic[irqm], pic[irqp], pic[irqlp], NULL); + /* this will contain the capatibility string with the "/" + * replaced by "," + */ + char *corrected_compat; - nodename = g_strdup_printf("/ethernet@%" PRIx64, base); - qemu_fdt_add_subnode(vbi->fdt, nodename); + char **pname = g_malloc0(sizeof(char *)); + char **pcompat = g_malloc0(sizeof(char *)); - /* Note that we can't use setprop_string because of the embedded NUL */ - qemu_fdt_setprop(vbi->fdt, nodename, "compatible", compat, sizeof(compat)); - qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, "reg", 2, base, 2, size); - qemu_fdt_setprop_cells(vbi->fdt, nodename, "interrupts", - 0x0, irqm, 0x4, - 0x0, irqp, 0x4, - 0x0, irqlp, 0x4); + int num_irqs; + size_t size; + int i; + uint32_t *irq_attr; - g_free(nodename); + if (!driver) { + qerror_report(QERR_MISSING_PARAMETER, "driver"); + return -1 ; + } + + if (strcasecmp(driver, "vfio-platform") == 0) { + dev = qdev_device_add(opts); + if (!dev) { + return -1; + } + s = SYS_BUS_DEVICE(dev); + + vfio_get_props(s, pname, pcompat, &num_irqs, &size); + + if (vbi->avail_vfio_base + size >= 0x10000000) { + /* register space size exceeds remaining VFIO space */ + qerror_report(QERR_DEVICE_INIT_FAILED, *pname); + } else if (irq_start + num_irqs >= NUM_IRQS) { + /* VFIO IRQ number exceeded */ + qerror_report(QERR_DEVICE_INIT_FAILED, *pname); + } + + /* + * process compatibility property string passed by end-user + * replaces / by , + * currently a single property compatibility value is supported! + */ + corrected_compat = g_strdup(*pcompat); + char *slash = strchr(corrected_compat, '/'); + *slash = ','; + + sysbus_mmio_map(s, 0, vbi->avail_vfio_base); + + nodename = g_strdup_printf("/%s@%" PRIx64, + *pname, vbi->avail_vfio_base); + qemu_fdt_add_subnode(vbi->fdt, nodename); + + qemu_fdt_setprop(vbi->fdt, nodename, "compatible", + corrected_compat, strlen(corrected_compat)); + + qemu_fdt_setprop_sized_cells(vbi->fdt, nodename, + "reg", 2, vbi->avail_vfio_base, 2, size); + + irq_attr = g_malloc0(num_irqs*3*sizeof(uint32_t)); + for (i = 0; i < num_irqs; i++) { + sysbus_connect_irq(s, i, vbi->pic[irq_start+i]); + + irq_attr[3*i] = cpu_to_be32(0); + irq_attr[3*i+1] = cpu_to_be32(irq_start+i); + irq_attr[3*i+2] = cpu_to_be32(0x4); + } + + qemu_fdt_setprop(vbi->fdt, nodename, "interrupts", + irq_attr, num_irqs*3*sizeof(uint32_t)); + + /* increment base address and IRQ index for next VFIO device */ + vbi->avail_vfio_base += size; + vbi->avail_vfio_irq += num_irqs; + + g_free(pcompat); + g_free(pname); + g_free(nodename); + g_free(corrected_compat); + g_free(irq_attr); + + object_unref(OBJECT(dev)); + + } + + return 0; +} + +/* + * parses the option line and look for -device option + * for each of time vfio_init_func is called. + * this later only applies to -device vfio-platform ones + */ + +static void create_vfio_devices(VirtBoardInfo *vbi) +{ + vbi->avail_vfio_base = vbi->memmap[VIRT_VFIO].base; + vbi->avail_vfio_irq = vbi->irqmap[VIRT_VFIO]; + + if (qemu_opts_foreach(qemu_find_opts("device"), + vfio_init_func, (void *)vbi, 1) != 0) + exit(1); } -static void create_virtio_devices(const VirtBoardInfo *vbi, qemu_irq *pic) + +static void create_virtio_devices(const VirtBoardInfo *vbi) { int i; hwaddr size = vbi->memmap[VIRT_MMIO].size; @@ -335,7 +448,7 @@ static void create_virtio_devices(const VirtBoardInfo *vbi, qemu_irq *pic) int irq = vbi->irqmap[VIRT_MMIO] + i; hwaddr base = vbi->memmap[VIRT_MMIO].base + i * size; - sysbus_create_simple("virtio-mmio", base, pic[irq]); + sysbus_create_simple("virtio-mmio", base, vbi->pic[irq]); } for (i = NUM_VIRTIO_TRANSPORTS - 1; i >= 0; i--) { @@ -366,7 +479,6 @@ static void *machvirt_dtb(const struct arm_boot_info *binfo, int *fdt_size) static void machvirt_init(QEMUMachineInitArgs *args) { - qemu_irq pic[NUM_IRQS]; MemoryRegion *sysmem = get_system_memory(); int n; MemoryRegion *ram = g_new(MemoryRegion, 1); @@ -451,17 +563,19 @@ static void machvirt_init(QEMUMachineInitArgs *args) } for (n = 0; n < NUM_IRQS; n++) { - pic[n] = qdev_get_gpio_in(dev, n); + vbi->pic[n] = qdev_get_gpio_in(dev, n); } - create_uart(vbi, pic); - create_ethernet(vbi, pic); + create_uart(vbi); + + /* create vfio platform devices if any are passed in command line*/ + create_vfio_devices(vbi); /* Create mmio transports, so the user can create virtio backends * (which will be automatically plugged in to the transports). If * no backend is created the transport will just sit harmlessly idle. */ - create_virtio_devices(vbi, pic); + create_virtio_devices(vbi); vbi->bootinfo.ram_size = args->ram_size; vbi->bootinfo.kernel_filename = args->kernel_filename; diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c index f148edd..8f30d41 100644 --- a/hw/vfio/platform.c +++ b/hw/vfio/platform.c @@ -93,6 +93,7 @@ typedef struct VFIODevice { int num_irqs; int interrupt; /* type of the interrupt, might disappear */ char *name; + char *compat; uint32_t mmap_timeout; /* mmap timeout value in ms */ VFIORegion regions[PLATFORM_NUM_REGIONS]; QLIST_ENTRY(VFIODevice) next; @@ -100,6 +101,22 @@ typedef struct VFIODevice { QLIST_HEAD(, VFIOINTp) intp_list; } VFIODevice; +/* + * returns properties from a QEMU VFIO device such as + * name, compatibility, num IRQs, size of the register set + */ +void vfio_get_props(SysBusDevice *s, char **pname, + char **pcompat, int *pnum_irqs, size_t *psize); + +void vfio_get_props(SysBusDevice *s, char **pname, + char **pcompat, int *pnum_irqs, size_t *psize) { + + VFIODevice *vdev = DO_UPCAST(VFIODevice, sbdev, s); + *pname = vdev->name; + *pcompat = vdev->compat; + *pnum_irqs = vdev->num_irqs; + *psize = vdev->regions[0].size; +} static void vfio_unmask_intp(VFIODevice *vdev, int index) @@ -556,11 +573,6 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp) struct stat st; int groupid, i, ret; - - /* TODO: pass device name on command line */ - vdev->name = malloc(PATH_MAX); - strcpy(vdev->name, "fff51000.ethernet"); - /* Check that the host device exists */ snprintf(path, sizeof(path), "/sys/bus/platform/devices/%s/", vdev->name); if (stat(path, &st) < 0) { @@ -568,6 +580,8 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp) return; } + DPRINTF("vfio device %s, compat = %s\n", path, vdev->compat); + strncat(path, "iommu_group", sizeof(path) - strlen(path) - 1); len = readlink(path, iommu_group_path, PATH_MAX); @@ -596,10 +610,15 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp) QLIST_FOREACH(pvdev, &group->device_list, next) { DPRINTF("compare %s versus %s\n", pvdev->name, vdev->name); if (strcmp(pvdev->name, vdev->name) == 0) { - + /* + * in current implementation realize is called twice: + * 1) once in the virt. machine (where qdev stuff are done + + * device tree generation, + * 2) once in vl.c (-device standard handling) + * on 2) realize completes here. + */ DPRINTF("vfio device %s already is attached to group %d\n", vdev->name, groupid); - vfio_put_group(group, NULL); return; } @@ -625,14 +644,22 @@ static const VMStateDescription vfio_platform_vmstate = { .unmigratable = 1, }; +static Property vfio_platform_dev_properties[] = { +DEFINE_PROP_STRING("vfio_device", VFIODevice, name), +DEFINE_PROP_STRING("compat", VFIODevice, compat), +DEFINE_PROP_UINT32("mmap-timeout-ms", VFIODevice, mmap_timeout, 1100), +DEFINE_PROP_END_OF_LIST(), +}; + static void vfio_platform_dev_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); dc->realize = vfio_platform_realize; dc->vmsd = &vfio_platform_vmstate; + dc->props = vfio_platform_dev_properties; dc->desc = "VFIO-based platform device assignment"; - dc->cannot_instantiate_with_device_add_yet = true; + dc->cannot_instantiate_with_device_add_yet = false; set_bit(DEVICE_CATEGORY_MISC, dc->categories); }