[2/2] HACK: qemu: aarch64: Use virtio-pci if user specifies PCI controller

Message ID b23aefb7c6e86f45eec83c77d6045a0c9202bda0.1454013286.git.crobinso@redhat.com
State New
Headers show

Commit Message

Cole Robinson Jan. 28, 2016, 9:14 p.m.
If a user manually specifies this XML snippet for aarch64 machvirt:

  <controller type='pci' index='0' model='pci-root'/>

Libvirt will interpret this to mean that the OS supports virtio-pci,
and will allocate PCI addresses (instead of virtio-mmio) for virtio
devices.

This is a giant hack. Trying to improve it led me into the maze of PCI
address code and I gave up for now. Here are the issues:

* I'd prefer that to be model='pcie-root' which matches what
qemu-system-aarch64 -M virt actually provides by default... however
libvirt isn't happy with a single pcie-root specified by the user, it
will error with:

error: unsupported configuration: failed to create PCI bridge on bus 1: too many devices with fixed addresses

Instead this patch uses hacks to make pci-root use the pcie.0 bus for
aarch64, since that code path already works.

* It may even be nice to make specifying <controller type='pci'/> map to
'give me the recommended PCI setup for this VM'... then we could adjust
what that default means in the future, maybe switching to something like
what q35 uses. This would make apps lives easier.

But presently that violates the XML schema, and libvirt can't handle lack
of model= anyways:

error: internal error: Invalid PCI controller model -1
---
 src/qemu/qemu_command.c                            | 34 ++++++++----
 ...l2argv-aarch64-pci-manual-nocontroller-fail.xml | 43 ++++++++++++++++
 ...l2argv-aarch64-virtio-pci-manual-addresses.args |  2 -
 ...ml2argv-aarch64-virtio-pci-manual-addresses.xml |  1 +
 ...2argv-aarch64-virtio-pci-manual-controller.args | 36 +++++++++++++
 ...l2argv-aarch64-virtio-pci-manual-controller.xml | 50 ++++++++++++++++++
 tests/qemuxml2argvtest.c                           | 20 ++++++--
 ...2xmlout-aarch64-virtio-pci-manual-addresses.xml | 11 +---
 ...xmlout-aarch64-virtio-pci-manual-controller.xml | 60 ++++++++++++++++++++++
 tests/qemuxml2xmltest.c                            |  8 ++-
 10 files changed, 239 insertions(+), 26 deletions(-)
 create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-aarch64-pci-manual-nocontroller-fail.xml
 create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-controller.args
 create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-controller.xml
 create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-aarch64-virtio-pci-manual-controller.xml

-- 
2.5.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Comments

Cole Robinson March 10, 2016, 10:17 p.m. | #1
On 03/10/2016 08:40 AM, Daniel P. Berrange wrote:
> On Thu, Mar 10, 2016 at 02:34:02PM +0100, Andrea Bolognani wrote:

>> On Thu, 2016-03-10 at 09:56 +0000, Daniel P. Berrange wrote:

>>> So, I've just seen that QEMU has decided that as of QEMU 2.6, the virt

>>> machine type will start to be versioned.  This is quite convenient I

>>> think as it gives us a nice thing to hook on. ie we see a non-versioned

>>> machine type of 'virt' then we use virtio-mmio addressing, however, if

>>> we see a versioned virt-X.Y.Z machine type, then we can assume pci by

>>> default.

>>>

>>> Since the long term plan for AArch64 is to use PCI for everything, this

>>> gives us nice default behaviour from this point onwards, while not

>>> breaking compatibility for existing early adopters.

>>>

>>> Of course people with "legacy" mmio-only guests will stll have a little

>>> pain to run then on new QEMU, but honestly I think that's worth it since

>>> it will avoid us long term pain in the world where aarch64 uses pci for

>>> everything

>>

>> I think it's way too early to flip the switch and default to

>> PCI addresses: my understanding is that guest OS support is

>> expected to be spotty at best for at least a couple more

>> years.

> 

> I don't buy that. For a start the only virtio-mmio and/or pci devices we

> are giving to guests are virtio devices (blk, net, balloon, etc). The

> only current guests supporting these devices are Linux, and Linux has

> support for PCI on aarch64. So any Linux distro that ships today is

> going to support PCI. Other guests (*BSD) which may choose to support

> aarch64 in future would likely go straight to PCI and not bother with

> virtio-mmio. Likewise if we did ever get a Windows guest on aarch64

> with win-virtio drivers I would expec them to be PCI based.

> 

> So AFAICT, the only distros which are going to be using virtio-mmio are

> the Linux aarch64 early-access/tech-preview releases. I don't really see

> them as an important target.  Even if we switch to PCI by default, I'd

> still expect virt-manager to be capable of being told to override the

> default and assign mmio addresses instad.

> 


It's not that simple. The kernel supports the devices fine, but there's other
bits in the chain. For example Fedora 23 + AAVMF does not work with
virtio-pci... I don't understand the specifics but it's likely to do with
as-yet-not-fully-upstreamed aarch64 ACPI support that we have in RHELSA. I
don't think any released distro works out of the box with AAVMF + virtio-pci.
CCing Drew who maybe can shed more light on specifics

That said, maybe the simpler thing is to do what you suggest: default to
virtio-pci for versioned virt-*, then teach tools to force <address
type='virtio-mmio'/> via the XML until we know distros can support it.

Means some difficult transition time for the tools in the short to medium
term, since for example an older virt-manager (which doesn't explicitly
specify virtio-mmio) will generate an incorrect XML config for Fedora 23 with
a new libvirt (which defaults to virtio-pci). But maybe it's just best to get
it out of the way now...

- Cole

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Cole Robinson March 10, 2016, 10:46 p.m. | #2
On 03/09/2016 02:09 PM, Laine Stump wrote:
> On 03/09/2016 09:54 AM, Daniel P. Berrange wrote:

>> On Wed, Mar 09, 2016 at 01:40:36PM +0100, Andrea Bolognani wrote:

>>> On Fri, 2016-03-04 at 17:05 -0500, Laine Stump wrote:

>>>>> I'm not sure I fully understand all of the above, but I'll pitch

>>>>> in with my own proposal regardless :)

>>>>>   First, we make sure that

>>>>>        <controller type='pci' index='0' model='pcie-root'/>

>>>>>   is always added automatically to the domain XML when using the

>>>>> mach-virt machine type. Then, if

>>>>>        <controller type='pci' index='1' model='dmi-to-pci-bridge'/>

>>>>>      <controller type='pci' index='2' model='pci-bridge'/>

>>>>>   is present as well we default to virtio-pci, otherwise we use

>>>>> the current default of virtio-mmio. This should allow management

>>>>> applications, based on knowledge about the guest OS, to easily

>>>>> pick between the two address schemes.

>>>>>   Does this sound like a good idea?

>>>> ... or a variation of that, anyway :-)

>>>>   What I think: If there are *any* pci controllers *beyond* pcie-root, or

>>>> if there are any devices that already have a PCI address, then assign

>>>> PCI addresses, else use mmio.

>>> This sounds reasonable.

>>>

>>> However, I'm wondering if we shouldn't be even more explicit about

>>> this... From libvirt's point of view we just need to agree on some

>>> sort of "trigger" that causes it to allocate PCI addresses instead

>>> of MMIO addresses, and for that purpose "any PCI controller or any

>>> device with a PCI address" is perfectly fine; looking at that from

>>> the point of view of an user, though? Not sure about it.

>>>

>>> What about adding something like

>>>

>>>    <preferences>

>>>      <preference name="defaultAddressType" value="pci"/>

>>>    </preferences>

>>>

>>> to the domain XML? AFAICT libvirt doesn't have any element that

>>> could be used for this purpose at the moment, but maybe having a

>>> generic way to set domain-wide preferences could be useful in

>>> other situations as well?

>> [snip]

>>

>> Looking at this mail and laine's before I really get the impression we

>> are over-thinking this all. The automatic address assignment by libvirt

>> was written such that it matched what QEMU would have done, so that we

>> could introduce the concept of device addressing without breaking

>> existing apps which didn't supply addresses. The automatic addressing

>> logic certainly wasn't ever intended to be able to implement arbitrary

>> policies.

>>

>> As a general rule libvirt has always aimed to stay out of the policy

>> business, and focus on providing the mechanism only. So when we start

>> talking about adding  <preferences> to the XML this is a sure sign

>> that we've gone too far into trying to implement policy in libvirt.

>>

>> >From the POV of being able to tell libvirt to assign PCI instead of

>> MMIO addresses for the virt machine, I think it suffices to have

>> libvirt accept a simple  <address type="pci"/> element (ie with no

>> bus/slot/func filled in).

> 

> That's just a more limited case of the same type of feature creep though -

> you're specifying a preference for what type of address you want, just in a

> different place. (The odd thing is that we're adding it only to remedy what is

> apparently a temporary problem, and even then it's a problem that wouldn't

> exist if we just said simply this:  "If virtio-mmio is specified, then use

> virtio-mmio; If no address is specified, use pci". This puts the burden on

> people insisting on the old / soon-to-be-deprecated (?) virtio-mmio address

> type to add a little something to the XML (and a very simple little something

> at that!). For a short time, this could cause problems for people who create

> new domains using qemu binaries that don't have a pci bus on aarch64/virt yet,

> but that will be easily solvable (by adding "<address type='virtio-mmio'/>",

> which is nearly as simple as typing "<address type='pci'/>").

> 

> 

> A downside of setting your preference with <address type="pci"/> vs. having

> the preference in a separate element is that you can no longer cause a

> "re-addressing" of all the devices by simply removing all the <address>

> elements (and it's really that that got me thinking about adding

> hints/preferences in the <target> element). I don't know how important that

> is; I suppose when the producer of the XML is some other software it's a

> non-issue, when the producer is a human using virsh edit it would make life

> much simpler once in awhile (and the suggestion in the previous paragraph

> would eliminate that problem anyway).

> 

> So is there a specific reason why we need to keep virtio-mmio as the default,

> and require explicitly saying "address type='pci'" in order to get a PCI address?

> 


I explained this in my reply to Dan just now, but basically all currently
released distros don't work with virtio-pci. However long term I've suggested
from the start that we switch to virtio-pci by default, keying off a new
enough -M virt version, as an arbitrary marker in time when hopefully most
common distros work with virtio-pci

>> If applictions care about assigning devices to specify PCI buses,

>> ie to distinguish between PCI & PCIe, or to pick a different bus

>> when there is one bus per NUMA node, then really it is time for

>> the application to start specifying the full <address> element

>> with all details, and not try to invent ever more complex policies

>> inside libvirt which will never deal with all application use cases.

> 

> First, I agree with your vigilance against unnecessary feature creep. Both

> because keeping things simple makes it easier to maintain and use, and also

> because it's very difficult (or even impossible) to change something once it's

> gone in, in the case that you have second thoughts.

> 

> But, expanding beyond just the aarch64/virt mmio vs. pci problem (which I had

> already done in this thread, and which is what got us into this "feature

> creep" discussion in the first place :-)...

> 

> I remember when we were first adding support for Q35 that we said it should be

> as easy to create a domain with Q35 machinetype as it is for 440fx, i.e. you

> should be able to do it without manually specifying any PCI addresses (that's

> why we have the strange default bus hierarchy, with a dmi-to-pci-bridge and a

> pci-bridge, and all devices going onto the pci-bridge). But of course 440fx is

> very simplistic - all slots are standard PCI and all are hotpluggable. On Q35

> there could be room for choices though, in particular PCI vs PCIe and

> hotpluggable vs. not. (and yeah, also what NUMA node the bus is on; I agree

> that one is "getting out there"). So since there are choices, libvirt has by

> definition begun implementing policy when it auto-assigns *any* PCI address

> (current policy is "require all auto-assigned device addresses to be

> hotpluggable standard PCI). And if libvirt doesn't provide an easy way to

> choose, it will have implemented a defacto mandatory policy (unless you force

> full specification of PCI addresses, which goes against the "make it easy" goal).

> 

> And the current policy is looking (for Q35 and aarch64/virt at least) less and

> less like the correct one - in the years since it was put in, 1) we've learned

> that qemu doesn't care, and will not ever care, that a PCI device has been

> plugged into a PCIe slot, 2) we now have support for several PCIe controllers

> we didn't previously have, 3) people have started complaining that their

> devices are in PCI slots rather than PCIe, 4) even though it was stated at the

> time I wrote the code to auto-add a pci-bridge to every Q35 domain that

> pci-bridge's failure to support hotplug was a temporary bug, I just learned

> yesterday that it apparently still doesn't work, and 5) some platforms (e.g.

> our favorite aarch64/virt) are emulating a hardware platform that has *never*

> had standard PCI slots, only PCIe.

> 

> Beyond that, there is no place that provides a simple encoding of which type

> of controller provides which type of slot, and what is allowed to plug into

> what. If you require the management application/human to manually specify all

> the PCI addresses as soon as they have a need for one of these basic

> characteristics, then not only has it become cumbersome to define a domain

> (because the management app has to maintain a data structure to keep track of

> which PCI addresses are in use and which aren't), but it means that the

> management application also needs to know all sorts of rules about which PCI

> controllers are actually pcie vs. pci, and which accept hotplug devices vs.

> which don't, as well as things like the min/max slot number for each

> controller, and which ones can plug into where, e.g. a pcie-root-port can only

> plug into pcie-root or pcie-expander-bus, and a pcie-switch-downstream-port

> can only plug into a pcie-switch-upstream-port, etc. Requiring a management

> app to get all of that right just so that they can pick between a hotpluggable

> and non-hotpluggable slot seems like an overly large burden (and prone to error).

> 

> In the end, if libvirt makes it simple for the management app to specify what

> kind of slot it wants, rather than requiring it to specify the exact slot,

> then libvirt isn't making any policy decisions, it's just making it easier for

> the management app to implement its own policy decisions, without requiring

> the management app to know all the details about which controller implements

> what kind of connection etc, and that does seem to fit within libvirt's purpose.


I guess the main thing would be to understand usecases... outside of aarch64
is there a real reason for q35  that apps might need one address policy over
another? If it's something super obscure, like only for testing or dev, then I
think asking people to do it manually is fine.

At least for the arm case I think we can side step the question by adding the
<address type='pci'/> allocation request, and flipping the default from
virtio-mmio to virtio-pci at some future point

- Cole

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Patch

diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c
index 8943270..4b0f070 100644
--- a/src/qemu/qemu_command.c
+++ b/src/qemu/qemu_command.c
@@ -1064,7 +1064,10 @@  qemuAssignDeviceControllerAlias(virDomainDefPtr domainDef,
          * (including the hardcoded pci-root controller on
          * multibus-capable qemus).
          */
-        return virAsprintf(&controller->info.alias, "pci.%d", controller->idx);
+        if (domainDef->os.arch == VIR_ARCH_AARCH64)
+            return virAsprintf(&controller->info.alias, "pcie.%d", controller->idx);
+        else
+            return virAsprintf(&controller->info.alias, "pci.%d", controller->idx);
     } else if (controller->type == VIR_DOMAIN_CONTROLLER_TYPE_IDE) {
         /* for any machine based on e.g. I440FX or G3Beige, the
          * first (and currently only) IDE controller is an integrated
@@ -1393,15 +1396,28 @@  static int
 qemuDomainAssignARMVirtioMMIOAddresses(virDomainDefPtr def,
                                        virQEMUCapsPtr qemuCaps)
 {
-    if (((def->os.arch == VIR_ARCH_ARMV7L) ||
-        (def->os.arch == VIR_ARCH_AARCH64)) &&
-        (STRPREFIX(def->os.machine, "vexpress-") ||
-            STREQ(def->os.machine, "virt") ||
-            STRPREFIX(def->os.machine, "virt-")) &&
-        virQEMUCapsGet(qemuCaps, QEMU_CAPS_DEVICE_VIRTIO_MMIO)) {
-        qemuDomainPrimeVirtioDeviceAddresses(
-            def, VIR_DOMAIN_DEVICE_ADDRESS_TYPE_VIRTIO_MMIO);
+    size_t i;
+
+    if ((def->os.arch != VIR_ARCH_ARMV7L) &&
+        (def->os.arch != VIR_ARCH_AARCH64))
+        return 0;
+
+    if (!(STRPREFIX(def->os.machine, "vexpress-") ||
+          STREQ(def->os.machine, "virt") ||
+          STRPREFIX(def->os.machine, "virt-")))
+        return 0;
+
+    if (!virQEMUCapsGet(qemuCaps, QEMU_CAPS_DEVICE_VIRTIO_MMIO))
+        return 0;
+
+    /* If there's a PCIe controller in the XML, we will use PCI virtio */
+    for (i = 0; i < def->ncontrollers; i++) {
+        if (def->controllers[i]->type == VIR_DOMAIN_CONTROLLER_TYPE_PCI)
+            return 0;
     }
+
+    qemuDomainPrimeVirtioDeviceAddresses(
+        def, VIR_DOMAIN_DEVICE_ADDRESS_TYPE_VIRTIO_MMIO);
     return 0;
 }
 
diff --git a/tests/qemuxml2argvdata/qemuxml2argv-aarch64-pci-manual-nocontroller-fail.xml b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-pci-manual-nocontroller-fail.xml
new file mode 100644
index 0000000..6a44f19
--- /dev/null
+++ b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-pci-manual-nocontroller-fail.xml
@@ -0,0 +1,43 @@ 
+<domain type='qemu'>
+  <name>aarch64test</name>
+  <uuid>496d7ea8-9739-544b-4ebd-ef08be936e8b</uuid>
+  <memory unit='KiB'>1048576</memory>
+  <currentMemory unit='KiB'>1048576</currentMemory>
+  <vcpu placement='static'>1</vcpu>
+  <os>
+    <type arch='aarch64' machine='virt'>hvm</type>
+    <kernel>/aarch64.kernel</kernel>
+    <initrd>/aarch64.initrd</initrd>
+    <cmdline>earlyprintk console=ttyAMA0,115200n8 rw root=/dev/vda rootwait</cmdline>
+    <dtb>/aarch64.dtb</dtb>
+    <boot dev='hd'/>
+  </os>
+  <features>
+    <acpi/>
+    <apic/>
+    <pae/>
+  </features>
+  <cpu mode='custom' match='exact'>
+    <model fallback='allow'>cortex-a53</model>
+  </cpu>
+  <clock offset='utc'/>
+  <on_poweroff>destroy</on_poweroff>
+  <on_reboot>restart</on_reboot>
+  <on_crash>restart</on_crash>
+  <devices>
+    <emulator>/usr/bin/qemu-system-aarch64</emulator>
+    <disk type='file' device='disk'>
+      <source file='/aarch64.raw'/>
+      <target dev='sda' bus='scsi'/>
+      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
+    </disk>
+    <controller type='scsi' index='0' model='virtio-scsi'>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
+    </controller>
+    <interface type='user'>
+      <mac address='52:54:00:09:a4:37'/>
+      <model type='virtio'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
+    </interface>
+  </devices>
+</domain>
diff --git a/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-addresses.args b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-addresses.args
index 142fd5b..f091c89 100644
--- a/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-addresses.args
+++ b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-addresses.args
@@ -21,8 +21,6 @@  QEMU_AUDIO_DRV=none \
 -initrd /aarch64.initrd \
 -append 'earlyprintk console=ttyAMA0,115200n8 rw root=/dev/vda rootwait' \
 -dtb /aarch64.dtb \
--device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1 \
--device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 \
 -device virtio-scsi-pci,id=scsi0,bus=pcie.0,addr=0x3 \
 -usb \
 -drive file=/aarch64.raw,format=raw,if=none,id=drive-scsi0-0-0-0 \
diff --git a/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-addresses.xml b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-addresses.xml
index 6a44f19..80a6592 100644
--- a/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-addresses.xml
+++ b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-addresses.xml
@@ -31,6 +31,7 @@ 
       <target dev='sda' bus='scsi'/>
       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
     </disk>
+    <controller type='pci' index='0' model='pci-root'/>
     <controller type='scsi' index='0' model='virtio-scsi'>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
     </controller>
diff --git a/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-controller.args b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-controller.args
new file mode 100644
index 0000000..dfa07da
--- /dev/null
+++ b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-controller.args
@@ -0,0 +1,36 @@ 
+LC_ALL=C \
+PATH=/bin \
+HOME=/home/test \
+USER=test \
+LOGNAME=test \
+QEMU_AUDIO_DRV=none \
+/usr/bin/qemu-system-aarch64 \
+-name aarch64test \
+-S \
+-M virt \
+-cpu cortex-a53 \
+-m 1024 \
+-smp 1 \
+-uuid 496d7ea8-9739-544b-4ebd-ef08be936e8b \
+-nographic \
+-nodefconfig \
+-nodefaults \
+-monitor unix:/tmp/test-monitor,server,nowait \
+-boot c \
+-kernel /aarch64.kernel \
+-initrd /aarch64.initrd \
+-append 'earlyprintk console=ttyAMA0,115200n8 rw root=/dev/vda rootwait' \
+-dtb /aarch64.dtb \
+-device virtio-serial-pci,id=virtio-serial0,bus=pcie.0,addr=0x2 \
+-usb \
+-drive file=/aarch64.raw,format=raw,if=none,id=drive-virtio-disk0 \
+-device virtio-blk-pci,bus=pcie.0,addr=0x3,drive=drive-virtio-disk0,\
+id=virtio-disk0 \
+-device virtio-net-pci,vlan=0,id=net0,mac=52:54:00:09:a4:37,bus=pcie.0,addr=0x1 \
+-net user,vlan=0,name=hostnet0 \
+-serial pty \
+-chardev pty,id=charconsole1 \
+-device virtconsole,chardev=charconsole1,id=console1 \
+-device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x4 \
+-object rng-random,id=objrng0,filename=/dev/random \
+-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pcie.0,addr=0x5
diff --git a/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-controller.xml b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-controller.xml
new file mode 100644
index 0000000..c5ea728
--- /dev/null
+++ b/tests/qemuxml2argvdata/qemuxml2argv-aarch64-virtio-pci-manual-controller.xml
@@ -0,0 +1,50 @@ 
+<domain type='qemu'>
+  <name>aarch64test</name>
+  <uuid>496d7ea8-9739-544b-4ebd-ef08be936e8b</uuid>
+  <memory unit='KiB'>1048576</memory>
+  <currentMemory unit='KiB'>1048576</currentMemory>
+  <vcpu placement='static'>1</vcpu>
+  <os>
+    <type arch='aarch64' machine='virt'>hvm</type>
+    <kernel>/aarch64.kernel</kernel>
+    <initrd>/aarch64.initrd</initrd>
+    <cmdline>earlyprintk console=ttyAMA0,115200n8 rw root=/dev/vda rootwait</cmdline>
+    <dtb>/aarch64.dtb</dtb>
+    <boot dev='hd'/>
+  </os>
+  <features>
+    <acpi/>
+    <apic/>
+    <pae/>
+  </features>
+  <cpu mode='custom' match='exact'>
+    <model fallback='allow'>cortex-a53</model>
+  </cpu>
+  <clock offset='utc'/>
+  <on_poweroff>destroy</on_poweroff>
+  <on_reboot>restart</on_reboot>
+  <on_crash>restart</on_crash>
+  <devices>
+    <emulator>/usr/bin/qemu-system-aarch64</emulator>
+    <disk type='file' device='disk'>
+      <source file='/aarch64.raw'/>
+      <target dev='vda' bus='virtio'/>
+    </disk>
+    <controller type='pci' index='0' model='pci-root'/>
+    <interface type='user'>
+      <mac address='52:54:00:09:a4:37'/>
+      <model type='virtio'/>
+    </interface>
+    <console type='pty'/>
+    <console type='pty'>
+      <target type='virtio' port='0'/>
+    </console>
+    <memballoon model='virtio'/>
+    <!--
+      This actually doesn't work in practice because vexpress only has
+      4 virtio slots available, rng makes 5 -->
+    <rng model='virtio'>
+      <backend model='random'>/dev/random</backend>
+    </rng>
+  </devices>
+</domain>
diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c
index b2f636e..6c5a8ab 100644
--- a/tests/qemuxml2argvtest.c
+++ b/tests/qemuxml2argvtest.c
@@ -1649,9 +1649,23 @@  mymain(void)
             QEMU_CAPS_DEVICE_VIRTIO_RNG, QEMU_CAPS_OBJECT_RNG_RANDOM,
             QEMU_CAPS_OBJECT_GPEX, QEMU_CAPS_DEVICE_PCI_BRIDGE,
             QEMU_CAPS_DEVICE_DMI_TO_PCI_BRIDGE);
-    /* Example of using virtio-pci with no explicit PCI controller
-       but with manual PCI addresses */
-    DO_TEST_PARSE_ERROR("aarch64-virtio-pci-manual-addresses",
+    /* Example of using virtio-pci with manual PCIe controller and auto
+       allocated addresses */
+    DO_TEST("aarch64-virtio-pci-manual-controller",
+            QEMU_CAPS_NODEFCONFIG, QEMU_CAPS_DTB,
+            QEMU_CAPS_DEVICE_VIRTIO_MMIO,
+            QEMU_CAPS_DEVICE_VIRTIO_RNG, QEMU_CAPS_OBJECT_RNG_RANDOM,
+            QEMU_CAPS_OBJECT_GPEX, QEMU_CAPS_DEVICE_PCI_BRIDGE,
+            QEMU_CAPS_DEVICE_DMI_TO_PCI_BRIDGE, QEMU_CAPS_VIRTIO_SCSI);
+    /* Example of using virtio-pci with manual PCIe controller and
+       manual PCI addresses */
+    DO_TEST("aarch64-virtio-pci-manual-addresses",
+            QEMU_CAPS_NODEFCONFIG, QEMU_CAPS_DTB,
+            QEMU_CAPS_DEVICE_VIRTIO_MMIO,
+            QEMU_CAPS_DEVICE_VIRTIO_RNG, QEMU_CAPS_OBJECT_RNG_RANDOM,
+            QEMU_CAPS_OBJECT_GPEX, QEMU_CAPS_DEVICE_PCI_BRIDGE,
+            QEMU_CAPS_DEVICE_DMI_TO_PCI_BRIDGE, QEMU_CAPS_VIRTIO_SCSI);
+    DO_TEST_PARSE_ERROR("aarch64-pci-manual-nocontroller-fail",
             QEMU_CAPS_NODEFCONFIG, QEMU_CAPS_DTB,
             QEMU_CAPS_DEVICE_VIRTIO_MMIO,
             QEMU_CAPS_DEVICE_VIRTIO_RNG, QEMU_CAPS_OBJECT_RNG_RANDOM,
diff --git a/tests/qemuxml2xmloutdata/qemuxml2xmlout-aarch64-virtio-pci-manual-addresses.xml b/tests/qemuxml2xmloutdata/qemuxml2xmlout-aarch64-virtio-pci-manual-addresses.xml
index 4add271..80a6592 100644
--- a/tests/qemuxml2xmloutdata/qemuxml2xmlout-aarch64-virtio-pci-manual-addresses.xml
+++ b/tests/qemuxml2xmloutdata/qemuxml2xmlout-aarch64-virtio-pci-manual-addresses.xml
@@ -31,19 +31,10 @@ 
       <target dev='sda' bus='scsi'/>
       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
     </disk>
+    <controller type='pci' index='0' model='pci-root'/>
     <controller type='scsi' index='0' model='virtio-scsi'>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
     </controller>
-    <controller type='pci' index='0' model='pcie-root'/>
-    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
-      <model name='i82801b11-bridge'/>
-      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
-    </controller>
-    <controller type='pci' index='2' model='pci-bridge'>
-      <model name='pci-bridge'/>
-      <target chassisNr='2'/>
-      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
-    </controller>
     <interface type='user'>
       <mac address='52:54:00:09:a4:37'/>
       <model type='virtio'/>
diff --git a/tests/qemuxml2xmloutdata/qemuxml2xmlout-aarch64-virtio-pci-manual-controller.xml b/tests/qemuxml2xmloutdata/qemuxml2xmlout-aarch64-virtio-pci-manual-controller.xml
new file mode 100644
index 0000000..7be2cbd
--- /dev/null
+++ b/tests/qemuxml2xmloutdata/qemuxml2xmlout-aarch64-virtio-pci-manual-controller.xml
@@ -0,0 +1,60 @@ 
+<domain type='qemu'>
+  <name>aarch64test</name>
+  <uuid>496d7ea8-9739-544b-4ebd-ef08be936e8b</uuid>
+  <memory unit='KiB'>1048576</memory>
+  <currentMemory unit='KiB'>1048576</currentMemory>
+  <vcpu placement='static'>1</vcpu>
+  <os>
+    <type arch='aarch64' machine='virt'>hvm</type>
+    <kernel>/aarch64.kernel</kernel>
+    <initrd>/aarch64.initrd</initrd>
+    <cmdline>earlyprintk console=ttyAMA0,115200n8 rw root=/dev/vda rootwait</cmdline>
+    <dtb>/aarch64.dtb</dtb>
+    <boot dev='hd'/>
+  </os>
+  <features>
+    <acpi/>
+    <apic/>
+    <pae/>
+  </features>
+  <cpu mode='custom' match='exact'>
+    <model fallback='allow'>cortex-a53</model>
+  </cpu>
+  <clock offset='utc'/>
+  <on_poweroff>destroy</on_poweroff>
+  <on_reboot>restart</on_reboot>
+  <on_crash>restart</on_crash>
+  <devices>
+    <emulator>/usr/bin/qemu-system-aarch64</emulator>
+    <disk type='file' device='disk'>
+      <source file='/aarch64.raw'/>
+      <target dev='vda' bus='virtio'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
+    </disk>
+    <controller type='pci' index='0' model='pci-root'/>
+    <controller type='virtio-serial' index='0'>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
+    </controller>
+    <interface type='user'>
+      <mac address='52:54:00:09:a4:37'/>
+      <model type='virtio'/>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
+    </interface>
+    <serial type='pty'>
+      <target port='0'/>
+    </serial>
+    <console type='pty'>
+      <target type='serial' port='0'/>
+    </console>
+    <console type='pty'>
+      <target type='virtio' port='1'/>
+    </console>
+    <memballoon model='virtio'>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
+    </memballoon>
+    <rng model='virtio'>
+      <backend model='random'>/dev/random</backend>
+      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
+    </rng>
+  </devices>
+</domain>
diff --git a/tests/qemuxml2xmltest.c b/tests/qemuxml2xmltest.c
index cac401c..e7f8845 100644
--- a/tests/qemuxml2xmltest.c
+++ b/tests/qemuxml2xmltest.c
@@ -731,14 +731,18 @@  mymain(void)
             QEMU_CAPS_DEVICE_VIRTIO_RNG, QEMU_CAPS_OBJECT_RNG_RANDOM,
             QEMU_CAPS_OBJECT_GPEX, QEMU_CAPS_DEVICE_PCI_BRIDGE,
             QEMU_CAPS_DEVICE_DMI_TO_PCI_BRIDGE, QEMU_CAPS_VIRTIO_SCSI);
-    /*
+    DO_TEST_FULL("aarch64-virtio-pci-manual-controller", WHEN_ACTIVE,
+            QEMU_CAPS_NODEFCONFIG, QEMU_CAPS_DTB,
+            QEMU_CAPS_DEVICE_VIRTIO_MMIO,
+            QEMU_CAPS_DEVICE_VIRTIO_RNG, QEMU_CAPS_OBJECT_RNG_RANDOM,
+            QEMU_CAPS_OBJECT_GPEX, QEMU_CAPS_DEVICE_PCI_BRIDGE,
+            QEMU_CAPS_DEVICE_DMI_TO_PCI_BRIDGE, QEMU_CAPS_VIRTIO_SCSI);
     DO_TEST_FULL("aarch64-virtio-pci-manual-addresses", WHEN_ACTIVE,
             QEMU_CAPS_NODEFCONFIG, QEMU_CAPS_DTB,
             QEMU_CAPS_DEVICE_VIRTIO_MMIO,
             QEMU_CAPS_DEVICE_VIRTIO_RNG, QEMU_CAPS_OBJECT_RNG_RANDOM,
             QEMU_CAPS_OBJECT_GPEX, QEMU_CAPS_DEVICE_PCI_BRIDGE,
             QEMU_CAPS_DEVICE_DMI_TO_PCI_BRIDGE, QEMU_CAPS_VIRTIO_SCSI);
-    */
     DO_TEST("aarch64-gic");
     DO_TEST("aarch64-gicv3");