| Message ID | 1500296815-10243-1-git-send-email-bhupinder.thakur@linaro.org |
|---|---|
| Headers | show |
| Series | SBSA UART emulation support in Xen | expand |
Hi Bhupinder, I've tried this series today on an ARM64 platform. When I enable pl011 for the guest, I am not able to fully destroy the guest. It stay in zombie mode: 42sh> xl list Name ID Mem VCPUs State Time(s) Domain-0 0 3072 2 r----- 62.1 (null) 6 0 2 --p--d 1.5 It does not happen when I don't have pl011 enabled in the guest config. The step to reproduce it is: 42sh> xl create guest.cfg Parsing config from guest.cfg 42sh> xl list Name ID Mem VCPUs State Time(s) Domain-0 0 3072 2 r----- 12.9 guest 1 3265 1 r----- 0.1 42sh> xl destroy guest 42sh> xl list Name ID Mem VCPUs State Time(s) Domain-0 0 3072 2 r----- 14.9 (null) 1 0 1 --p--d 0.1 And my guest.cfg is: 42sh> cat guest.cfg kernel="/home/julien/works/guest/Image" name="guest" memory=3265 vcpus=2 vuart="sbsa_uart" I haven't dug into the problem but I would look at how you unmap the ring from Xen and xenconsole. Likely we still have a reference on it. Let me know if you need any help. Cheers, On 17/07/17 14:06, Bhupinder Thakur wrote: > SBSA UART emulation for guests in Xen > ====================================== > Linaro has published VM System specification for ARM Processors, which > provides a set of guidelines for both guest OS and hypervisor implementations, > such that building OS images according to these guidelines guarantees > that those images can also run on hypervisors compliant with this specification. > > One of the spec requirements is that the hypervisor must provide an > emulated SBSA UART as a serial console which meets the minimum requirements in > SBSA UART as defined in appendix B of the following > ARM Server Base Architecture Document: > > https://static.docs.arm.com/den0029/a/Server_Base_System_Architecture_v3_1_ARM_DEN_0029A.pdf. > > This feature allows the Xen guests to use SBSA compliant pl011 UART as > as a console. > > Note that SBSA pl011 UART is a subset of full featured ARM pl011 UART and > supports only a subset of registers as mentioned below. It does not support > rx/tx DMA. > > Currently, Xen supports paravirtualized (aka PV console) and an emulated serial > consoles. This feature will expose an emulated SBSA pl011 UART console to the > guest, which a user can access using xenconsole. > > The device tree passed to the guest VM will contain the pl011 MMIO address > range and an irq for receiving rx/tx pl011 interrupts. The device tree format > is specified in Documentation/devicetree/bindings/serial/arm_sbsa_uart.txt. > > The Xen hypervisor will expose two types of interfaces to the backend and domU. > > The interface exposed to domU will be an emulated pl011 UART by emulating the > access to the following pl011 registers by the guest. > > - Data register (DR) - RW > - Raw interrupt status register (RIS) - RO > - Masked interrupt status register (MIS)- RO > - Interrupt Mask (IMSC) - RW > - Interrupt Clear (ICR) - WO > > It will also inject the pl011 interrupts to the guest in the following > conditions: > > - incoming data in the rx buffer for the guest > - there is space in the tx buffer for the guest to write more data > > The interface exposed to the backend will be the same PV console interface, > which minimizes the changes required in xenconsole to support a new pl011 console. > > This interface has rx and tx ring buffers and an event channel for > sending/receiving events from the backend. > > So essentially Xen handles the data on behalf of domU and the backend. Any data > written by domU is captured by Xen and written to the TX (OUT) ring buffer > and a pl011 event is raised to the backend to read the TX ring buffer. > > Similarly on reciving a pl011 event, Xen injects an interrupt to guest to > indicate there is data available in the RX (IN) ring buffer. > > The pl011 UART state is completely captured in the set of registers > mentioned above and this state is updated everytime there is an event from > the backend or there is register read/write access from domU. > > For example, if domU has masked the rx interrupt in the IMSC register, then Xen > will not inject an interrupt to guest and will just update the RIS register. > Once the interrupt is unmasked by guest, the interrupt will be delivered to the > guest. > > Changes summary: > > Xen Hypervisor > =============== > > 1. Add emulation code to emulate read/write access to pl011 registers and pl011 > interrupts: > - It emulates DR read/write by reading and writing from/to the IN and > OUT ring buffers and raising an event to dom0 when there is data in > the OUT ring buffer and injecting an interrupt to the guest when there > is data in the IN ring buffer. > - Other registers are related to interrupt management and essentially > control when interrupts are delivered to the guest. > > 2. Add a new domctl API to initialize vpl011 emulation in Xen. > > 3. Enable vpl011 emulation for a domain based on a libxl option passed during > domain creation. > > Toolstack > ========== > > 1. Add a new option "vuart" in the domU configuration file to enable/disable vuart. > > 2. Create a SBSA UART DT node in the guest device tree. It uses a fixed > vpl011 SPI IRQ number and MMIO address. > > 3. Call vpl011 init DOMCTL API to enable vpl011 emulation. > > 5. Add a new vuart xenstore node, which contains: > - ring-ref > - event channel > - buffer limit > - type > > Xenconsoled > ============ > > 1. Split the domain structure to support multiple consoles. > > 2. Modify different APIs such as buffer_append() etc. to operate on the > console structure. > > 3. Add support for handling multiple consoles. > > 4. Add support for vuart console: > > The vpl011 changes available at the following repo: > > url: https://git@git.linaro.org:/people/bhupinder.thakur/xen.git > branch: vpl011_v6 > > Kindly wait for one day to checkout the code from the above URL. > > There are some TBD items which need to be looked at in the future: > > 1. Currently UEFI firmware logs the output to hvc console only. How can > UEFI firmware be made aware of pl011 console and how it can use it > as a console instead of hvc. > > There was a discussion on this and it was decided that SBSA UART should > be used as a debug port by the UEFI firmware so that all debug output > is redirected to this port. > > 2. Linux seems to have hvc console as the default console i.e. if no > console is specified then it uses hvc as the console. How can an > option be provided in Linux to select either hvc or pl011 as the > default console. > > It was suggeted to use the SPCR in ACPI and the stdout-path option in the > device tree to specify the default console. However, currently hvc console > is not describable in the ACPI/device tree. This support will have to be > added to allow the user to specify the default console. > > 3. ACPI support for pl011 device. > > CC: Ian Jackson <ian.jackson@eu.citrix.com> > CC: Wei Liu <wei.liu2@citrix.com> > CC: Stefano Stabellini <sstabellini@kernel.org> > CC: Julien Grall <julien.grall@arm.com> > CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > > > Bhupinder Thakur (25): > xen/arm: vpl011: Define common ring buffer helper functions in > console.h > xen/arm: vpl011: Add SBSA UART emulation in Xen > xen/arm: vpl011: Allocate a new GFN in the toolstack for vuart > xen/arm: vpl011: Add support for vuart in libxl > xen/arm: vpl011: Rearrange xen header includes in alphabetical order > in domctl.c > xen/arm: vpl011: Add a new domctl API to initialize vpl011 > xen/arm: vpl011: Add a new vuart node in the xenstore > xen/arm: vpl011: Modify xenconsole to define and use a new console > structure > xen/arm: vpl011: Rename the console structure field conspath to xspath > xen/arm: vpl011: Modify xenconsole functions to take console structure > as input > xen/arm: vpl011: Add a new console_init function in xenconsole > xen/arm: vpl011: Add a new buffer_available function in xenconsole > xen/arm: vpl011: Add a new add_console_evtchn_fd function in > xenconsole > xen/arm: vpl011: Add a new add_console_tty_fd function in xenconsole > xen/arm: vpl011: Add a new console_evtchn_unmask function in > xenconsole > xen/arm: vpl011: Add a new handle_console_ring function in xenconsole > xen/arm: vpl011: Add a new handle_console_tty function in xenconsole > xen/arm: vpl011: Add a new console_cleanup function in xenconsole > xen/arm: vpl011: Add a new console_open_log function in xenconsole > xen/arm: vpl011: Add a new console_close_evtchn function in xenconsole > xen/arm: vpl011: Add support for multiple consoles in xenconsole > xen/arm: vpl011: Add support for vuart console in xenconsole > xen/arm: vpl011: Add a new vuart console type to xenconsole client > xen/arm: vpl011: Add a pl011 uart DT node in the guest device tree > xen/arm: vpl011: Update documentation for vuart console support > > config/arm32.mk | 1 + > config/arm64.mk | 1 + > docs/man/xl.cfg.pod.5.in | 9 + > docs/misc/console.txt | 44 ++- > tools/console/Makefile | 3 +- > tools/console/client/main.c | 13 +- > tools/console/daemon/io.c | 662 ++++++++++++++++++++++++----------- > tools/libxc/include/xc_dom.h | 2 + > tools/libxc/include/xenctrl.h | 18 + > tools/libxc/xc_dom_arm.c | 5 +- > tools/libxc/xc_dom_boot.c | 2 + > tools/libxc/xc_domain.c | 24 ++ > tools/libxl/libxl.h | 6 + > tools/libxl/libxl_arch.h | 6 + > tools/libxl/libxl_arm.c | 72 +++- > tools/libxl/libxl_console.c | 47 +++ > tools/libxl/libxl_create.c | 9 +- > tools/libxl/libxl_device.c | 9 +- > tools/libxl/libxl_dom.c | 5 + > tools/libxl/libxl_internal.h | 6 + > tools/libxl/libxl_types.idl | 7 + > tools/libxl/libxl_types_internal.idl | 1 + > tools/libxl/libxl_x86.c | 8 + > tools/xl/xl_cmdtable.c | 2 +- > tools/xl/xl_console.c | 5 +- > tools/xl/xl_parse.c | 8 + > xen/arch/arm/Kconfig | 7 + > xen/arch/arm/Makefile | 1 + > xen/arch/arm/domain.c | 5 + > xen/arch/arm/domctl.c | 43 ++- > xen/arch/arm/vpl011.c | 455 ++++++++++++++++++++++++ > xen/include/asm-arm/domain.h | 6 + > xen/include/asm-arm/pl011-uart.h | 2 + > xen/include/asm-arm/vpl011.h | 72 ++++ > xen/include/public/arch-arm.h | 6 + > xen/include/public/domctl.h | 21 ++ > xen/include/public/io/console.h | 4 + > 37 files changed, 1360 insertions(+), 237 deletions(-) > create mode 100644 xen/arch/arm/vpl011.c > create mode 100644 xen/include/asm-arm/vpl011.h >
Hi Julien, The issue seems to be that domain_vpl011_deinit() is not getting called when we destroy the domain. On further debugging, i found that domain_relinquish_resources was failing in domain_kill() function. I believe that issue could be that arch_domain_destroy() (from where domain_vpl011_deinit() is called) is called later and because domain_kill() fails, it is not getting called. Once I moved the domain_vpl011_deinit() inside domain_kill(), the domain is getting cleaned up properly. Is it ok to call domain_vpl011_deinit() inside domain_kill? Regards, Bhupinder On 19 July 2017 at 20:57, Julien Grall <julien.grall@arm.com> wrote: > Hi Bhupinder, > > I've tried this series today on an ARM64 platform. When I enable pl011 for the guest, > I am not able to fully destroy the guest. It stay in zombie mode: > > 42sh> xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 3072 2 r----- 62.1 > (null) 6 0 2 --p--d 1.5 > > It does not happen when I don't have pl011 enabled in the guest config. > > The step to reproduce it is: > > 42sh> xl create guest.cfg > Parsing config from guest.cfg > 42sh> xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 3072 2 r----- 12.9 > guest 1 3265 1 r----- 0.1 > 42sh> xl destroy guest > 42sh> xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 3072 2 r----- 14.9 > (null) 1 0 1 --p--d 0.1 > > And my guest.cfg is: > > 42sh> cat guest.cfg > kernel="/home/julien/works/guest/Image" > name="guest" > memory=3265 > vcpus=2 > vuart="sbsa_uart" > > I haven't dug into the problem but I would look at how you unmap the ring from Xen > and xenconsole. Likely we still have a reference on it. > > Let me know if you need any help. > > Cheers, > > On 17/07/17 14:06, Bhupinder Thakur wrote: >> SBSA UART emulation for guests in Xen >> ====================================== >> Linaro has published VM System specification for ARM Processors, which >> provides a set of guidelines for both guest OS and hypervisor implementations, >> such that building OS images according to these guidelines guarantees >> that those images can also run on hypervisors compliant with this specification. >> >> One of the spec requirements is that the hypervisor must provide an >> emulated SBSA UART as a serial console which meets the minimum requirements in >> SBSA UART as defined in appendix B of the following >> ARM Server Base Architecture Document: >> >> https://static.docs.arm.com/den0029/a/Server_Base_System_Architecture_v3_1_ARM_DEN_0029A.pdf. >> >> This feature allows the Xen guests to use SBSA compliant pl011 UART as >> as a console. >> >> Note that SBSA pl011 UART is a subset of full featured ARM pl011 UART and >> supports only a subset of registers as mentioned below. It does not support >> rx/tx DMA. >> >> Currently, Xen supports paravirtualized (aka PV console) and an emulated serial >> consoles. This feature will expose an emulated SBSA pl011 UART console to the >> guest, which a user can access using xenconsole. >> >> The device tree passed to the guest VM will contain the pl011 MMIO address >> range and an irq for receiving rx/tx pl011 interrupts. The device tree format >> is specified in Documentation/devicetree/bindings/serial/arm_sbsa_uart.txt. >> >> The Xen hypervisor will expose two types of interfaces to the backend and domU. >> >> The interface exposed to domU will be an emulated pl011 UART by emulating the >> access to the following pl011 registers by the guest. >> >> - Data register (DR) - RW >> - Raw interrupt status register (RIS) - RO >> - Masked interrupt status register (MIS)- RO >> - Interrupt Mask (IMSC) - RW >> - Interrupt Clear (ICR) - WO >> >> It will also inject the pl011 interrupts to the guest in the following >> conditions: >> >> - incoming data in the rx buffer for the guest >> - there is space in the tx buffer for the guest to write more data >> >> The interface exposed to the backend will be the same PV console interface, >> which minimizes the changes required in xenconsole to support a new pl011 console. >> >> This interface has rx and tx ring buffers and an event channel for >> sending/receiving events from the backend. >> >> So essentially Xen handles the data on behalf of domU and the backend. Any data >> written by domU is captured by Xen and written to the TX (OUT) ring buffer >> and a pl011 event is raised to the backend to read the TX ring buffer. >> >> Similarly on reciving a pl011 event, Xen injects an interrupt to guest to >> indicate there is data available in the RX (IN) ring buffer. >> >> The pl011 UART state is completely captured in the set of registers >> mentioned above and this state is updated everytime there is an event from >> the backend or there is register read/write access from domU. >> >> For example, if domU has masked the rx interrupt in the IMSC register, then Xen >> will not inject an interrupt to guest and will just update the RIS register. >> Once the interrupt is unmasked by guest, the interrupt will be delivered to the >> guest. >> >> Changes summary: >> >> Xen Hypervisor >> =============== >> >> 1. Add emulation code to emulate read/write access to pl011 registers and pl011 >> interrupts: >> - It emulates DR read/write by reading and writing from/to the IN and >> OUT ring buffers and raising an event to dom0 when there is data in >> the OUT ring buffer and injecting an interrupt to the guest when there >> is data in the IN ring buffer. >> - Other registers are related to interrupt management and essentially >> control when interrupts are delivered to the guest. >> >> 2. Add a new domctl API to initialize vpl011 emulation in Xen. >> >> 3. Enable vpl011 emulation for a domain based on a libxl option passed during >> domain creation. >> >> Toolstack >> ========== >> >> 1. Add a new option "vuart" in the domU configuration file to enable/disable vuart. >> >> 2. Create a SBSA UART DT node in the guest device tree. It uses a fixed >> vpl011 SPI IRQ number and MMIO address. >> >> 3. Call vpl011 init DOMCTL API to enable vpl011 emulation. >> >> 5. Add a new vuart xenstore node, which contains: >> - ring-ref >> - event channel >> - buffer limit >> - type >> >> Xenconsoled >> ============ >> >> 1. Split the domain structure to support multiple consoles. >> >> 2. Modify different APIs such as buffer_append() etc. to operate on the >> console structure. >> >> 3. Add support for handling multiple consoles. >> >> 4. Add support for vuart console: >> >> The vpl011 changes available at the following repo: >> >> url: https://git@git.linaro.org:/people/bhupinder.thakur/xen.git >> branch: vpl011_v6 >> >> Kindly wait for one day to checkout the code from the above URL. >> >> There are some TBD items which need to be looked at in the future: >> >> 1. Currently UEFI firmware logs the output to hvc console only. How can >> UEFI firmware be made aware of pl011 console and how it can use it >> as a console instead of hvc. >> >> There was a discussion on this and it was decided that SBSA UART should >> be used as a debug port by the UEFI firmware so that all debug output >> is redirected to this port. >> >> 2. Linux seems to have hvc console as the default console i.e. if no >> console is specified then it uses hvc as the console. How can an >> option be provided in Linux to select either hvc or pl011 as the >> default console. >> >> It was suggeted to use the SPCR in ACPI and the stdout-path option in the >> device tree to specify the default console. However, currently hvc console >> is not describable in the ACPI/device tree. This support will have to be >> added to allow the user to specify the default console. >> >> 3. ACPI support for pl011 device. >> >> CC: Ian Jackson <ian.jackson@eu.citrix.com> >> CC: Wei Liu <wei.liu2@citrix.com> >> CC: Stefano Stabellini <sstabellini@kernel.org> >> CC: Julien Grall <julien.grall@arm.com> >> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> >> >> >> Bhupinder Thakur (25): >> xen/arm: vpl011: Define common ring buffer helper functions in >> console.h >> xen/arm: vpl011: Add SBSA UART emulation in Xen >> xen/arm: vpl011: Allocate a new GFN in the toolstack for vuart >> xen/arm: vpl011: Add support for vuart in libxl >> xen/arm: vpl011: Rearrange xen header includes in alphabetical order >> in domctl.c >> xen/arm: vpl011: Add a new domctl API to initialize vpl011 >> xen/arm: vpl011: Add a new vuart node in the xenstore >> xen/arm: vpl011: Modify xenconsole to define and use a new console >> structure >> xen/arm: vpl011: Rename the console structure field conspath to xspath >> xen/arm: vpl011: Modify xenconsole functions to take console structure >> as input >> xen/arm: vpl011: Add a new console_init function in xenconsole >> xen/arm: vpl011: Add a new buffer_available function in xenconsole >> xen/arm: vpl011: Add a new add_console_evtchn_fd function in >> xenconsole >> xen/arm: vpl011: Add a new add_console_tty_fd function in xenconsole >> xen/arm: vpl011: Add a new console_evtchn_unmask function in >> xenconsole >> xen/arm: vpl011: Add a new handle_console_ring function in xenconsole >> xen/arm: vpl011: Add a new handle_console_tty function in xenconsole >> xen/arm: vpl011: Add a new console_cleanup function in xenconsole >> xen/arm: vpl011: Add a new console_open_log function in xenconsole >> xen/arm: vpl011: Add a new console_close_evtchn function in xenconsole >> xen/arm: vpl011: Add support for multiple consoles in xenconsole >> xen/arm: vpl011: Add support for vuart console in xenconsole >> xen/arm: vpl011: Add a new vuart console type to xenconsole client >> xen/arm: vpl011: Add a pl011 uart DT node in the guest device tree >> xen/arm: vpl011: Update documentation for vuart console support >> >> config/arm32.mk | 1 + >> config/arm64.mk | 1 + >> docs/man/xl.cfg.pod.5.in | 9 + >> docs/misc/console.txt | 44 ++- >> tools/console/Makefile | 3 +- >> tools/console/client/main.c | 13 +- >> tools/console/daemon/io.c | 662 ++++++++++++++++++++++++----------- >> tools/libxc/include/xc_dom.h | 2 + >> tools/libxc/include/xenctrl.h | 18 + >> tools/libxc/xc_dom_arm.c | 5 +- >> tools/libxc/xc_dom_boot.c | 2 + >> tools/libxc/xc_domain.c | 24 ++ >> tools/libxl/libxl.h | 6 + >> tools/libxl/libxl_arch.h | 6 + >> tools/libxl/libxl_arm.c | 72 +++- >> tools/libxl/libxl_console.c | 47 +++ >> tools/libxl/libxl_create.c | 9 +- >> tools/libxl/libxl_device.c | 9 +- >> tools/libxl/libxl_dom.c | 5 + >> tools/libxl/libxl_internal.h | 6 + >> tools/libxl/libxl_types.idl | 7 + >> tools/libxl/libxl_types_internal.idl | 1 + >> tools/libxl/libxl_x86.c | 8 + >> tools/xl/xl_cmdtable.c | 2 +- >> tools/xl/xl_console.c | 5 +- >> tools/xl/xl_parse.c | 8 + >> xen/arch/arm/Kconfig | 7 + >> xen/arch/arm/Makefile | 1 + >> xen/arch/arm/domain.c | 5 + >> xen/arch/arm/domctl.c | 43 ++- >> xen/arch/arm/vpl011.c | 455 ++++++++++++++++++++++++ >> xen/include/asm-arm/domain.h | 6 + >> xen/include/asm-arm/pl011-uart.h | 2 + >> xen/include/asm-arm/vpl011.h | 72 ++++ >> xen/include/public/arch-arm.h | 6 + >> xen/include/public/domctl.h | 21 ++ >> xen/include/public/io/console.h | 4 + >> 37 files changed, 1360 insertions(+), 237 deletions(-) >> create mode 100644 xen/arch/arm/vpl011.c >> create mode 100644 xen/include/asm-arm/vpl011.h >> > > -- > Julien Grall
On 20/07/17 10:40, Bhupinder Thakur wrote: > Hi Julien, Hi Bhupinder, > The issue seems to be that domain_vpl011_deinit() is not getting > called when we destroy the domain. On further debugging, i found that > domain_relinquish_resources was failing in domain_kill() function. > > I believe that issue could be that arch_domain_destroy() (from where > domain_vpl011_deinit() is called) is called later and because > domain_kill() fails, it is not getting called. Once I moved the > domain_vpl011_deinit() inside domain_kill(), the domain is getting > cleaned up properly. What do you mean by domain_kill fails? The function may take time to execute so it may return -ERESTART for preemption and be restarted later on. The problem here is until domain_vpl011_deinit is called, there is still a reference taken on the guest memory and therefore on the domain. So the domain will not get fully destroyed (domain_destroy is called when the last reference on the domain is dropped). Because domain_vpl011_deinit is called from domain_destroy, there will always be a reference on the guest memory and domain. So no destruction. > > Is it ok to call domain_vpl011_deinit() inside domain_kill? domain_kill is a arch agnostic function. You want to call domain_vpl011_init from domain_relinquish_resources. I would do in the case RELMEM_not_started after iommu_release_dt_devices is called. Cheers,
SBSA UART emulation for guests in Xen ====================================== Linaro has published VM System specification for ARM Processors, which provides a set of guidelines for both guest OS and hypervisor implementations, such that building OS images according to these guidelines guarantees that those images can also run on hypervisors compliant with this specification. One of the spec requirements is that the hypervisor must provide an emulated SBSA UART as a serial console which meets the minimum requirements in SBSA UART as defined in appendix B of the following ARM Server Base Architecture Document: https://static.docs.arm.com/den0029/a/Server_Base_System_Architecture_v3_1_ARM_DEN_0029A.pdf. This feature allows the Xen guests to use SBSA compliant pl011 UART as as a console. Note that SBSA pl011 UART is a subset of full featured ARM pl011 UART and supports only a subset of registers as mentioned below. It does not support rx/tx DMA. Currently, Xen supports paravirtualized (aka PV console) and an emulated serial consoles. This feature will expose an emulated SBSA pl011 UART console to the guest, which a user can access using xenconsole. The device tree passed to the guest VM will contain the pl011 MMIO address range and an irq for receiving rx/tx pl011 interrupts. The device tree format is specified in Documentation/devicetree/bindings/serial/arm_sbsa_uart.txt. The Xen hypervisor will expose two types of interfaces to the backend and domU. The interface exposed to domU will be an emulated pl011 UART by emulating the access to the following pl011 registers by the guest. - Data register (DR) - RW - Raw interrupt status register (RIS) - RO - Masked interrupt status register (MIS)- RO - Interrupt Mask (IMSC) - RW - Interrupt Clear (ICR) - WO It will also inject the pl011 interrupts to the guest in the following conditions: - incoming data in the rx buffer for the guest - there is space in the tx buffer for the guest to write more data The interface exposed to the backend will be the same PV console interface, which minimizes the changes required in xenconsole to support a new pl011 console. This interface has rx and tx ring buffers and an event channel for sending/receiving events from the backend. So essentially Xen handles the data on behalf of domU and the backend. Any data written by domU is captured by Xen and written to the TX (OUT) ring buffer and a pl011 event is raised to the backend to read the TX ring buffer. Similarly on reciving a pl011 event, Xen injects an interrupt to guest to indicate there is data available in the RX (IN) ring buffer. The pl011 UART state is completely captured in the set of registers mentioned above and this state is updated everytime there is an event from the backend or there is register read/write access from domU. For example, if domU has masked the rx interrupt in the IMSC register, then Xen will not inject an interrupt to guest and will just update the RIS register. Once the interrupt is unmasked by guest, the interrupt will be delivered to the guest. Changes summary: Xen Hypervisor =============== 1. Add emulation code to emulate read/write access to pl011 registers and pl011 interrupts: - It emulates DR read/write by reading and writing from/to the IN and OUT ring buffers and raising an event to dom0 when there is data in the OUT ring buffer and injecting an interrupt to the guest when there is data in the IN ring buffer. - Other registers are related to interrupt management and essentially control when interrupts are delivered to the guest. 2. Add a new domctl API to initialize vpl011 emulation in Xen. 3. Enable vpl011 emulation for a domain based on a libxl option passed during domain creation. Toolstack ========== 1. Add a new option "vuart" in the domU configuration file to enable/disable vuart. 2. Create a SBSA UART DT node in the guest device tree. It uses a fixed vpl011 SPI IRQ number and MMIO address. 3. Call vpl011 init DOMCTL API to enable vpl011 emulation. 5. Add a new vuart xenstore node, which contains: - ring-ref - event channel - buffer limit - type Xenconsoled ============ 1. Split the domain structure to support multiple consoles. 2. Modify different APIs such as buffer_append() etc. to operate on the console structure. 3. Add support for handling multiple consoles. 4. Add support for vuart console: The vpl011 changes available at the following repo: url: https://git@git.linaro.org:/people/bhupinder.thakur/xen.git branch: vpl011_v6 Kindly wait for one day to checkout the code from the above URL. There are some TBD items which need to be looked at in the future: 1. Currently UEFI firmware logs the output to hvc console only. How can UEFI firmware be made aware of pl011 console and how it can use it as a console instead of hvc. There was a discussion on this and it was decided that SBSA UART should be used as a debug port by the UEFI firmware so that all debug output is redirected to this port. 2. Linux seems to have hvc console as the default console i.e. if no console is specified then it uses hvc as the console. How can an option be provided in Linux to select either hvc or pl011 as the default console. It was suggeted to use the SPCR in ACPI and the stdout-path option in the device tree to specify the default console. However, currently hvc console is not describable in the ACPI/device tree. This support will have to be added to allow the user to specify the default console. 3. ACPI support for pl011 device. CC: Ian Jackson <ian.jackson@eu.citrix.com> CC: Wei Liu <wei.liu2@citrix.com> CC: Stefano Stabellini <sstabellini@kernel.org> CC: Julien Grall <julien.grall@arm.com> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Bhupinder Thakur (25): xen/arm: vpl011: Define common ring buffer helper functions in console.h xen/arm: vpl011: Add SBSA UART emulation in Xen xen/arm: vpl011: Allocate a new GFN in the toolstack for vuart xen/arm: vpl011: Add support for vuart in libxl xen/arm: vpl011: Rearrange xen header includes in alphabetical order in domctl.c xen/arm: vpl011: Add a new domctl API to initialize vpl011 xen/arm: vpl011: Add a new vuart node in the xenstore xen/arm: vpl011: Modify xenconsole to define and use a new console structure xen/arm: vpl011: Rename the console structure field conspath to xspath xen/arm: vpl011: Modify xenconsole functions to take console structure as input xen/arm: vpl011: Add a new console_init function in xenconsole xen/arm: vpl011: Add a new buffer_available function in xenconsole xen/arm: vpl011: Add a new add_console_evtchn_fd function in xenconsole xen/arm: vpl011: Add a new add_console_tty_fd function in xenconsole xen/arm: vpl011: Add a new console_evtchn_unmask function in xenconsole xen/arm: vpl011: Add a new handle_console_ring function in xenconsole xen/arm: vpl011: Add a new handle_console_tty function in xenconsole xen/arm: vpl011: Add a new console_cleanup function in xenconsole xen/arm: vpl011: Add a new console_open_log function in xenconsole xen/arm: vpl011: Add a new console_close_evtchn function in xenconsole xen/arm: vpl011: Add support for multiple consoles in xenconsole xen/arm: vpl011: Add support for vuart console in xenconsole xen/arm: vpl011: Add a new vuart console type to xenconsole client xen/arm: vpl011: Add a pl011 uart DT node in the guest device tree xen/arm: vpl011: Update documentation for vuart console support config/arm32.mk | 1 + config/arm64.mk | 1 + docs/man/xl.cfg.pod.5.in | 9 + docs/misc/console.txt | 44 ++- tools/console/Makefile | 3 +- tools/console/client/main.c | 13 +- tools/console/daemon/io.c | 662 ++++++++++++++++++++++++----------- tools/libxc/include/xc_dom.h | 2 + tools/libxc/include/xenctrl.h | 18 + tools/libxc/xc_dom_arm.c | 5 +- tools/libxc/xc_dom_boot.c | 2 + tools/libxc/xc_domain.c | 24 ++ tools/libxl/libxl.h | 6 + tools/libxl/libxl_arch.h | 6 + tools/libxl/libxl_arm.c | 72 +++- tools/libxl/libxl_console.c | 47 +++ tools/libxl/libxl_create.c | 9 +- tools/libxl/libxl_device.c | 9 +- tools/libxl/libxl_dom.c | 5 + tools/libxl/libxl_internal.h | 6 + tools/libxl/libxl_types.idl | 7 + tools/libxl/libxl_types_internal.idl | 1 + tools/libxl/libxl_x86.c | 8 + tools/xl/xl_cmdtable.c | 2 +- tools/xl/xl_console.c | 5 +- tools/xl/xl_parse.c | 8 + xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/domain.c | 5 + xen/arch/arm/domctl.c | 43 ++- xen/arch/arm/vpl011.c | 455 ++++++++++++++++++++++++ xen/include/asm-arm/domain.h | 6 + xen/include/asm-arm/pl011-uart.h | 2 + xen/include/asm-arm/vpl011.h | 72 ++++ xen/include/public/arch-arm.h | 6 + xen/include/public/domctl.h | 21 ++ xen/include/public/io/console.h | 4 + 37 files changed, 1360 insertions(+), 237 deletions(-) create mode 100644 xen/arch/arm/vpl011.c create mode 100644 xen/include/asm-arm/vpl011.h