[Xen-devel,RFC] tools/libxl: Switch Arm guest type to PVH

Message ID 20180625130814.29771-1-julien.grall@arm.com
State New
Headers show
Series
  • [Xen-devel,RFC] tools/libxl: Switch Arm guest type to PVH
Related show

Commit Message

Julien Grall June 25, 2018, 1:08 p.m.
Currently, the toolstack is considering Arm guest always PV. However,
they are very similar to PVH because HW virtualization extension are used
and QEMU is not started. So switch Arm guest type to PVH.

Furthermore, the default type for Arm in xl will now be PVH to allow
smooth transition for user.

Signed-off-by: Julien Grall <julien.grall@arm.com>

---

This was discussed at Xen Summit and also in various thread on
xen-devel. The latest one was when Andrew sent a patch to deny guest creation
on Arm with XEN_DOMCTL_CDF_hap unset.

I suspect we first implemented Arm guest as PV in libxl because PVH was
non-existent and the type was easier to avoid spawning QEMU. Note that
Linux and Xen are already considering Arm guest as PVH.

This patch is denying the creation of PV guest on Arm. This will have an
impact on any toolstack selecting PV for Arm (e.g libvirt) or anyone
using 'type="pv"' in libxl configuration file.

I am not entirely sure whether this is the best solution, any opinions?
---
 docs/man/xl.cfg.pod.5.in |  3 ++-
 tools/libxl/libxl_arm.c  | 10 +++++++++-
 tools/xl/xl_parse.c      |  4 ++++
 3 files changed, 15 insertions(+), 2 deletions(-)

Comments

Wei Liu June 25, 2018, 1:19 p.m. | #1
On Mon, Jun 25, 2018 at 02:08:14PM +0100, Julien Grall wrote:
> Currently, the toolstack is considering Arm guest always PV. However,
> they are very similar to PVH because HW virtualization extension are used
> and QEMU is not started. So switch Arm guest type to PVH.
> 
> Furthermore, the default type for Arm in xl will now be PVH to allow
> smooth transition for user.
> 
> Signed-off-by: Julien Grall <julien.grall@arm.com>
> 
> ---
> 
> This was discussed at Xen Summit and also in various thread on
> xen-devel. The latest one was when Andrew sent a patch to deny guest creation
> on Arm with XEN_DOMCTL_CDF_hap unset.
> 
> I suspect we first implemented Arm guest as PV in libxl because PVH was
> non-existent and the type was easier to avoid spawning QEMU. Note that
> Linux and Xen are already considering Arm guest as PVH.
> 
> This patch is denying the creation of PV guest on Arm. This will have an
> impact on any toolstack selecting PV for Arm (e.g libvirt) or anyone
> using 'type="pv"' in libxl configuration file.
> 
> I am not entirely sure whether this is the best solution, any opinions?
> ---
>  docs/man/xl.cfg.pod.5.in |  3 ++-
>  tools/libxl/libxl_arm.c  | 10 +++++++++-
>  tools/xl/xl_parse.c      |  4 ++++
>  3 files changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/docs/man/xl.cfg.pod.5.in b/docs/man/xl.cfg.pod.5.in
> index 47d88243b1..2c2af01515 100644
> --- a/docs/man/xl.cfg.pod.5.in
> +++ b/docs/man/xl.cfg.pod.5.in
> @@ -86,7 +86,8 @@ guest operating systems. This is the default.
>  
>  Specifies that this is to be an PVH domain. That is a lightweight HVM-like
>  guest without a device model and without many of the emulated devices
> -available to HVM guests. Note that this mode requires a PVH aware kernel.
> +available to HVM guests. Note that this mode requires a PVH aware kernel on
> +x86.

Have we ever advertised on ARM which guest type users should set? Are
they aware of the fact that it is PV in toolstack and HVM in Xen?

If users are oblivious to the underlying implementation, I think we can
skip telling them PVH in ARM world.

Wei.
Julien Grall June 25, 2018, 4:18 p.m. | #2
Hi Wei,

On 25/06/18 14:19, Wei Liu wrote:
> On Mon, Jun 25, 2018 at 02:08:14PM +0100, Julien Grall wrote:
>> Currently, the toolstack is considering Arm guest always PV. However,
>> they are very similar to PVH because HW virtualization extension are used
>> and QEMU is not started. So switch Arm guest type to PVH.
>>
>> Furthermore, the default type for Arm in xl will now be PVH to allow
>> smooth transition for user.
>>
>> Signed-off-by: Julien Grall <julien.grall@arm.com>
>>
>> ---
>>
>> This was discussed at Xen Summit and also in various thread on
>> xen-devel. The latest one was when Andrew sent a patch to deny guest creation
>> on Arm with XEN_DOMCTL_CDF_hap unset.
>>
>> I suspect we first implemented Arm guest as PV in libxl because PVH was
>> non-existent and the type was easier to avoid spawning QEMU. Note that
>> Linux and Xen are already considering Arm guest as PVH.
>>
>> This patch is denying the creation of PV guest on Arm. This will have an
>> impact on any toolstack selecting PV for Arm (e.g libvirt) or anyone
>> using 'type="pv"' in libxl configuration file.
>>
>> I am not entirely sure whether this is the best solution, any opinions?
>> ---
>>   docs/man/xl.cfg.pod.5.in |  3 ++-
>>   tools/libxl/libxl_arm.c  | 10 +++++++++-
>>   tools/xl/xl_parse.c      |  4 ++++
>>   3 files changed, 15 insertions(+), 2 deletions(-)
>>
>> diff --git a/docs/man/xl.cfg.pod.5.in b/docs/man/xl.cfg.pod.5.in
>> index 47d88243b1..2c2af01515 100644
>> --- a/docs/man/xl.cfg.pod.5.in
>> +++ b/docs/man/xl.cfg.pod.5.in
>> @@ -86,7 +86,8 @@ guest operating systems. This is the default.
>>   
>>   Specifies that this is to be an PVH domain. That is a lightweight HVM-like
>>   guest without a device model and without many of the emulated devices
>> -available to HVM guests. Note that this mode requires a PVH aware kernel.
>> +available to HVM guests. Note that this mode requires a PVH aware kernel on
>> +x86.
> 
> Have we ever advertised on ARM which guest type users should set? Are
> they aware of the fact that it is PV in toolstack and HVM in Xen?

We always advertised "ARM guest" and not a specific type of guest. So 
most of the users of xl will likely not specified the type of guest.

> 
> If users are oblivious to the underlying implementation, I think we can
> skip telling them PVH in ARM world.
PVH is quite confusing for Arm. We only used the term to make a 
comparison with x86. This is the closest to Arm guest but I am not too 
sure whether they would be the exactly the same.

At the moment, I would rather avoid to say 'PVH' for the end user and 
just let the different toolstack to set the type to PVH.

Another solution is for libxl to force the guest type to PVH on Arm. I 
am not sure what would be the impact here.

Cheers,
Roger Pau Monné June 25, 2018, 4:28 p.m. | #3
On Mon, Jun 25, 2018 at 05:18:31PM +0100, Julien Grall wrote:
> Hi Wei,
> 
> On 25/06/18 14:19, Wei Liu wrote:
> > On Mon, Jun 25, 2018 at 02:08:14PM +0100, Julien Grall wrote:
> > > Currently, the toolstack is considering Arm guest always PV. However,
> > > they are very similar to PVH because HW virtualization extension are used
> > > and QEMU is not started. So switch Arm guest type to PVH.
> > > 
> > > Furthermore, the default type for Arm in xl will now be PVH to allow
> > > smooth transition for user.
> > > 
> > > Signed-off-by: Julien Grall <julien.grall@arm.com>
> > > 
> > > ---
> > > 
> > > This was discussed at Xen Summit and also in various thread on
> > > xen-devel. The latest one was when Andrew sent a patch to deny guest creation
> > > on Arm with XEN_DOMCTL_CDF_hap unset.
> > > 
> > > I suspect we first implemented Arm guest as PV in libxl because PVH was
> > > non-existent and the type was easier to avoid spawning QEMU. Note that
> > > Linux and Xen are already considering Arm guest as PVH.
> > > 
> > > This patch is denying the creation of PV guest on Arm. This will have an
> > > impact on any toolstack selecting PV for Arm (e.g libvirt) or anyone
> > > using 'type="pv"' in libxl configuration file.
> > > 
> > > I am not entirely sure whether this is the best solution, any opinions?
> > > ---
> > >   docs/man/xl.cfg.pod.5.in |  3 ++-
> > >   tools/libxl/libxl_arm.c  | 10 +++++++++-
> > >   tools/xl/xl_parse.c      |  4 ++++
> > >   3 files changed, 15 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/docs/man/xl.cfg.pod.5.in b/docs/man/xl.cfg.pod.5.in
> > > index 47d88243b1..2c2af01515 100644
> > > --- a/docs/man/xl.cfg.pod.5.in
> > > +++ b/docs/man/xl.cfg.pod.5.in
> > > @@ -86,7 +86,8 @@ guest operating systems. This is the default.
> > >   Specifies that this is to be an PVH domain. That is a lightweight HVM-like
> > >   guest without a device model and without many of the emulated devices
> > > -available to HVM guests. Note that this mode requires a PVH aware kernel.
> > > +available to HVM guests. Note that this mode requires a PVH aware kernel on
> > > +x86.
> > 
> > Have we ever advertised on ARM which guest type users should set? Are
> > they aware of the fact that it is PV in toolstack and HVM in Xen?
> 
> We always advertised "ARM guest" and not a specific type of guest. So most
> of the users of xl will likely not specified the type of guest.
> 
> > 
> > If users are oblivious to the underlying implementation, I think we can
> > skip telling them PVH in ARM world.
> PVH is quite confusing for Arm. We only used the term to make a comparison
> with x86. This is the closest to Arm guest but I am not too sure whether
> they would be the exactly the same.
> 
> At the moment, I would rather avoid to say 'PVH' for the end user and just
> let the different toolstack to set the type to PVH.
> 
> Another solution is for libxl to force the guest type to PVH on Arm. I am
> not sure what would be the impact here.

IMO I would remove the 'type' option from xl.cfg (so that it's
basically ignored) in the ARM case and force it internally to PVH (if
that's the best route for current ARM guests).

Iff ARM even gains more guest types the option can be added and the
guest types can be set to meaningful ARM guest types.

In any case, this requires changes to xl.cfg man page.

Roger.
Ian Jackson June 25, 2018, 4:39 p.m. | #4
Roger Pau Monné writes ("Re: [PATCH RFC] tools/libxl: Switch Arm guest type to PVH"):
> IMO I would remove the 'type' option from xl.cfg (so that it's
> basically ignored) in the ARM case and force it internally to PVH (if
> that's the best route for current ARM guests).

What about libvirt users ?  I haven't seen what a libvirt Xen ARM
guest config looks like but we need to meak sure that existing guests
don't break.

Ian.
Roger Pau Monné June 26, 2018, 6:49 a.m. | #5
On Mon, Jun 25, 2018 at 05:39:12PM +0100, Ian Jackson wrote:
> Roger Pau Monné writes ("Re: [PATCH RFC] tools/libxl: Switch Arm guest type to PVH"):
> > IMO I would remove the 'type' option from xl.cfg (so that it's
> > basically ignored) in the ARM case and force it internally to PVH (if
> > that's the best route for current ARM guests).
> 
> What about libvirt users ?  I haven't seen what a libvirt Xen ARM
> guest config looks like but we need to meak sure that existing guests
> don't break.

For livbirt (or users of libxl library) we could force the type to
pvh, regardless of the value set by the client, but I guess that would
make adding types later on quite complicated.

Roger.
Wei Liu June 27, 2018, 12:50 p.m. | #6
On Mon, Jun 25, 2018 at 05:39:12PM +0100, Ian Jackson wrote:
> Roger Pau Monné writes ("Re: [PATCH RFC] tools/libxl: Switch Arm guest type to PVH"):
> > IMO I would remove the 'type' option from xl.cfg (so that it's
> > basically ignored) in the ARM case and force it internally to PVH (if
> > that's the best route for current ARM guests).
> 
> What about libvirt users ?  I haven't seen what a libvirt Xen ARM
> guest config looks like but we need to meak sure that existing guests
> don't break.

It may set type to pv. The following file is generated by the config
converter.

logs.test-lab.xenproject.org/osstest/logs/124566/test-armhf-armhf-libvirt-xsm/arndale-metrocentre---var-log-libvirt-libxl-debian.guest.osstest.log

Wei.
Julien Grall July 2, 2018, 2:13 p.m. | #7
On 06/26/2018 07:49 AM, Roger Pau Monné wrote:
> On Mon, Jun 25, 2018 at 05:39:12PM +0100, Ian Jackson wrote:
>> Roger Pau Monné writes ("Re: [PATCH RFC] tools/libxl: Switch Arm guest type to PVH"):
>>> IMO I would remove the 'type' option from xl.cfg (so that it's
>>> basically ignored) in the ARM case and force it internally to PVH (if
>>> that's the best route for current ARM guests).
>>
>> What about libvirt users ?  I haven't seen what a libvirt Xen ARM
>> guest config looks like but we need to meak sure that existing guests
>> don't break.
> 
> For livbirt (or users of libxl library) we could force the type to
> pvh, regardless of the value set by the client, but I guess that would
> make adding types later on quite complicated.

I am fairly confident we will never have PV guest on Arm. So one 
solution would be to alias PV to PVH for Arm. This still give us the 
liberty to add more guest type in the future.

Any opinions?

Cheers,
Wei Liu July 2, 2018, 3:55 p.m. | #8
On Mon, Jul 02, 2018 at 03:13:38PM +0100, Julien Grall wrote:
> 
> 
> On 06/26/2018 07:49 AM, Roger Pau Monné wrote:
> > On Mon, Jun 25, 2018 at 05:39:12PM +0100, Ian Jackson wrote:
> > > Roger Pau Monné writes ("Re: [PATCH RFC] tools/libxl: Switch Arm guest type to PVH"):
> > > > IMO I would remove the 'type' option from xl.cfg (so that it's
> > > > basically ignored) in the ARM case and force it internally to PVH (if
> > > > that's the best route for current ARM guests).
> > > 
> > > What about libvirt users ?  I haven't seen what a libvirt Xen ARM
> > > guest config looks like but we need to meak sure that existing guests
> > > don't break.
> > 
> > For livbirt (or users of libxl library) we could force the type to
> > pvh, regardless of the value set by the client, but I guess that would
> > make adding types later on quite complicated.
> 
> I am fairly confident we will never have PV guest on Arm. So one solution
> would be to alias PV to PVH for Arm. This still give us the liberty to add
> more guest type in the future.
> 
> Any opinions?

Yes, we could make PV synonym to PVH (but don't talk about it, since Xen
Arm never emphasises it guest type).  Whatever new guest type we're
going to add can't be called PV or PVH. I don't think that would be a
big problem -- if you want a different guest type, you definitely want a
different name.

Wei.
Stefano Stabellini July 2, 2018, 5:18 p.m. | #9
On Mon, 2 Jul 2018, Julien Grall wrote:
> On 06/26/2018 07:49 AM, Roger Pau Monné wrote:

> > On Mon, Jun 25, 2018 at 05:39:12PM +0100, Ian Jackson wrote:

> > > Roger Pau Monné writes ("Re: [PATCH RFC] tools/libxl: Switch Arm guest

> > > type to PVH"):

> > > > IMO I would remove the 'type' option from xl.cfg (so that it's

> > > > basically ignored) in the ARM case and force it internally to PVH (if

> > > > that's the best route for current ARM guests).

> > > 

> > > What about libvirt users ?  I haven't seen what a libvirt Xen ARM

> > > guest config looks like but we need to meak sure that existing guests

> > > don't break.

> > 

> > For livbirt (or users of libxl library) we could force the type to

> > pvh, regardless of the value set by the client, but I guess that would

> > make adding types later on quite complicated.

> 

> I am fairly confident we will never have PV guest on Arm. So one solution

> would be to alias PV to PVH for Arm. This still give us the liberty to add

> more guest type in the future.

> 

> Any opinions?


Roger, what is the plan for x86? Wasn't there an idea to silently and
transparently "upgrade" PV guests to PVH when possible (when hardware
support is available)?

If that is the case, basically we could do the same for ARM. We could
have an hardware features check, that would always return true on ARM
because without virtualization extensions Xen cannot even boot, then
upgrade PV to PVH. On x86 the upgrade would only happen when the
required features are present.
Roger Pau Monné July 3, 2018, 7:28 a.m. | #10
On Mon, Jul 02, 2018 at 10:18:17AM -0700, Stefano Stabellini wrote:
> On Mon, 2 Jul 2018, Julien Grall wrote:
> > On 06/26/2018 07:49 AM, Roger Pau Monné wrote:
> > > On Mon, Jun 25, 2018 at 05:39:12PM +0100, Ian Jackson wrote:
> > > > Roger Pau Monné writes ("Re: [PATCH RFC] tools/libxl: Switch Arm guest
> > > > type to PVH"):
> > > > > IMO I would remove the 'type' option from xl.cfg (so that it's
> > > > > basically ignored) in the ARM case and force it internally to PVH (if
> > > > > that's the best route for current ARM guests).
> > > > 
> > > > What about libvirt users ?  I haven't seen what a libvirt Xen ARM
> > > > guest config looks like but we need to meak sure that existing guests
> > > > don't break.
> > > 
> > > For livbirt (or users of libxl library) we could force the type to
> > > pvh, regardless of the value set by the client, but I guess that would
> > > make adding types later on quite complicated.
> > 
> > I am fairly confident we will never have PV guest on Arm. So one solution
> > would be to alias PV to PVH for Arm. This still give us the liberty to add
> > more guest type in the future.
> > 
> > Any opinions?
> 
> Roger, what is the plan for x86? Wasn't there an idea to silently and
> transparently "upgrade" PV guests to PVH when possible (when hardware
> support is available)?

This could only be done for xl config files that don't specify a
'type' option IMO, for libvirt and other users of libxl I don't think
this is feasible.

> If that is the case, basically we could do the same for ARM. We could
> have an hardware features check, that would always return true on ARM
> because without virtualization extensions Xen cannot even boot, then
> upgrade PV to PVH. On x86 the upgrade would only happen when the
> required features are present.

It's slightly more convoluted on x86 because apart from the hardware
features you also need to parse the kernel in order to figure out if
it has the PVH entry point before setting the type to PVH.

Roger.
Stefano Stabellini July 3, 2018, 8:27 p.m. | #11
On Tue, 3 Jul 2018, Roger Pau Monné wrote:
> On Mon, Jul 02, 2018 at 10:18:17AM -0700, Stefano Stabellini wrote:

> > On Mon, 2 Jul 2018, Julien Grall wrote:

> > > On 06/26/2018 07:49 AM, Roger Pau Monné wrote:

> > > > On Mon, Jun 25, 2018 at 05:39:12PM +0100, Ian Jackson wrote:

> > > > > Roger Pau Monné writes ("Re: [PATCH RFC] tools/libxl: Switch Arm guest

> > > > > type to PVH"):

> > > > > > IMO I would remove the 'type' option from xl.cfg (so that it's

> > > > > > basically ignored) in the ARM case and force it internally to PVH (if

> > > > > > that's the best route for current ARM guests).

> > > > > 

> > > > > What about libvirt users ?  I haven't seen what a libvirt Xen ARM

> > > > > guest config looks like but we need to meak sure that existing guests

> > > > > don't break.

> > > > 

> > > > For livbirt (or users of libxl library) we could force the type to

> > > > pvh, regardless of the value set by the client, but I guess that would

> > > > make adding types later on quite complicated.

> > > 

> > > I am fairly confident we will never have PV guest on Arm. So one solution

> > > would be to alias PV to PVH for Arm. This still give us the liberty to add

> > > more guest type in the future.

> > > 

> > > Any opinions?

> > 

> > Roger, what is the plan for x86? Wasn't there an idea to silently and

> > transparently "upgrade" PV guests to PVH when possible (when hardware

> > support is available)?

> 

> This could only be done for xl config files that don't specify a

> 'type' option IMO, for libvirt and other users of libxl I don't think

> this is feasible.

> 

> > If that is the case, basically we could do the same for ARM. We could

> > have an hardware features check, that would always return true on ARM

> > because without virtualization extensions Xen cannot even boot, then

> > upgrade PV to PVH. On x86 the upgrade would only happen when the

> > required features are present.

> 

> It's slightly more convoluted on x86 because apart from the hardware

> features you also need to parse the kernel in order to figure out if

> it has the PVH entry point before setting the type to PVH.


I see. Too bad. In that case, if we have to go with an ARM specific
solution and nobody has a better suggestion, then I think it is OK to go
ahead with aliasing PV to PVH on ARM as Julien mentioned.

Patch

diff --git a/docs/man/xl.cfg.pod.5.in b/docs/man/xl.cfg.pod.5.in
index 47d88243b1..2c2af01515 100644
--- a/docs/man/xl.cfg.pod.5.in
+++ b/docs/man/xl.cfg.pod.5.in
@@ -86,7 +86,8 @@  guest operating systems. This is the default.
 
 Specifies that this is to be an PVH domain. That is a lightweight HVM-like
 guest without a device model and without many of the emulated devices
-available to HVM guests. Note that this mode requires a PVH aware kernel.
+available to HVM guests. Note that this mode requires a PVH aware kernel on
+x86.
 
 =item B<type="hvm">
 
diff --git a/tools/libxl/libxl_arm.c b/tools/libxl/libxl_arm.c
index 8af9f6f992..3acac7fa93 100644
--- a/tools/libxl/libxl_arm.c
+++ b/tools/libxl/libxl_arm.c
@@ -978,7 +978,15 @@  int libxl__arch_domain_init_hw_description(libxl__gc *gc,
     int rc;
     uint64_t val;
 
-    assert(info->type == LIBXL_DOMAIN_TYPE_PV);
+    if (info->type == LIBXL_DOMAIN_TYPE_PV) {
+        LOG(ERROR, "PV guest not supported. Please use PVH.\n");
+        return ERROR_FEATURE_REMOVED;
+    }
+    else if (info->type != LIBXL_DOMAIN_TYPE_PVH) {
+        LOG(ERROR, "Unsupported Arm guest type %s\n",
+            libxl_domain_type_to_string(info->type));
+        return ERROR_FAIL;
+    }
 
     /* Set the value of domain param HVM_PARAM_CALLBACK_IRQ. */
     val = MASK_INSR(HVM_PARAM_CALLBACK_TYPE_PPI,
diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
index e6c54483e0..b705f27277 100644
--- a/tools/xl/xl_parse.c
+++ b/tools/xl/xl_parse.c
@@ -937,7 +937,11 @@  void parse_config_data(const char *config_source,
     }
 
     if (c_info->type == LIBXL_DOMAIN_TYPE_INVALID)
+#if defined(__arm__) || defined(__aarch64__)
+        c_info->type = LIBXL_DOMAIN_TYPE_PVH;
+#else
         c_info->type = LIBXL_DOMAIN_TYPE_PV;
+#endif
 
     xlu_cfg_get_defbool(config, "hap", &c_info->hap, 0);