[RFC,v2] docs/interop: define PROBE feature for vhost-user VirtIO devices

Currently QEMU has to know some details about the VirtIO device
supported by a vhost-user daemon to be able to setup the guest. This
makes it hard for QEMU to add support for additional vhost-user
daemons without adding specific stubs for each additional VirtIO
device.

This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
which the back-end can advertise which allows a probe message to be
sent to get all the details QEMU needs to know in one message.

Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
daemons which are capable of handling all aspects of the VirtIO
transactions with only a generic stub on the QEMU side. These daemons
can also be used without QEMU in situations where there isn't a full
VMM managing their setup.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - dropped F_STANDALONE in favour of F_PROBE
  - split probe details across several messages
  - probe messages don't automatically imply a standalone daemon
  - add wording where probe details interact (F_MQ/F_CONFIG)
  - define VMM and make clear QEMU is only one of many potential VMMs
  - reword commit message
---
 docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
 hw/virtio/vhost-user.c      |  8 ++++
 2 files changed, 88 insertions(+), 10 deletions(-)

This looks great! Thanks for this proposal.

On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:

> Currently QEMU has to know some details about the VirtIO device
> supported by a vhost-user daemon to be able to setup the guest. This
> makes it hard for QEMU to add support for additional vhost-user
> daemons without adding specific stubs for each additional VirtIO
> device.
>
> This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> which the back-end can advertise which allows a probe message to be
> sent to get all the details QEMU needs to know in one message.
>
> Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> daemons which are capable of handling all aspects of the VirtIO
> transactions with only a generic stub on the QEMU side. These daemons
> can also be used without QEMU in situations where there isn't a full
> VMM managing their setup.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> ---
> v2
>   - dropped F_STANDALONE in favour of F_PROBE
>   - split probe details across several messages
>   - probe messages don't automatically imply a standalone daemon
>   - add wording where probe details interact (F_MQ/F_CONFIG)
>   - define VMM and make clear QEMU is only one of many potential VMMs
>   - reword commit message
> ---
>  docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>  hw/virtio/vhost-user.c      |  8 ++++
>  2 files changed, 88 insertions(+), 10 deletions(-)
>
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index 5a070adbc1..ba3b5e07b7 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -7,6 +7,7 @@ Vhost-user Protocol
>  ..
>    Copyright 2014 Virtual Open Systems Sarl.
>    Copyright 2019 Intel Corporation
> +  Copyright 2023 Linaro Ltd
>    Licence: This work is licensed under the terms of the GNU GPL,
>             version 2 or later. See the COPYING file in the top-level
>             directory.
> @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication,
> *front-end* and
>  *back-end*. The *front-end* is the application that shares its
> virtqueues, in
>  our case QEMU. The *back-end* is the consumer of the virtqueues.
>
> -In the current implementation QEMU is the *front-end*, and the *back-end*
> -is the external process consuming the virtio queues, for example a
> -software Ethernet switch running in user space, such as Snabbswitch,
> -or a block device back-end processing read & write to a virtual
> -disk. In order to facilitate interoperability between various back-end
> -implementations, it is recommended to follow the :ref:`Backend program
> -conventions <backend_conventions>`.
> +In the current implementation a Virtual Machine Manager (VMM) such as
> +QEMU is the *front-end*, and the *back-end* is the external process
> +consuming the virtio queues, for example a software Ethernet switch
> +running in user space, such as Snabbswitch, or a block device back-end
> +processing read & write to a virtual disk. In order to facilitate
> +interoperability between various back-end implementations, it is
> +recommended to follow the :ref:`Backend program conventions
> +<backend_conventions>`.
>
>  The *front-end* and *back-end* can be either a client (i.e. connecting) or
>  server (listening) in the socket communication.
>
> +Probing device details
> +----------------------
> +
> +Traditionally the vhost-user daemon *back-end* shares configuration
> +responsibilities with the VMM *front-end* which needs to know certain
> +key bits of information about the device. This means the VMM needs to
> +define at least a minimal stub for each VirtIO device it wants to
> +support. If the daemon supports the right set of protocol features the
> +VMM can probe the daemon for the information it needs to setup the
> +device. See :ref:`Probing features for standalone daemons
> +<probing_features>` for more details.
> +
> +
>  Support for platforms other than Linux
>  --------------------------------------
>
> @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>  * ``VHOST_USER_GET_VRING_BASE``
>  * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>  * ``VHOST_USER_GET_INFLIGHT_FD`` (if
> ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> +* ``VHOST_USER_GET_BACKEND_SPECS`` (if
> ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>
>  .. seealso::
>
> @@ -396,9 +412,10 @@ must support changing some configuration aspects on
> the fly.
>  Multiple queue support
>  ----------------------
>
> -Many devices have a fixed number of virtqueues.  In this case the
> front-end
> -already knows the number of available virtqueues without communicating
> with the
> -back-end.
> +Many devices have a fixed number of virtqueues. In this case the
> +*front-end* usually already knows the number of available virtqueues
> +without communicating with the back-end. For standalone daemons this
> +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>
>  Some devices do not have a fixed number of virtqueues.  Instead the
> maximum
>  number of virtqueues is chosen by the back-end.  The number can depend on
> host
> @@ -885,6 +902,23 @@ Protocol features
>    #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
>    #define VHOST_USER_PROTOCOL_F_STATUS               16
>    #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
> +  #define VHOST_USER_PROTOCOL_F_PROBE                18
> +
> +.. _probing_features:
> +
> +Probing features for standalone daemons
> +---------------------------------------
> +
> +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> +of additional messages which allow the *front-end* to probe details
> +about the VirtIO device from the *back-end*. However for a *back-end*
> +to be described as standalone it must also support:
> +
> +  * ``VHOST_USER_PROTOCOL_F_STATUS``
> +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> +
> +which are required to ensure the *back-end* daemon can operate
> +without the *front-end* managing some aspects of its configuration.
>
>  Front-end message types
>  -----------------------
> @@ -1440,6 +1474,42 @@ Front-end message types
>    query the back-end for its device status as defined in the Virtio
>    specification.
>
> +``VHOST_USER_GET_DEVICE_ID``
> +  :id: 41
> +  :request payload: N/A
> +  :reply payload: ``u32``
> +
> +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> +  successfully negotiated, this message is submitted by the front-end
> +  to query what VirtIO device the back-end support. This is intended
> +  to remove the need for the front-end to know ahead of time what the
> +  VirtIO device the backend emulates is.
> +
> +``VHOST_USER_GET_CONFIG_SIZE``
> +  :id: 42
> +  :request payload: N/A
> +  :reply payload: ``u32``
> +
> +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> +  successfully negotiated, this message is submitted by the front-end
> +  to query the size of the VirtIO device's config space. This is
> +  intended to remove the need for the front-end to know ahead of time
> +  what the size is. Replying with 0 when
> +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> +  an bug.
> +
> +``VHOST_USER_GET_MIN_VQ``
> +  :id: 43
> +  :request payload: N/A
> +  :reply payload: ``u32``
> +
> +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> +  successfully negotiated, this message is submitted by the front-end to
> +  query minimum number of VQ's required to support the device. A
> +  device may support more than this number of VQ's if it advertises
> +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> +  indicate a bug.
>
>
Maybe I lack some background, but not sure what min_vq is here?
This looks like quering the number of VQs the backend requires/uses.
Which, in case of MQ, it may be bigger (which is where I assume comes the
`min`
part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).

Couldn't we reuse the `VHOST_USER_GET_QUEUE_NUM` type for this?


>  Back-end message types
>  ----------------------
> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> index 8dcf049d42..4d433cdf2b 100644
> --- a/hw/virtio/vhost-user.c
> +++ b/hw/virtio/vhost-user.c
> @@ -202,6 +202,13 @@ typedef struct VhostUserInflight {
>      uint16_t queue_size;
>  } VhostUserInflight;
>
> +typedef struct VhostUserBackendSpecs {
> +    uint32_t device_id;
> +    uint32_t config_size;
> +    uint32_t min_vqs;
> +    uint32_t max_vqs;
> +} VhostUserBackendSpecs;
> +
>  typedef struct {
>      VhostUserRequest request;
>
> @@ -226,6 +233,7 @@ typedef union {
>          VhostUserCryptoSession session;
>          VhostUserVringArea area;
>          VhostUserInflight inflight;
> +        VhostUserBackendSpecs specs;
>  } VhostUserPayload;
>
>  typedef struct VhostUserMsg {
> --
> 2.39.2
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>
>

Albert Esteve <aesteve@redhat.com> writes:

> This looks great! Thanks for this proposal.
>
> On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>  Currently QEMU has to know some details about the VirtIO device
>  supported by a vhost-user daemon to be able to setup the guest. This
>  makes it hard for QEMU to add support for additional vhost-user
>  daemons without adding specific stubs for each additional VirtIO
>  device.
>
>  This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>  which the back-end can advertise which allows a probe message to be
>  sent to get all the details QEMU needs to know in one message.
>
>  Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>  VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>  daemons which are capable of handling all aspects of the VirtIO
>  transactions with only a generic stub on the QEMU side. These daemons
>  can also be used without QEMU in situations where there isn't a full
>  VMM managing their setup.
>
>  Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
>  ---
>  v2
>    - dropped F_STANDALONE in favour of F_PROBE
>    - split probe details across several messages
>    - probe messages don't automatically imply a standalone daemon
>    - add wording where probe details interact (F_MQ/F_CONFIG)
>    - define VMM and make clear QEMU is only one of many potential VMMs
>    - reword commit message
>  ---
>   docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>   hw/virtio/vhost-user.c      |  8 ++++
>   2 files changed, 88 insertions(+), 10 deletions(-)
>
>  diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>  index 5a070adbc1..ba3b5e07b7 100644
>  --- a/docs/interop/vhost-user.rst
>  +++ b/docs/interop/vhost-user.rst
>  @@ -7,6 +7,7 @@ Vhost-user Protocol
>   ..
>     Copyright 2014 Virtual Open Systems Sarl.
>     Copyright 2019 Intel Corporation
>  +  Copyright 2023 Linaro Ltd
>     Licence: This work is licensed under the terms of the GNU GPL,
>              version 2 or later. See the COPYING file in the top-level
>              directory.
>  @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>   *back-end*. The *front-end* is the application that shares its virtqueues, in
>   our case QEMU. The *back-end* is the consumer of the virtqueues.
>
>  -In the current implementation QEMU is the *front-end*, and the *back-end*
>  -is the external process consuming the virtio queues, for example a
>  -software Ethernet switch running in user space, such as Snabbswitch,
>  -or a block device back-end processing read & write to a virtual
>  -disk. In order to facilitate interoperability between various back-end
>  -implementations, it is recommended to follow the :ref:`Backend program
>  -conventions <backend_conventions>`.
>  +In the current implementation a Virtual Machine Manager (VMM) such as
>  +QEMU is the *front-end*, and the *back-end* is the external process
>  +consuming the virtio queues, for example a software Ethernet switch
>  +running in user space, such as Snabbswitch, or a block device back-end
>  +processing read & write to a virtual disk. In order to facilitate
>  +interoperability between various back-end implementations, it is
>  +recommended to follow the :ref:`Backend program conventions
>  +<backend_conventions>`.
>
>   The *front-end* and *back-end* can be either a client (i.e. connecting) or
>   server (listening) in the socket communication.
>
>  +Probing device details
>  +----------------------
>  +
>  +Traditionally the vhost-user daemon *back-end* shares configuration
>  +responsibilities with the VMM *front-end* which needs to know certain
>  +key bits of information about the device. This means the VMM needs to
>  +define at least a minimal stub for each VirtIO device it wants to
>  +support. If the daemon supports the right set of protocol features the
>  +VMM can probe the daemon for the information it needs to setup the
>  +device. See :ref:`Probing features for standalone daemons
>  +<probing_features>` for more details.
>  +
>  +
>   Support for platforms other than Linux
>   --------------------------------------
>
>  @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>   * ``VHOST_USER_GET_VRING_BASE``
>   * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>   * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>  +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>
>   .. seealso::
>
>  @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>   Multiple queue support
>   ----------------------
>
>  -Many devices have a fixed number of virtqueues.  In this case the front-end
>  -already knows the number of available virtqueues without communicating with the
>  -back-end.
>  +Many devices have a fixed number of virtqueues. In this case the
>  +*front-end* usually already knows the number of available virtqueues
>  +without communicating with the back-end. For standalone daemons this
>  +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>
>   Some devices do not have a fixed number of virtqueues.  Instead the maximum
>   number of virtqueues is chosen by the back-end.  The number can depend on host
>  @@ -885,6 +902,23 @@ Protocol features
>     #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
>     #define VHOST_USER_PROTOCOL_F_STATUS               16
>     #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
>  +  #define VHOST_USER_PROTOCOL_F_PROBE                18
>  +
>  +.. _probing_features:
>  +
>  +Probing features for standalone daemons
>  +---------------------------------------
>  +
>  +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>  +of additional messages which allow the *front-end* to probe details
>  +about the VirtIO device from the *back-end*. However for a *back-end*
>  +to be described as standalone it must also support:
>  +
>  +  * ``VHOST_USER_PROTOCOL_F_STATUS``
>  +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>  +
>  +which are required to ensure the *back-end* daemon can operate
>  +without the *front-end* managing some aspects of its configuration.
>
>   Front-end message types
>   -----------------------
>  @@ -1440,6 +1474,42 @@ Front-end message types
>     query the back-end for its device status as defined in the Virtio
>     specification.
>
>  +``VHOST_USER_GET_DEVICE_ID``
>  +  :id: 41
>  +  :request payload: N/A
>  +  :reply payload: ``u32``
>  +
>  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>  +  successfully negotiated, this message is submitted by the front-end
>  +  to query what VirtIO device the back-end support. This is intended
>  +  to remove the need for the front-end to know ahead of time what the
>  +  VirtIO device the backend emulates is.
>  +
>  +``VHOST_USER_GET_CONFIG_SIZE``
>  +  :id: 42
>  +  :request payload: N/A
>  +  :reply payload: ``u32``
>  +
>  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>  +  successfully negotiated, this message is submitted by the front-end
>  +  to query the size of the VirtIO device's config space. This is
>  +  intended to remove the need for the front-end to know ahead of time
>  +  what the size is. Replying with 0 when
>  +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>  +  an bug.
>  +
>  +``VHOST_USER_GET_MIN_VQ``
>  +  :id: 43
>  +  :request payload: N/A
>  +  :reply payload: ``u32``
>  +
>  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>  +  successfully negotiated, this message is submitted by the front-end to
>  +  query minimum number of VQ's required to support the device. A
>  +  device may support more than this number of VQ's if it advertises
>  +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>  +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>  +  indicate a bug.
>
> Maybe I lack some background, but not sure what min_vq is here?

There will be a minimum number of queues you need to support the device.
For example the virtio-sound spec specifies you need four queues:
control, event, tx, rx

> This looks like quering the number of VQs the backend requires/uses.
> Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
> part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).

The MQ extension is currently used by networking but in theory any
device could attempt to parallelism by extending the number of virt
queues needed. So for net you get:

  receiveq1
  transmitq1
  optional controlq

So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
negotiated). However VHOST_USER_GET_QUEUE_NUM is only usable if
VIRTIO_NET_F_MQ has been negotiated and could report more.

>
> Couldn't we reuse the `VHOST_USER_GET_QUEUE_NUM` type for this?
>  
>   Back-end message types
>   ----------------------
>  diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
>  index 8dcf049d42..4d433cdf2b 100644
>  --- a/hw/virtio/vhost-user.c
>  +++ b/hw/virtio/vhost-user.c
>  @@ -202,6 +202,13 @@ typedef struct VhostUserInflight {
>       uint16_t queue_size;
>   } VhostUserInflight;
>
>  +typedef struct VhostUserBackendSpecs {
>  +    uint32_t device_id;
>  +    uint32_t config_size;
>  +    uint32_t min_vqs;
>  +    uint32_t max_vqs;
>  +} VhostUserBackendSpecs;
>  +
>   typedef struct {
>       VhostUserRequest request;
>
>  @@ -226,6 +233,7 @@ typedef union {
>           VhostUserCryptoSession session;
>           VhostUserVringArea area;
>           VhostUserInflight inflight;
>  +        VhostUserBackendSpecs specs;

Oops these snuck in, I shall clean them up

>   } VhostUserPayload;
>
>   typedef struct VhostUserMsg {
>  -- 
>  2.39.2
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>  For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

On Tue, Sep 5, 2023 at 11:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:

>
> Albert Esteve <aesteve@redhat.com> writes:
>
> > This looks great! Thanks for this proposal.
> >
> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org>
> wrote:
> >
> >  Currently QEMU has to know some details about the VirtIO device
> >  supported by a vhost-user daemon to be able to setup the guest. This
> >  makes it hard for QEMU to add support for additional vhost-user
> >  daemons without adding specific stubs for each additional VirtIO
> >  device.
> >
> >  This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> >  which the back-end can advertise which allows a probe message to be
> >  sent to get all the details QEMU needs to know in one message.
> >
> >  Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> >  VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> >  daemons which are capable of handling all aspects of the VirtIO
> >  transactions with only a generic stub on the QEMU side. These daemons
> >  can also be used without QEMU in situations where there isn't a full
> >  VMM managing their setup.
> >
> >  Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >
> >  ---
> >  v2
> >    - dropped F_STANDALONE in favour of F_PROBE
> >    - split probe details across several messages
> >    - probe messages don't automatically imply a standalone daemon
> >    - add wording where probe details interact (F_MQ/F_CONFIG)
> >    - define VMM and make clear QEMU is only one of many potential VMMs
> >    - reword commit message
> >  ---
> >   docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> >   hw/virtio/vhost-user.c      |  8 ++++
> >   2 files changed, 88 insertions(+), 10 deletions(-)
> >
> >  diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> >  index 5a070adbc1..ba3b5e07b7 100644
> >  --- a/docs/interop/vhost-user.rst
> >  +++ b/docs/interop/vhost-user.rst
> >  @@ -7,6 +7,7 @@ Vhost-user Protocol
> >   ..
> >     Copyright 2014 Virtual Open Systems Sarl.
> >     Copyright 2019 Intel Corporation
> >  +  Copyright 2023 Linaro Ltd
> >     Licence: This work is licensed under the terms of the GNU GPL,
> >              version 2 or later. See the COPYING file in the top-level
> >              directory.
> >  @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication,
> *front-end* and
> >   *back-end*. The *front-end* is the application that shares its
> virtqueues, in
> >   our case QEMU. The *back-end* is the consumer of the virtqueues.
> >
> >  -In the current implementation QEMU is the *front-end*, and the
> *back-end*
> >  -is the external process consuming the virtio queues, for example a
> >  -software Ethernet switch running in user space, such as Snabbswitch,
> >  -or a block device back-end processing read & write to a virtual
> >  -disk. In order to facilitate interoperability between various back-end
> >  -implementations, it is recommended to follow the :ref:`Backend program
> >  -conventions <backend_conventions>`.
> >  +In the current implementation a Virtual Machine Manager (VMM) such as
> >  +QEMU is the *front-end*, and the *back-end* is the external process
> >  +consuming the virtio queues, for example a software Ethernet switch
> >  +running in user space, such as Snabbswitch, or a block device back-end
> >  +processing read & write to a virtual disk. In order to facilitate
> >  +interoperability between various back-end implementations, it is
> >  +recommended to follow the :ref:`Backend program conventions
> >  +<backend_conventions>`.
> >
> >   The *front-end* and *back-end* can be either a client (i.e.
> connecting) or
> >   server (listening) in the socket communication.
> >
> >  +Probing device details
> >  +----------------------
> >  +
> >  +Traditionally the vhost-user daemon *back-end* shares configuration
> >  +responsibilities with the VMM *front-end* which needs to know certain
> >  +key bits of information about the device. This means the VMM needs to
> >  +define at least a minimal stub for each VirtIO device it wants to
> >  +support. If the daemon supports the right set of protocol features the
> >  +VMM can probe the daemon for the information it needs to setup the
> >  +device. See :ref:`Probing features for standalone daemons
> >  +<probing_features>` for more details.
> >  +
> >  +
> >   Support for platforms other than Linux
> >   --------------------------------------
> >
> >  @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> >   * ``VHOST_USER_GET_VRING_BASE``
> >   * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> >   * ``VHOST_USER_GET_INFLIGHT_FD`` (if
> ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> >  +* ``VHOST_USER_GET_BACKEND_SPECS`` (if
> ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >
> >   .. seealso::
> >
> >  @@ -396,9 +412,10 @@ must support changing some configuration aspects
> on the fly.
> >   Multiple queue support
> >   ----------------------
> >
> >  -Many devices have a fixed number of virtqueues.  In this case the
> front-end
> >  -already knows the number of available virtqueues without communicating
> with the
> >  -back-end.
> >  +Many devices have a fixed number of virtqueues. In this case the
> >  +*front-end* usually already knows the number of available virtqueues
> >  +without communicating with the back-end. For standalone daemons this
> >  +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
> >
> >   Some devices do not have a fixed number of virtqueues.  Instead the
> maximum
> >   number of virtqueues is chosen by the back-end.  The number can depend
> on host
> >  @@ -885,6 +902,23 @@ Protocol features
> >     #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
> >     #define VHOST_USER_PROTOCOL_F_STATUS               16
> >     #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
> >  +  #define VHOST_USER_PROTOCOL_F_PROBE                18
> >  +
> >  +.. _probing_features:
> >  +
> >  +Probing features for standalone daemons
> >  +---------------------------------------
> >  +
> >  +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> >  +of additional messages which allow the *front-end* to probe details
> >  +about the VirtIO device from the *back-end*. However for a *back-end*
> >  +to be described as standalone it must also support:
> >  +
> >  +  * ``VHOST_USER_PROTOCOL_F_STATUS``
> >  +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> >  +
> >  +which are required to ensure the *back-end* daemon can operate
> >  +without the *front-end* managing some aspects of its configuration.
> >
> >   Front-end message types
> >   -----------------------
> >  @@ -1440,6 +1474,42 @@ Front-end message types
> >     query the back-end for its device status as defined in the Virtio
> >     specification.
> >
> >  +``VHOST_USER_GET_DEVICE_ID``
> >  +  :id: 41
> >  +  :request payload: N/A
> >  +  :reply payload: ``u32``
> >  +
> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >  +  successfully negotiated, this message is submitted by the front-end
> >  +  to query what VirtIO device the back-end support. This is intended
> >  +  to remove the need for the front-end to know ahead of time what the
> >  +  VirtIO device the backend emulates is.
> >  +
> >  +``VHOST_USER_GET_CONFIG_SIZE``
> >  +  :id: 42
> >  +  :request payload: N/A
> >  +  :reply payload: ``u32``
> >  +
> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >  +  successfully negotiated, this message is submitted by the front-end
> >  +  to query the size of the VirtIO device's config space. This is
> >  +  intended to remove the need for the front-end to know ahead of time
> >  +  what the size is. Replying with 0 when
> >  +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> >  +  an bug.
> >  +
> >  +``VHOST_USER_GET_MIN_VQ``
> >  +  :id: 43
> >  +  :request payload: N/A
> >  +  :reply payload: ``u32``
> >  +
> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >  +  successfully negotiated, this message is submitted by the front-end
> to
> >  +  query minimum number of VQ's required to support the device. A
> >  +  device may support more than this number of VQ's if it advertises
> >  +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> >  +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> >  +  indicate a bug.
> >
> > Maybe I lack some background, but not sure what min_vq is here?
>
> There will be a minimum number of queues you need to support the device.
> For example the virtio-sound spec specifies you need four queues:
> control, event, tx, rx
>
> > This looks like quering the number of VQs the backend requires/uses.
> > Which, in case of MQ, it may be bigger (which is where I assume comes
> the `min`
> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
>
> The MQ extension is currently used by networking but in theory any
> device could attempt to parallelism by extending the number of virt
> queues needed. So for net you get:
>
>   receiveq1
>   transmitq1
>   optional controlq
>
> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
> negotiated). However VHOST_USER_GET_QUEUE_NUM is only usable if
> VIRTIO_NET_F_MQ has been negotiated and could report more.
>
>
Ah I see, I understand it better now. It is a pity that we cannot
multipurpose
the GET_QUEUE_NUM request.

FWIW:
Acked-by: Albert Esteve <aesteve@redhat.com>


> >
> > Couldn't we reuse the `VHOST_USER_GET_QUEUE_NUM` type for this?
> >
> >   Back-end message types
> >   ----------------------
> >  diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> >  index 8dcf049d42..4d433cdf2b 100644
> >  --- a/hw/virtio/vhost-user.c
> >  +++ b/hw/virtio/vhost-user.c
> >  @@ -202,6 +202,13 @@ typedef struct VhostUserInflight {
> >       uint16_t queue_size;
> >   } VhostUserInflight;
> >
> >  +typedef struct VhostUserBackendSpecs {
> >  +    uint32_t device_id;
> >  +    uint32_t config_size;
> >  +    uint32_t min_vqs;
> >  +    uint32_t max_vqs;
> >  +} VhostUserBackendSpecs;
> >  +
> >   typedef struct {
> >       VhostUserRequest request;
> >
> >  @@ -226,6 +233,7 @@ typedef union {
> >           VhostUserCryptoSession session;
> >           VhostUserVringArea area;
> >           VhostUserInflight inflight;
> >  +        VhostUserBackendSpecs specs;
>
> Oops these snuck in, I shall clean them up
>
> >   } VhostUserPayload;
> >
> >   typedef struct VhostUserMsg {
> >  --
> >  2.39.2
> >
> >  ---------------------------------------------------------------------
> >  To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >  For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro
>
>

On Fri, Sep 01, 2023 at 12:00:18PM +0100, Alex Bennée wrote:
> Currently QEMU has to know some details about the VirtIO device
> supported by a vhost-user daemon to be able to setup the guest. This
> makes it hard for QEMU to add support for additional vhost-user
> daemons without adding specific stubs for each additional VirtIO
> device.
> 
> This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> which the back-end can advertise which allows a probe message to be
> sent to get all the details QEMU needs to know in one message.
> 
> Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> daemons which are capable of handling all aspects of the VirtIO
> transactions with only a generic stub on the QEMU side. These daemons
> can also be used without QEMU in situations where there isn't a full
> VMM managing their setup.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

I think the mindset for this change should be "vhost-user is becoming a
VIRTIO Transport". VIRTIO Transports have a reasonably well-defined
feature set in the VIRTIO specification. The goal should be to cover
every VIRTIO Transport operation via vhost-user protocol messages so
that the VIRTIO device model can be fully conveyed over vhost-user.

Anything less is yet another ad-hoc protocol extension that will lead to
more bugs and hacks when it turns out some VIRTIO devices cannot be
expressed due to limitations in the protocol.

This requires going through the VIRTIO spec to find a correspondence
between virtio-pci/virtio-mmio/virtio-ccw's interfaces and vhost-user
protocol messages. In most cases vhost-user already offers messages and
your patch adds more of what is missing. I think this effort is already
very close but missing the final check that it really matches the VIRTIO
spec.

Please do the comparison against the VIRTIO Transports and then adjust
this patch to make it clear that the back-end is becoming a full-fledged
VIRTIO Transport:
- The name of the patch series should reflect that.
- The vhost-user protocol feature should be named F_TRANSPORT.
- The messages added in this patch should have a 1:1 correspondence with
  the VIRTIO spec including using the same terminology for consistency.

Sorry for the hassle, but I think this is a really crucial point where
we have the chance to make vhost-user work smoothly in the future...but
only if we can faithfully expose VIRTIO Transport semantics.

> 
> ---
> v2
>   - dropped F_STANDALONE in favour of F_PROBE
>   - split probe details across several messages
>   - probe messages don't automatically imply a standalone daemon
>   - add wording where probe details interact (F_MQ/F_CONFIG)
>   - define VMM and make clear QEMU is only one of many potential VMMs
>   - reword commit message
> ---
>  docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>  hw/virtio/vhost-user.c      |  8 ++++
>  2 files changed, 88 insertions(+), 10 deletions(-)
> 
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index 5a070adbc1..ba3b5e07b7 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -7,6 +7,7 @@ Vhost-user Protocol
>  ..
>    Copyright 2014 Virtual Open Systems Sarl.
>    Copyright 2019 Intel Corporation
> +  Copyright 2023 Linaro Ltd
>    Licence: This work is licensed under the terms of the GNU GPL,
>             version 2 or later. See the COPYING file in the top-level
>             directory.
> @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>  *back-end*. The *front-end* is the application that shares its virtqueues, in
>  our case QEMU. The *back-end* is the consumer of the virtqueues.
>  
> -In the current implementation QEMU is the *front-end*, and the *back-end*
> -is the external process consuming the virtio queues, for example a
> -software Ethernet switch running in user space, such as Snabbswitch,
> -or a block device back-end processing read & write to a virtual
> -disk. In order to facilitate interoperability between various back-end
> -implementations, it is recommended to follow the :ref:`Backend program
> -conventions <backend_conventions>`.
> +In the current implementation a Virtual Machine Manager (VMM) such as
> +QEMU is the *front-end*, and the *back-end* is the external process
> +consuming the virtio queues, for example a software Ethernet switch
> +running in user space, such as Snabbswitch, or a block device back-end
> +processing read & write to a virtual disk. In order to facilitate
> +interoperability between various back-end implementations, it is
> +recommended to follow the :ref:`Backend program conventions
> +<backend_conventions>`.
>  
>  The *front-end* and *back-end* can be either a client (i.e. connecting) or
>  server (listening) in the socket communication.
>  
> +Probing device details
> +----------------------
> +
> +Traditionally the vhost-user daemon *back-end* shares configuration
> +responsibilities with the VMM *front-end* which needs to know certain
> +key bits of information about the device. This means the VMM needs to
> +define at least a minimal stub for each VirtIO device it wants to
> +support. If the daemon supports the right set of protocol features the
> +VMM can probe the daemon for the information it needs to setup the
> +device.

"... without a per-device stub in the VMM"

This makes it clear that this sentence is describing an alternative
to the per-device stub in the VMM.

> See :ref:`Probing features for standalone daemons
> +<probing_features>` for more details.

The current section is named "Probing device details" and one being
reference is called "Probing features for standalone daemons". Are
"features" or "device details" two terms for the same thing? Why
"daemons" and not "back-end"?

I suggest calling this section "Standalone back-ends" and the other
section "Probing standalone back-ends" to keep the terminology
consistent.

> +
> +
>  Support for platforms other than Linux
>  --------------------------------------
>  
> @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>  * ``VHOST_USER_GET_VRING_BASE``
>  * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>  * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)

F_STANDALONE vs F_PROBE

"SPECS" vs "features" vs "details".

Please be consistent.

>  
>  .. seealso::
>  
> @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>  Multiple queue support
>  ----------------------
>  
> -Many devices have a fixed number of virtqueues.  In this case the front-end
> -already knows the number of available virtqueues without communicating with the
> -back-end.
> +Many devices have a fixed number of virtqueues. In this case the
> +*front-end* usually already knows the number of available virtqueues
> +without communicating with the back-end. For standalone daemons this

"Usually" is vague. It's possible to be precise:

  In this case a front-end that is aware of the device type already
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  knows the number of available virtqueues without communicating with
  the back-end.

> +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.

Then this sentence can be adjusted to:

  When the front-end is not aware of the device type, the number can be
  probed with the ``VHOST_USER_GET_MIN_VQ`` message.

>  
>  Some devices do not have a fixed number of virtqueues.  Instead the maximum
>  number of virtqueues is chosen by the back-end.  The number can depend on host
> @@ -885,6 +902,23 @@ Protocol features
>    #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
>    #define VHOST_USER_PROTOCOL_F_STATUS               16
>    #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
> +  #define VHOST_USER_PROTOCOL_F_PROBE                18
> +
> +.. _probing_features:
> +
> +Probing features for standalone daemons
> +---------------------------------------
> +
> +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> +of additional messages which allow the *front-end* to probe details
> +about the VirtIO device from the *back-end*. However for a *back-end*
> +to be described as standalone it must also support:
> +
> +  * ``VHOST_USER_PROTOCOL_F_STATUS``
> +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> +
> +which are required to ensure the *back-end* daemon can operate
> +without the *front-end* managing some aspects of its configuration.
>  
>  Front-end message types
>  -----------------------
> @@ -1440,6 +1474,42 @@ Front-end message types
>    query the back-end for its device status as defined in the Virtio
>    specification.
>  
> +``VHOST_USER_GET_DEVICE_ID``
> +  :id: 41
> +  :request payload: N/A
> +  :reply payload: ``u32``
> +
> +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> +  successfully negotiated, this message is submitted by the front-end
> +  to query what VirtIO device the back-end support. This is intended
> +  to remove the need for the front-end to know ahead of time what the
> +  VirtIO device the backend emulates is.

"... VIRTIO device type that the backend emulates is."

"Device type" is the name used in the VIRTIO spec.

> +
> +``VHOST_USER_GET_CONFIG_SIZE``
> +  :id: 42
> +  :request payload: N/A
> +  :reply payload: ``u32``
> +
> +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> +  successfully negotiated, this message is submitted by the front-end
> +  to query the size of the VirtIO device's config space. This is
> +  intended to remove the need for the front-end to know ahead of time
> +  what the size is. Replying with 0 when
> +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> +  an bug.

"a bug"

What is the harm in returning 0 when the device has an empty
Configuration Space like the Entropy device, the I2C Adapter, the SCMI
device, etc?

> +
> +``VHOST_USER_GET_MIN_VQ``
> +  :id: 43
> +  :request payload: N/A
> +  :reply payload: ``u32``
> +
> +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> +  successfully negotiated, this message is submitted by the front-end to
> +  query minimum number of VQ's required to support the device. A
> +  device may support more than this number of VQ's if it advertises
> +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> +  indicate a bug.

What is the purpose of this message? I don't see an equivalent in the
VIRTIO specification.

>  
>  Back-end message types
>  ----------------------
> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> index 8dcf049d42..4d433cdf2b 100644
> --- a/hw/virtio/vhost-user.c
> +++ b/hw/virtio/vhost-user.c
> @@ -202,6 +202,13 @@ typedef struct VhostUserInflight {
>      uint16_t queue_size;
>  } VhostUserInflight;
>  
> +typedef struct VhostUserBackendSpecs {
> +    uint32_t device_id;
> +    uint32_t config_size;
> +    uint32_t min_vqs;
> +    uint32_t max_vqs;
> +} VhostUserBackendSpecs;

This message is undocumented? I think it may be outdated and you split
it up into individual messages.

> +
>  typedef struct {
>      VhostUserRequest request;
>  
> @@ -226,6 +233,7 @@ typedef union {
>          VhostUserCryptoSession session;
>          VhostUserVringArea area;
>          VhostUserInflight inflight;
> +        VhostUserBackendSpecs specs;
>  } VhostUserPayload;
>  
>  typedef struct VhostUserMsg {
> -- 
> 2.39.2
>

On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
> 
> Albert Esteve <aesteve@redhat.com> writes:
> 
> > This looks great! Thanks for this proposal.
> >
> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> >  Currently QEMU has to know some details about the VirtIO device
> >  supported by a vhost-user daemon to be able to setup the guest. This
> >  makes it hard for QEMU to add support for additional vhost-user
> >  daemons without adding specific stubs for each additional VirtIO
> >  device.
> >
> >  This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> >  which the back-end can advertise which allows a probe message to be
> >  sent to get all the details QEMU needs to know in one message.
> >
> >  Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> >  VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> >  daemons which are capable of handling all aspects of the VirtIO
> >  transactions with only a generic stub on the QEMU side. These daemons
> >  can also be used without QEMU in situations where there isn't a full
> >  VMM managing their setup.
> >
> >  Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >
> >  ---
> >  v2
> >    - dropped F_STANDALONE in favour of F_PROBE
> >    - split probe details across several messages
> >    - probe messages don't automatically imply a standalone daemon
> >    - add wording where probe details interact (F_MQ/F_CONFIG)
> >    - define VMM and make clear QEMU is only one of many potential VMMs
> >    - reword commit message
> >  ---
> >   docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> >   hw/virtio/vhost-user.c      |  8 ++++
> >   2 files changed, 88 insertions(+), 10 deletions(-)
> >
> >  diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> >  index 5a070adbc1..ba3b5e07b7 100644
> >  --- a/docs/interop/vhost-user.rst
> >  +++ b/docs/interop/vhost-user.rst
> >  @@ -7,6 +7,7 @@ Vhost-user Protocol
> >   ..
> >     Copyright 2014 Virtual Open Systems Sarl.
> >     Copyright 2019 Intel Corporation
> >  +  Copyright 2023 Linaro Ltd
> >     Licence: This work is licensed under the terms of the GNU GPL,
> >              version 2 or later. See the COPYING file in the top-level
> >              directory.
> >  @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
> >   *back-end*. The *front-end* is the application that shares its virtqueues, in
> >   our case QEMU. The *back-end* is the consumer of the virtqueues.
> >
> >  -In the current implementation QEMU is the *front-end*, and the *back-end*
> >  -is the external process consuming the virtio queues, for example a
> >  -software Ethernet switch running in user space, such as Snabbswitch,
> >  -or a block device back-end processing read & write to a virtual
> >  -disk. In order to facilitate interoperability between various back-end
> >  -implementations, it is recommended to follow the :ref:`Backend program
> >  -conventions <backend_conventions>`.
> >  +In the current implementation a Virtual Machine Manager (VMM) such as
> >  +QEMU is the *front-end*, and the *back-end* is the external process
> >  +consuming the virtio queues, for example a software Ethernet switch
> >  +running in user space, such as Snabbswitch, or a block device back-end
> >  +processing read & write to a virtual disk. In order to facilitate
> >  +interoperability between various back-end implementations, it is
> >  +recommended to follow the :ref:`Backend program conventions
> >  +<backend_conventions>`.
> >
> >   The *front-end* and *back-end* can be either a client (i.e. connecting) or
> >   server (listening) in the socket communication.
> >
> >  +Probing device details
> >  +----------------------
> >  +
> >  +Traditionally the vhost-user daemon *back-end* shares configuration
> >  +responsibilities with the VMM *front-end* which needs to know certain
> >  +key bits of information about the device. This means the VMM needs to
> >  +define at least a minimal stub for each VirtIO device it wants to
> >  +support. If the daemon supports the right set of protocol features the
> >  +VMM can probe the daemon for the information it needs to setup the
> >  +device. See :ref:`Probing features for standalone daemons
> >  +<probing_features>` for more details.
> >  +
> >  +
> >   Support for platforms other than Linux
> >   --------------------------------------
> >
> >  @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> >   * ``VHOST_USER_GET_VRING_BASE``
> >   * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> >   * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> >  +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >
> >   .. seealso::
> >
> >  @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
> >   Multiple queue support
> >   ----------------------
> >
> >  -Many devices have a fixed number of virtqueues.  In this case the front-end
> >  -already knows the number of available virtqueues without communicating with the
> >  -back-end.
> >  +Many devices have a fixed number of virtqueues. In this case the
> >  +*front-end* usually already knows the number of available virtqueues
> >  +without communicating with the back-end. For standalone daemons this
> >  +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
> >
> >   Some devices do not have a fixed number of virtqueues.  Instead the maximum
> >   number of virtqueues is chosen by the back-end.  The number can depend on host
> >  @@ -885,6 +902,23 @@ Protocol features
> >     #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
> >     #define VHOST_USER_PROTOCOL_F_STATUS               16
> >     #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
> >  +  #define VHOST_USER_PROTOCOL_F_PROBE                18
> >  +
> >  +.. _probing_features:
> >  +
> >  +Probing features for standalone daemons
> >  +---------------------------------------
> >  +
> >  +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> >  +of additional messages which allow the *front-end* to probe details
> >  +about the VirtIO device from the *back-end*. However for a *back-end*
> >  +to be described as standalone it must also support:
> >  +
> >  +  * ``VHOST_USER_PROTOCOL_F_STATUS``
> >  +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> >  +
> >  +which are required to ensure the *back-end* daemon can operate
> >  +without the *front-end* managing some aspects of its configuration.
> >
> >   Front-end message types
> >   -----------------------
> >  @@ -1440,6 +1474,42 @@ Front-end message types
> >     query the back-end for its device status as defined in the Virtio
> >     specification.
> >
> >  +``VHOST_USER_GET_DEVICE_ID``
> >  +  :id: 41
> >  +  :request payload: N/A
> >  +  :reply payload: ``u32``
> >  +
> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >  +  successfully negotiated, this message is submitted by the front-end
> >  +  to query what VirtIO device the back-end support. This is intended
> >  +  to remove the need for the front-end to know ahead of time what the
> >  +  VirtIO device the backend emulates is.
> >  +
> >  +``VHOST_USER_GET_CONFIG_SIZE``
> >  +  :id: 42
> >  +  :request payload: N/A
> >  +  :reply payload: ``u32``
> >  +
> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >  +  successfully negotiated, this message is submitted by the front-end
> >  +  to query the size of the VirtIO device's config space. This is
> >  +  intended to remove the need for the front-end to know ahead of time
> >  +  what the size is. Replying with 0 when
> >  +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> >  +  an bug.
> >  +
> >  +``VHOST_USER_GET_MIN_VQ``
> >  +  :id: 43
> >  +  :request payload: N/A
> >  +  :reply payload: ``u32``
> >  +
> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >  +  successfully negotiated, this message is submitted by the front-end to
> >  +  query minimum number of VQ's required to support the device. A
> >  +  device may support more than this number of VQ's if it advertises
> >  +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> >  +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> >  +  indicate a bug.
> >
> > Maybe I lack some background, but not sure what min_vq is here?
> 
> There will be a minimum number of queues you need to support the device.
> For example the virtio-sound spec specifies you need four queues:
> control, event, tx, rx

I don't understand why the front-end needs to know that? The backend
already reports the number of queues and not all of them need to be
initialized by the driver.

> 
> > This looks like quering the number of VQs the backend requires/uses.
> > Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
> 
> The MQ extension is currently used by networking but in theory any
> device could attempt to parallelism by extending the number of virt
> queues needed. So for net you get:
> 
>   receiveq1
>   transmitq1
>   optional controlq
> 
> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
> negotiated).

I'm confused. VHOST_USER_GET_MIN_VQ comes before VIRTIO Feature Bit
negotiation (VIRTIO_NET_F_CTRL_VQ).

> However VHOST_USER_GET_QUEUE_NUM is only usable if
> VIRTIO_NET_F_MQ has been negotiated and could report more.

I don't understand. This patch adds a new feature and it can require
VHOST_USER_PROTOCOL_F_MQ. There are no existing back-ends that require
backwards compatibility.

Stefan

Stefan Hajnoczi <stefanha@redhat.com> writes:

> On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>> 
>> Albert Esteve <aesteve@redhat.com> writes:
>> 
>> > This looks great! Thanks for this proposal.
>> >
>> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>> >
>> >  Currently QEMU has to know some details about the VirtIO device
>> >  supported by a vhost-user daemon to be able to setup the guest. This
>> >  makes it hard for QEMU to add support for additional vhost-user
>> >  daemons without adding specific stubs for each additional VirtIO
>> >  device.
>> >
>> >  This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>> >  which the back-end can advertise which allows a probe message to be
>> >  sent to get all the details QEMU needs to know in one message.
>> >
>> >  Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>> >  VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>> >  daemons which are capable of handling all aspects of the VirtIO
>> >  transactions with only a generic stub on the QEMU side. These daemons
>> >  can also be used without QEMU in situations where there isn't a full
>> >  VMM managing their setup.
>> >
>> >  Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> >
>> >  ---
>> >  v2
>> >    - dropped F_STANDALONE in favour of F_PROBE
>> >    - split probe details across several messages
>> >    - probe messages don't automatically imply a standalone daemon
>> >    - add wording where probe details interact (F_MQ/F_CONFIG)
>> >    - define VMM and make clear QEMU is only one of many potential VMMs
>> >    - reword commit message
>> >  ---
>> >   docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>> >   hw/virtio/vhost-user.c      |  8 ++++
>> >   2 files changed, 88 insertions(+), 10 deletions(-)
>> >
>> >  diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> >  index 5a070adbc1..ba3b5e07b7 100644
>> >  --- a/docs/interop/vhost-user.rst
>> >  +++ b/docs/interop/vhost-user.rst
>> >  @@ -7,6 +7,7 @@ Vhost-user Protocol
>> >   ..
>> >     Copyright 2014 Virtual Open Systems Sarl.
>> >     Copyright 2019 Intel Corporation
>> >  +  Copyright 2023 Linaro Ltd
>> >     Licence: This work is licensed under the terms of the GNU GPL,
>> >              version 2 or later. See the COPYING file in the top-level
>> >              directory.
>> >  @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>> >   *back-end*. The *front-end* is the application that shares its virtqueues, in
>> >   our case QEMU. The *back-end* is the consumer of the virtqueues.
>> >
>> >  -In the current implementation QEMU is the *front-end*, and the *back-end*
>> >  -is the external process consuming the virtio queues, for example a
>> >  -software Ethernet switch running in user space, such as Snabbswitch,
>> >  -or a block device back-end processing read & write to a virtual
>> >  -disk. In order to facilitate interoperability between various back-end
>> >  -implementations, it is recommended to follow the :ref:`Backend program
>> >  -conventions <backend_conventions>`.
>> >  +In the current implementation a Virtual Machine Manager (VMM) such as
>> >  +QEMU is the *front-end*, and the *back-end* is the external process
>> >  +consuming the virtio queues, for example a software Ethernet switch
>> >  +running in user space, such as Snabbswitch, or a block device back-end
>> >  +processing read & write to a virtual disk. In order to facilitate
>> >  +interoperability between various back-end implementations, it is
>> >  +recommended to follow the :ref:`Backend program conventions
>> >  +<backend_conventions>`.
>> >
>> >   The *front-end* and *back-end* can be either a client (i.e. connecting) or
>> >   server (listening) in the socket communication.
>> >
>> >  +Probing device details
>> >  +----------------------
>> >  +
>> >  +Traditionally the vhost-user daemon *back-end* shares configuration
>> >  +responsibilities with the VMM *front-end* which needs to know certain
>> >  +key bits of information about the device. This means the VMM needs to
>> >  +define at least a minimal stub for each VirtIO device it wants to
>> >  +support. If the daemon supports the right set of protocol features the
>> >  +VMM can probe the daemon for the information it needs to setup the
>> >  +device. See :ref:`Probing features for standalone daemons
>> >  +<probing_features>` for more details.
>> >  +
>> >  +
>> >   Support for platforms other than Linux
>> >   --------------------------------------
>> >
>> >  @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>> >   * ``VHOST_USER_GET_VRING_BASE``
>> >   * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>> >   * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>> >  +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>> >
>> >   .. seealso::
>> >
>> >  @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>> >   Multiple queue support
>> >   ----------------------
>> >
>> >  -Many devices have a fixed number of virtqueues.  In this case the front-end
>> >  -already knows the number of available virtqueues without communicating with the
>> >  -back-end.
>> >  +Many devices have a fixed number of virtqueues. In this case the
>> >  +*front-end* usually already knows the number of available virtqueues
>> >  +without communicating with the back-end. For standalone daemons this
>> >  +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>> >
>> >   Some devices do not have a fixed number of virtqueues.  Instead the maximum
>> >   number of virtqueues is chosen by the back-end.  The number can depend on host
>> >  @@ -885,6 +902,23 @@ Protocol features
>> >     #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
>> >     #define VHOST_USER_PROTOCOL_F_STATUS               16
>> >     #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
>> >  +  #define VHOST_USER_PROTOCOL_F_PROBE                18
>> >  +
>> >  +.. _probing_features:
>> >  +
>> >  +Probing features for standalone daemons
>> >  +---------------------------------------
>> >  +
>> >  +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>> >  +of additional messages which allow the *front-end* to probe details
>> >  +about the VirtIO device from the *back-end*. However for a *back-end*
>> >  +to be described as standalone it must also support:
>> >  +
>> >  +  * ``VHOST_USER_PROTOCOL_F_STATUS``
>> >  +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>> >  +
>> >  +which are required to ensure the *back-end* daemon can operate
>> >  +without the *front-end* managing some aspects of its configuration.
>> >
>> >   Front-end message types
>> >   -----------------------
>> >  @@ -1440,6 +1474,42 @@ Front-end message types
>> >     query the back-end for its device status as defined in the Virtio
>> >     specification.
>> >
>> >  +``VHOST_USER_GET_DEVICE_ID``
>> >  +  :id: 41
>> >  +  :request payload: N/A
>> >  +  :reply payload: ``u32``
>> >  +
>> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >  +  successfully negotiated, this message is submitted by the front-end
>> >  +  to query what VirtIO device the back-end support. This is intended
>> >  +  to remove the need for the front-end to know ahead of time what the
>> >  +  VirtIO device the backend emulates is.
>> >  +
>> >  +``VHOST_USER_GET_CONFIG_SIZE``
>> >  +  :id: 42
>> >  +  :request payload: N/A
>> >  +  :reply payload: ``u32``
>> >  +
>> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >  +  successfully negotiated, this message is submitted by the front-end
>> >  +  to query the size of the VirtIO device's config space. This is
>> >  +  intended to remove the need for the front-end to know ahead of time
>> >  +  what the size is. Replying with 0 when
>> >  +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>> >  +  an bug.
>> >  +
>> >  +``VHOST_USER_GET_MIN_VQ``
>> >  +  :id: 43
>> >  +  :request payload: N/A
>> >  +  :reply payload: ``u32``
>> >  +
>> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >  +  successfully negotiated, this message is submitted by the front-end to
>> >  +  query minimum number of VQ's required to support the device. A
>> >  +  device may support more than this number of VQ's if it advertises
>> >  +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>> >  +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>> >  +  indicate a bug.
>> >
>> > Maybe I lack some background, but not sure what min_vq is here?
>> 
>> There will be a minimum number of queues you need to support the device.
>> For example the virtio-sound spec specifies you need four queues:
>> control, event, tx, rx
>
> I don't understand why the front-end needs to know that? The backend
> already reports the number of queues and not all of them need to be
> initialized by the driver.

But how many don't need to be initialised? We can't just skip:

    /* Allocate queues */
    vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
    for (int i = 0; i < vub->num_vqs; i++) {
        g_ptr_array_add(vub->vqs,
                        virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
    }

Or are you saying just require probe-able backends to support
VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
of queues if it is not a MQ capable device?

>> > This looks like quering the number of VQs the backend requires/uses.
>> > Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
>> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
>> 
>> The MQ extension is currently used by networking but in theory any
>> device could attempt to parallelism by extending the number of virt
>> queues needed. So for net you get:
>> 
>>   receiveq1
>>   transmitq1
>>   optional controlq
>> 
>> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
>> negotiated).
>
> I'm confused. VHOST_USER_GET_MIN_VQ comes before VIRTIO Feature Bit
> negotiation (VIRTIO_NET_F_CTRL_VQ).
>
>> However VHOST_USER_GET_QUEUE_NUM is only usable if
>> VIRTIO_NET_F_MQ has been negotiated and could report more.
>
> I don't understand. This patch adds a new feature and it can require
> VHOST_USER_PROTOCOL_F_MQ. There are no existing back-ends that require
> backwards compatibility.
>
> Stefan

On Fri, 8 Sept 2023 at 02:43, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Stefan Hajnoczi <stefanha@redhat.com> writes:
>
> > On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
> >>
> >> Albert Esteve <aesteve@redhat.com> writes:
> >>
> >> > This looks great! Thanks for this proposal.
> >> >
> >> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >> >
> >> >  Currently QEMU has to know some details about the VirtIO device
> >> >  supported by a vhost-user daemon to be able to setup the guest. This
> >> >  makes it hard for QEMU to add support for additional vhost-user
> >> >  daemons without adding specific stubs for each additional VirtIO
> >> >  device.
> >> >
> >> >  This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> >> >  which the back-end can advertise which allows a probe message to be
> >> >  sent to get all the details QEMU needs to know in one message.
> >> >
> >> >  Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> >> >  VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> >> >  daemons which are capable of handling all aspects of the VirtIO
> >> >  transactions with only a generic stub on the QEMU side. These daemons
> >> >  can also be used without QEMU in situations where there isn't a full
> >> >  VMM managing their setup.
> >> >
> >> >  Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >> >
> >> >  ---
> >> >  v2
> >> >    - dropped F_STANDALONE in favour of F_PROBE
> >> >    - split probe details across several messages
> >> >    - probe messages don't automatically imply a standalone daemon
> >> >    - add wording where probe details interact (F_MQ/F_CONFIG)
> >> >    - define VMM and make clear QEMU is only one of many potential VMMs
> >> >    - reword commit message
> >> >  ---
> >> >   docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> >> >   hw/virtio/vhost-user.c      |  8 ++++
> >> >   2 files changed, 88 insertions(+), 10 deletions(-)
> >> >
> >> >  diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> >> >  index 5a070adbc1..ba3b5e07b7 100644
> >> >  --- a/docs/interop/vhost-user.rst
> >> >  +++ b/docs/interop/vhost-user.rst
> >> >  @@ -7,6 +7,7 @@ Vhost-user Protocol
> >> >   ..
> >> >     Copyright 2014 Virtual Open Systems Sarl.
> >> >     Copyright 2019 Intel Corporation
> >> >  +  Copyright 2023 Linaro Ltd
> >> >     Licence: This work is licensed under the terms of the GNU GPL,
> >> >              version 2 or later. See the COPYING file in the top-level
> >> >              directory.
> >> >  @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
> >> >   *back-end*. The *front-end* is the application that shares its virtqueues, in
> >> >   our case QEMU. The *back-end* is the consumer of the virtqueues.
> >> >
> >> >  -In the current implementation QEMU is the *front-end*, and the *back-end*
> >> >  -is the external process consuming the virtio queues, for example a
> >> >  -software Ethernet switch running in user space, such as Snabbswitch,
> >> >  -or a block device back-end processing read & write to a virtual
> >> >  -disk. In order to facilitate interoperability between various back-end
> >> >  -implementations, it is recommended to follow the :ref:`Backend program
> >> >  -conventions <backend_conventions>`.
> >> >  +In the current implementation a Virtual Machine Manager (VMM) such as
> >> >  +QEMU is the *front-end*, and the *back-end* is the external process
> >> >  +consuming the virtio queues, for example a software Ethernet switch
> >> >  +running in user space, such as Snabbswitch, or a block device back-end
> >> >  +processing read & write to a virtual disk. In order to facilitate
> >> >  +interoperability between various back-end implementations, it is
> >> >  +recommended to follow the :ref:`Backend program conventions
> >> >  +<backend_conventions>`.
> >> >
> >> >   The *front-end* and *back-end* can be either a client (i.e. connecting) or
> >> >   server (listening) in the socket communication.
> >> >
> >> >  +Probing device details
> >> >  +----------------------
> >> >  +
> >> >  +Traditionally the vhost-user daemon *back-end* shares configuration
> >> >  +responsibilities with the VMM *front-end* which needs to know certain
> >> >  +key bits of information about the device. This means the VMM needs to
> >> >  +define at least a minimal stub for each VirtIO device it wants to
> >> >  +support. If the daemon supports the right set of protocol features the
> >> >  +VMM can probe the daemon for the information it needs to setup the
> >> >  +device. See :ref:`Probing features for standalone daemons
> >> >  +<probing_features>` for more details.
> >> >  +
> >> >  +
> >> >   Support for platforms other than Linux
> >> >   --------------------------------------
> >> >
> >> >  @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> >> >   * ``VHOST_USER_GET_VRING_BASE``
> >> >   * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> >> >   * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> >> >  +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >> >
> >> >   .. seealso::
> >> >
> >> >  @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
> >> >   Multiple queue support
> >> >   ----------------------
> >> >
> >> >  -Many devices have a fixed number of virtqueues.  In this case the front-end
> >> >  -already knows the number of available virtqueues without communicating with the
> >> >  -back-end.
> >> >  +Many devices have a fixed number of virtqueues. In this case the
> >> >  +*front-end* usually already knows the number of available virtqueues
> >> >  +without communicating with the back-end. For standalone daemons this
> >> >  +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
> >> >
> >> >   Some devices do not have a fixed number of virtqueues.  Instead the maximum
> >> >   number of virtqueues is chosen by the back-end.  The number can depend on host
> >> >  @@ -885,6 +902,23 @@ Protocol features
> >> >     #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
> >> >     #define VHOST_USER_PROTOCOL_F_STATUS               16
> >> >     #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
> >> >  +  #define VHOST_USER_PROTOCOL_F_PROBE                18
> >> >  +
> >> >  +.. _probing_features:
> >> >  +
> >> >  +Probing features for standalone daemons
> >> >  +---------------------------------------
> >> >  +
> >> >  +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> >> >  +of additional messages which allow the *front-end* to probe details
> >> >  +about the VirtIO device from the *back-end*. However for a *back-end*
> >> >  +to be described as standalone it must also support:
> >> >  +
> >> >  +  * ``VHOST_USER_PROTOCOL_F_STATUS``
> >> >  +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> >> >  +
> >> >  +which are required to ensure the *back-end* daemon can operate
> >> >  +without the *front-end* managing some aspects of its configuration.
> >> >
> >> >   Front-end message types
> >> >   -----------------------
> >> >  @@ -1440,6 +1474,42 @@ Front-end message types
> >> >     query the back-end for its device status as defined in the Virtio
> >> >     specification.
> >> >
> >> >  +``VHOST_USER_GET_DEVICE_ID``
> >> >  +  :id: 41
> >> >  +  :request payload: N/A
> >> >  +  :reply payload: ``u32``
> >> >  +
> >> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >  +  successfully negotiated, this message is submitted by the front-end
> >> >  +  to query what VirtIO device the back-end support. This is intended
> >> >  +  to remove the need for the front-end to know ahead of time what the
> >> >  +  VirtIO device the backend emulates is.
> >> >  +
> >> >  +``VHOST_USER_GET_CONFIG_SIZE``
> >> >  +  :id: 42
> >> >  +  :request payload: N/A
> >> >  +  :reply payload: ``u32``
> >> >  +
> >> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >  +  successfully negotiated, this message is submitted by the front-end
> >> >  +  to query the size of the VirtIO device's config space. This is
> >> >  +  intended to remove the need for the front-end to know ahead of time
> >> >  +  what the size is. Replying with 0 when
> >> >  +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> >> >  +  an bug.
> >> >  +
> >> >  +``VHOST_USER_GET_MIN_VQ``
> >> >  +  :id: 43
> >> >  +  :request payload: N/A
> >> >  +  :reply payload: ``u32``
> >> >  +
> >> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >  +  successfully negotiated, this message is submitted by the front-end to
> >> >  +  query minimum number of VQ's required to support the device. A
> >> >  +  device may support more than this number of VQ's if it advertises
> >> >  +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> >> >  +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> >> >  +  indicate a bug.
> >> >
> >> > Maybe I lack some background, but not sure what min_vq is here?
> >>
> >> There will be a minimum number of queues you need to support the device.
> >> For example the virtio-sound spec specifies you need four queues:
> >> control, event, tx, rx
> >
> > I don't understand why the front-end needs to know that? The backend
> > already reports the number of queues and not all of them need to be
> > initialized by the driver.
>
> But how many don't need to be initialised? We can't just skip:
>
>     /* Allocate queues */
>     vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
>     for (int i = 0; i < vub->num_vqs; i++) {
>         g_ptr_array_add(vub->vqs,
>                         virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
>     }
>
> Or are you saying just require probe-able backends to support
> VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
> of queues if it is not a MQ capable device?

The front-end should prepare to allow the maximum number of virtqueues
returned by VHOST_USER_GET_QUEUE_NUM (VHOST_USER_PROTOCOL_F_MQ).

VIRTIO Transports have a way to query the maximum number of queues but
not a way to query the minimum number of queues. Why is the minimum
necessary?

Stefan

>
> >> > This looks like quering the number of VQs the backend requires/uses.
> >> > Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
> >> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
> >>
> >> The MQ extension is currently used by networking but in theory any
> >> device could attempt to parallelism by extending the number of virt
> >> queues needed. So for net you get:
> >>
> >>   receiveq1
> >>   transmitq1
> >>   optional controlq
> >>
> >> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
> >> negotiated).
> >
> > I'm confused. VHOST_USER_GET_MIN_VQ comes before VIRTIO Feature Bit
> > negotiation (VIRTIO_NET_F_CTRL_VQ).
> >
> >> However VHOST_USER_GET_QUEUE_NUM is only usable if
> >> VIRTIO_NET_F_MQ has been negotiated and could report more.
> >
> > I don't understand. This patch adds a new feature and it can require
> > VHOST_USER_PROTOCOL_F_MQ. There are no existing back-ends that require
> > backwards compatibility.
> >
> > Stefan
>
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro
>

Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Fri, 8 Sept 2023 at 02:43, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Stefan Hajnoczi <stefanha@redhat.com> writes:
>>
>> > On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>> >>
>> >> Albert Esteve <aesteve@redhat.com> writes:
>> >>
>> >> > This looks great! Thanks for this proposal.
>> >> >
>> >> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>> >> >
>> >> >  Currently QEMU has to know some details about the VirtIO device
>> >> >  supported by a vhost-user daemon to be able to setup the guest. This
>> >> >  makes it hard for QEMU to add support for additional vhost-user
>> >> >  daemons without adding specific stubs for each additional VirtIO
>> >> >  device.
>> >> >
>> >> >  This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>> >> >  which the back-end can advertise which allows a probe message to be
>> >> >  sent to get all the details QEMU needs to know in one message.
>> >> >
>> >> >  Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>> >> >  VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>> >> >  daemons which are capable of handling all aspects of the VirtIO
>> >> >  transactions with only a generic stub on the QEMU side. These daemons
>> >> >  can also be used without QEMU in situations where there isn't a full
>> >> >  VMM managing their setup.
>> >> >
>> >> >  Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> >> >
>> >> >  ---
>> >> >  v2
>> >> >    - dropped F_STANDALONE in favour of F_PROBE
>> >> >    - split probe details across several messages
>> >> >    - probe messages don't automatically imply a standalone daemon
>> >> >    - add wording where probe details interact (F_MQ/F_CONFIG)
>> >> >    - define VMM and make clear QEMU is only one of many potential VMMs
>> >> >    - reword commit message
>> >> >  ---
>> >> >   docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>> >> >   hw/virtio/vhost-user.c      |  8 ++++
>> >> >   2 files changed, 88 insertions(+), 10 deletions(-)
>> >> >
>> >> >  diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> >> >  index 5a070adbc1..ba3b5e07b7 100644
>> >> >  --- a/docs/interop/vhost-user.rst
>> >> >  +++ b/docs/interop/vhost-user.rst
>> >> >  @@ -7,6 +7,7 @@ Vhost-user Protocol
>> >> >   ..
>> >> >     Copyright 2014 Virtual Open Systems Sarl.
>> >> >     Copyright 2019 Intel Corporation
>> >> >  +  Copyright 2023 Linaro Ltd
>> >> >     Licence: This work is licensed under the terms of the GNU GPL,
>> >> >              version 2 or later. See the COPYING file in the top-level
>> >> >              directory.
>> >> >  @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>> >> >   *back-end*. The *front-end* is the application that shares its virtqueues, in
>> >> >   our case QEMU. The *back-end* is the consumer of the virtqueues.
>> >> >
>> >> >  -In the current implementation QEMU is the *front-end*, and the *back-end*
>> >> >  -is the external process consuming the virtio queues, for example a
>> >> >  -software Ethernet switch running in user space, such as Snabbswitch,
>> >> >  -or a block device back-end processing read & write to a virtual
>> >> >  -disk. In order to facilitate interoperability between various back-end
>> >> >  -implementations, it is recommended to follow the :ref:`Backend program
>> >> >  -conventions <backend_conventions>`.
>> >> >  +In the current implementation a Virtual Machine Manager (VMM) such as
>> >> >  +QEMU is the *front-end*, and the *back-end* is the external process
>> >> >  +consuming the virtio queues, for example a software Ethernet switch
>> >> >  +running in user space, such as Snabbswitch, or a block device back-end
>> >> >  +processing read & write to a virtual disk. In order to facilitate
>> >> >  +interoperability between various back-end implementations, it is
>> >> >  +recommended to follow the :ref:`Backend program conventions
>> >> >  +<backend_conventions>`.
>> >> >
>> >> >   The *front-end* and *back-end* can be either a client (i.e. connecting) or
>> >> >   server (listening) in the socket communication.
>> >> >
>> >> >  +Probing device details
>> >> >  +----------------------
>> >> >  +
>> >> >  +Traditionally the vhost-user daemon *back-end* shares configuration
>> >> >  +responsibilities with the VMM *front-end* which needs to know certain
>> >> >  +key bits of information about the device. This means the VMM needs to
>> >> >  +define at least a minimal stub for each VirtIO device it wants to
>> >> >  +support. If the daemon supports the right set of protocol features the
>> >> >  +VMM can probe the daemon for the information it needs to setup the
>> >> >  +device. See :ref:`Probing features for standalone daemons
>> >> >  +<probing_features>` for more details.
>> >> >  +
>> >> >  +
>> >> >   Support for platforms other than Linux
>> >> >   --------------------------------------
>> >> >
>> >> >  @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>> >> >   * ``VHOST_USER_GET_VRING_BASE``
>> >> >   * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>> >> >   * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>> >> >  +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>> >> >
>> >> >   .. seealso::
>> >> >
>> >> >  @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>> >> >   Multiple queue support
>> >> >   ----------------------
>> >> >
>> >> >  -Many devices have a fixed number of virtqueues.  In this case the front-end
>> >> >  -already knows the number of available virtqueues without communicating with the
>> >> >  -back-end.
>> >> >  +Many devices have a fixed number of virtqueues. In this case the
>> >> >  +*front-end* usually already knows the number of available virtqueues
>> >> >  +without communicating with the back-end. For standalone daemons this
>> >> >  +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>> >> >
>> >> >   Some devices do not have a fixed number of virtqueues.  Instead the maximum
>> >> >   number of virtqueues is chosen by the back-end.  The number can depend on host
>> >> >  @@ -885,6 +902,23 @@ Protocol features
>> >> >     #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
>> >> >     #define VHOST_USER_PROTOCOL_F_STATUS               16
>> >> >     #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
>> >> >  +  #define VHOST_USER_PROTOCOL_F_PROBE                18
>> >> >  +
>> >> >  +.. _probing_features:
>> >> >  +
>> >> >  +Probing features for standalone daemons
>> >> >  +---------------------------------------
>> >> >  +
>> >> >  +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>> >> >  +of additional messages which allow the *front-end* to probe details
>> >> >  +about the VirtIO device from the *back-end*. However for a *back-end*
>> >> >  +to be described as standalone it must also support:
>> >> >  +
>> >> >  +  * ``VHOST_USER_PROTOCOL_F_STATUS``
>> >> >  +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>> >> >  +
>> >> >  +which are required to ensure the *back-end* daemon can operate
>> >> >  +without the *front-end* managing some aspects of its configuration.
>> >> >
>> >> >   Front-end message types
>> >> >   -----------------------
>> >> >  @@ -1440,6 +1474,42 @@ Front-end message types
>> >> >     query the back-end for its device status as defined in the Virtio
>> >> >     specification.
>> >> >
>> >> >  +``VHOST_USER_GET_DEVICE_ID``
>> >> >  +  :id: 41
>> >> >  +  :request payload: N/A
>> >> >  +  :reply payload: ``u32``
>> >> >  +
>> >> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> >  +  successfully negotiated, this message is submitted by the front-end
>> >> >  +  to query what VirtIO device the back-end support. This is intended
>> >> >  +  to remove the need for the front-end to know ahead of time what the
>> >> >  +  VirtIO device the backend emulates is.
>> >> >  +
>> >> >  +``VHOST_USER_GET_CONFIG_SIZE``
>> >> >  +  :id: 42
>> >> >  +  :request payload: N/A
>> >> >  +  :reply payload: ``u32``
>> >> >  +
>> >> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> >  +  successfully negotiated, this message is submitted by the front-end
>> >> >  +  to query the size of the VirtIO device's config space. This is
>> >> >  +  intended to remove the need for the front-end to know ahead of time
>> >> >  +  what the size is. Replying with 0 when
>> >> >  +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>> >> >  +  an bug.
>> >> >  +
>> >> >  +``VHOST_USER_GET_MIN_VQ``
>> >> >  +  :id: 43
>> >> >  +  :request payload: N/A
>> >> >  +  :reply payload: ``u32``
>> >> >  +
>> >> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> >  +  successfully negotiated, this message is submitted by the front-end to
>> >> >  +  query minimum number of VQ's required to support the device. A
>> >> >  +  device may support more than this number of VQ's if it advertises
>> >> >  +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>> >> >  +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>> >> >  +  indicate a bug.
>> >> >
>> >> > Maybe I lack some background, but not sure what min_vq is here?
>> >>
>> >> There will be a minimum number of queues you need to support the device.
>> >> For example the virtio-sound spec specifies you need four queues:
>> >> control, event, tx, rx
>> >
>> > I don't understand why the front-end needs to know that? The backend
>> > already reports the number of queues and not all of them need to be
>> > initialized by the driver.
>>
>> But how many don't need to be initialised? We can't just skip:
>>
>>     /* Allocate queues */
>>     vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
>>     for (int i = 0; i < vub->num_vqs; i++) {
>>         g_ptr_array_add(vub->vqs,
>>                         virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
>>     }
>>
>> Or are you saying just require probe-able backends to support
>> VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
>> of queues if it is not a MQ capable device?
>
> The front-end should prepare to allow the maximum number of virtqueues
> returned by VHOST_USER_GET_QUEUE_NUM (VHOST_USER_PROTOCOL_F_MQ).
>
> VIRTIO Transports have a way to query the maximum number of queues but
> not a way to query the minimum number of queues. Why is the minimum
> necessary?

It seems excessive to automatically create the maximum number of VQs. I
guess for backends that don't support the MQ feature (i.e. a variable
number of VQs) we could just say VHOST_USER_GET_QUEUE_NUM == min. But
now we are overloading a different message originally added for
something else.

Stefan Hajnoczi <stefanha@redhat.com> writes:

> On Fri, Sep 01, 2023 at 12:00:18PM +0100, Alex Bennée wrote:
>> Currently QEMU has to know some details about the VirtIO device
>> supported by a vhost-user daemon to be able to setup the guest. This
>> makes it hard for QEMU to add support for additional vhost-user
>> daemons without adding specific stubs for each additional VirtIO
>> device.
>> 
>> This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>> which the back-end can advertise which allows a probe message to be
>> sent to get all the details QEMU needs to know in one message.
>> 
>> Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>> VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>> daemons which are capable of handling all aspects of the VirtIO
>> transactions with only a generic stub on the QEMU side. These daemons
>> can also be used without QEMU in situations where there isn't a full
>> VMM managing their setup.
>> 
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> I think the mindset for this change should be "vhost-user is becoming a
> VIRTIO Transport". VIRTIO Transports have a reasonably well-defined
> feature set in the VIRTIO specification. The goal should be to cover
> every VIRTIO Transport operation via vhost-user protocol messages so
> that the VIRTIO device model can be fully conveyed over vhost-user.

Is it though? The transport is a guest visible construct whereas
vhost-user is purely a backend implementation detail that should be
invisible to the guest.

Also the various backends do things a different set of ways. The
differences between MMIO and PCI are mostly around where config space is
and how IRQs are handled. For CCW we do actually have a set of commands
we can look at:

  #define CCW_CMD_SET_VQ 0x13 
  #define CCW_CMD_VDEV_RESET 0x33 
  #define CCW_CMD_SET_IND 0x43 
  #define CCW_CMD_SET_CONF_IND 0x53 
  #define CCW_CMD_SET_IND_ADAPTER 0x73 
  #define CCW_CMD_READ_FEAT 0x12 
  #define CCW_CMD_WRITE_FEAT 0x11 
  #define CCW_CMD_READ_CONF 0x22 
  #define CCW_CMD_WRITE_CONF 0x21 
  #define CCW_CMD_WRITE_STATUS 0x31 
  #define CCW_CMD_READ_VQ_CONF 0x32 
  #define CCW_CMD_SET_VIRTIO_REV 0x83 
  #define CCW_CMD_READ_STATUS 0x72

which I think we already have mappings for.

> Anything less is yet another ad-hoc protocol extension that will lead to
> more bugs and hacks when it turns out some VIRTIO devices cannot be
> expressed due to limitations in the protocol.

I agree we want to do this right.

> This requires going through the VIRTIO spec to find a correspondence
> between virtio-pci/virtio-mmio/virtio-ccw's interfaces and vhost-user
> protocol messages. In most cases vhost-user already offers messages and
> your patch adds more of what is missing. I think this effort is already
> very close but missing the final check that it really matches the VIRTIO
> spec.
>
> Please do the comparison against the VIRTIO Transports and then adjust
> this patch to make it clear that the back-end is becoming a full-fledged
> VIRTIO Transport:
> - The name of the patch series should reflect that.
> - The vhost-user protocol feature should be named F_TRANSPORT.
> - The messages added in this patch should have a 1:1 correspondence with
>   the VIRTIO spec including using the same terminology for consistency.
>
> Sorry for the hassle, but I think this is a really crucial point where
> we have the chance to make vhost-user work smoothly in the future...but
> only if we can faithfully expose VIRTIO Transport semantics.

I wonder if first be handled by cleaning up the VirtIO spec to make it
clear what capabilities each transport needs to support?

>> ---
>> v2
>>   - dropped F_STANDALONE in favour of F_PROBE
>>   - split probe details across several messages
>>   - probe messages don't automatically imply a standalone daemon
>>   - add wording where probe details interact (F_MQ/F_CONFIG)
>>   - define VMM and make clear QEMU is only one of many potential VMMs
>>   - reword commit message
>> ---
>>  docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>>  hw/virtio/vhost-user.c      |  8 ++++
>>  2 files changed, 88 insertions(+), 10 deletions(-)
>> 
>> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> index 5a070adbc1..ba3b5e07b7 100644
>> --- a/docs/interop/vhost-user.rst
>> +++ b/docs/interop/vhost-user.rst
>> @@ -7,6 +7,7 @@ Vhost-user Protocol
>>  ..
>>    Copyright 2014 Virtual Open Systems Sarl.
>>    Copyright 2019 Intel Corporation
>> +  Copyright 2023 Linaro Ltd
>>    Licence: This work is licensed under the terms of the GNU GPL,
>>             version 2 or later. See the COPYING file in the top-level
>>             directory.
>> @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>>  *back-end*. The *front-end* is the application that shares its virtqueues, in
>>  our case QEMU. The *back-end* is the consumer of the virtqueues.
>>  
>> -In the current implementation QEMU is the *front-end*, and the *back-end*
>> -is the external process consuming the virtio queues, for example a
>> -software Ethernet switch running in user space, such as Snabbswitch,
>> -or a block device back-end processing read & write to a virtual
>> -disk. In order to facilitate interoperability between various back-end
>> -implementations, it is recommended to follow the :ref:`Backend program
>> -conventions <backend_conventions>`.
>> +In the current implementation a Virtual Machine Manager (VMM) such as
>> +QEMU is the *front-end*, and the *back-end* is the external process
>> +consuming the virtio queues, for example a software Ethernet switch
>> +running in user space, such as Snabbswitch, or a block device back-end
>> +processing read & write to a virtual disk. In order to facilitate
>> +interoperability between various back-end implementations, it is
>> +recommended to follow the :ref:`Backend program conventions
>> +<backend_conventions>`.
>>  
>>  The *front-end* and *back-end* can be either a client (i.e. connecting) or
>>  server (listening) in the socket communication.
>>  
>> +Probing device details
>> +----------------------
>> +
>> +Traditionally the vhost-user daemon *back-end* shares configuration
>> +responsibilities with the VMM *front-end* which needs to know certain
>> +key bits of information about the device. This means the VMM needs to
>> +define at least a minimal stub for each VirtIO device it wants to
>> +support. If the daemon supports the right set of protocol features the
>> +VMM can probe the daemon for the information it needs to setup the
>> +device.
>
> "... without a per-device stub in the VMM"
>
> This makes it clear that this sentence is describing an alternative
> to the per-device stub in the VMM.
>
>> See :ref:`Probing features for standalone daemons
>> +<probing_features>` for more details.
>
> The current section is named "Probing device details" and one being
> reference is called "Probing features for standalone daemons". Are
> "features" or "device details" two terms for the same thing? Why
> "daemons" and not "back-end"?
>
> I suggest calling this section "Standalone back-ends" and the other
> section "Probing standalone back-ends" to keep the terminology
> consistent.
>
>> +
>> +
>>  Support for platforms other than Linux
>>  --------------------------------------
>>  
>> @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>>  * ``VHOST_USER_GET_VRING_BASE``
>>  * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>>  * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>> +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>
> F_STANDALONE vs F_PROBE
>
> "SPECS" vs "features" vs "details".
>
> Please be consistent.
>
>>  
>>  .. seealso::
>>  
>> @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>>  Multiple queue support
>>  ----------------------
>>  
>> -Many devices have a fixed number of virtqueues.  In this case the front-end
>> -already knows the number of available virtqueues without communicating with the
>> -back-end.
>> +Many devices have a fixed number of virtqueues. In this case the
>> +*front-end* usually already knows the number of available virtqueues
>> +without communicating with the back-end. For standalone daemons this
>
> "Usually" is vague. It's possible to be precise:
>
>   In this case a front-end that is aware of the device type already
>                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   knows the number of available virtqueues without communicating with
>   the back-end.
>
>> +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>
> Then this sentence can be adjusted to:
>
>   When the front-end is not aware of the device type, the number can be
>   probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>
>>  
>>  Some devices do not have a fixed number of virtqueues.  Instead the maximum
>>  number of virtqueues is chosen by the back-end.  The number can depend on host
>> @@ -885,6 +902,23 @@ Protocol features
>>    #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
>>    #define VHOST_USER_PROTOCOL_F_STATUS               16
>>    #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
>> +  #define VHOST_USER_PROTOCOL_F_PROBE                18
>> +
>> +.. _probing_features:
>> +
>> +Probing features for standalone daemons
>> +---------------------------------------
>> +
>> +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>> +of additional messages which allow the *front-end* to probe details
>> +about the VirtIO device from the *back-end*. However for a *back-end*
>> +to be described as standalone it must also support:
>> +
>> +  * ``VHOST_USER_PROTOCOL_F_STATUS``
>> +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>> +
>> +which are required to ensure the *back-end* daemon can operate
>> +without the *front-end* managing some aspects of its configuration.
>>  
>>  Front-end message types
>>  -----------------------
>> @@ -1440,6 +1474,42 @@ Front-end message types
>>    query the back-end for its device status as defined in the Virtio
>>    specification.
>>  
>> +``VHOST_USER_GET_DEVICE_ID``
>> +  :id: 41
>> +  :request payload: N/A
>> +  :reply payload: ``u32``
>> +
>> +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> +  successfully negotiated, this message is submitted by the front-end
>> +  to query what VirtIO device the back-end support. This is intended
>> +  to remove the need for the front-end to know ahead of time what the
>> +  VirtIO device the backend emulates is.
>
> "... VIRTIO device type that the backend emulates is."
>
> "Device type" is the name used in the VIRTIO spec.
>
>> +
>> +``VHOST_USER_GET_CONFIG_SIZE``
>> +  :id: 42
>> +  :request payload: N/A
>> +  :reply payload: ``u32``
>> +
>> +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> +  successfully negotiated, this message is submitted by the front-end
>> +  to query the size of the VirtIO device's config space. This is
>> +  intended to remove the need for the front-end to know ahead of time
>> +  what the size is. Replying with 0 when
>> +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>> +  an bug.
>
> "a bug"
>
> What is the harm in returning 0 when the device has an empty
> Configuration Space like the Entropy device, the I2C Adapter, the SCMI
> device, etc?
>
>> +
>> +``VHOST_USER_GET_MIN_VQ``
>> +  :id: 43
>> +  :request payload: N/A
>> +  :reply payload: ``u32``
>> +
>> +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> +  successfully negotiated, this message is submitted by the front-end to
>> +  query minimum number of VQ's required to support the device. A
>> +  device may support more than this number of VQ's if it advertises
>> +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>> +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>> +  indicate a bug.
>
> What is the purpose of this message? I don't see an equivalent in the
> VIRTIO specification.
>
>>  
>>  Back-end message types
>>  ----------------------
>> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
>> index 8dcf049d42..4d433cdf2b 100644
>> --- a/hw/virtio/vhost-user.c
>> +++ b/hw/virtio/vhost-user.c
>> @@ -202,6 +202,13 @@ typedef struct VhostUserInflight {
>>      uint16_t queue_size;
>>  } VhostUserInflight;
>>  
>> +typedef struct VhostUserBackendSpecs {
>> +    uint32_t device_id;
>> +    uint32_t config_size;
>> +    uint32_t min_vqs;
>> +    uint32_t max_vqs;
>> +} VhostUserBackendSpecs;
>
> This message is undocumented? I think it may be outdated and you split
> it up into individual messages.
>
>> +
>>  typedef struct {
>>      VhostUserRequest request;
>>  
>> @@ -226,6 +233,7 @@ typedef union {
>>          VhostUserCryptoSession session;
>>          VhostUserVringArea area;
>>          VhostUserInflight inflight;
>> +        VhostUserBackendSpecs specs;
>>  } VhostUserPayload;
>>  
>>  typedef struct VhostUserMsg {
>> -- 
>> 2.39.2
>>

A QEMU built-in VIRTIO device will also call virtio_add_queue() for
the maximum number of virtqueues.

I'm not sure what the concern is about adding as few virtqueues as possible?

If the front-end's implementation is inefficient, then it should be
optimized so that untouched virtqueues don't consume resources. I
don't see the need to try to add a special message to vhost-user to
try to reduce the number of virtqueues.

Stefan

On Fri, 8 Sept 2023 at 08:03, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Stefan Hajnoczi <stefanha@gmail.com> writes:
>
> > On Fri, 8 Sept 2023 at 02:43, Alex Bennée <alex.bennee@linaro.org> wrote:
> >>
> >>
> >> Stefan Hajnoczi <stefanha@redhat.com> writes:
> >>
> >> > On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
> >> >>
> >> >> Albert Esteve <aesteve@redhat.com> writes:
> >> >>
> >> >> > This looks great! Thanks for this proposal.
> >> >> >
> >> >> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >> >> >
> >> >> >  Currently QEMU has to know some details about the VirtIO device
> >> >> >  supported by a vhost-user daemon to be able to setup the guest. This
> >> >> >  makes it hard for QEMU to add support for additional vhost-user
> >> >> >  daemons without adding specific stubs for each additional VirtIO
> >> >> >  device.
> >> >> >
> >> >> >  This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> >> >> >  which the back-end can advertise which allows a probe message to be
> >> >> >  sent to get all the details QEMU needs to know in one message.
> >> >> >
> >> >> >  Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> >> >> >  VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> >> >> >  daemons which are capable of handling all aspects of the VirtIO
> >> >> >  transactions with only a generic stub on the QEMU side. These daemons
> >> >> >  can also be used without QEMU in situations where there isn't a full
> >> >> >  VMM managing their setup.
> >> >> >
> >> >> >  Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >> >> >
> >> >> >  ---
> >> >> >  v2
> >> >> >    - dropped F_STANDALONE in favour of F_PROBE
> >> >> >    - split probe details across several messages
> >> >> >    - probe messages don't automatically imply a standalone daemon
> >> >> >    - add wording where probe details interact (F_MQ/F_CONFIG)
> >> >> >    - define VMM and make clear QEMU is only one of many potential VMMs
> >> >> >    - reword commit message
> >> >> >  ---
> >> >> >   docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> >> >> >   hw/virtio/vhost-user.c      |  8 ++++
> >> >> >   2 files changed, 88 insertions(+), 10 deletions(-)
> >> >> >
> >> >> >  diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> >> >> >  index 5a070adbc1..ba3b5e07b7 100644
> >> >> >  --- a/docs/interop/vhost-user.rst
> >> >> >  +++ b/docs/interop/vhost-user.rst
> >> >> >  @@ -7,6 +7,7 @@ Vhost-user Protocol
> >> >> >   ..
> >> >> >     Copyright 2014 Virtual Open Systems Sarl.
> >> >> >     Copyright 2019 Intel Corporation
> >> >> >  +  Copyright 2023 Linaro Ltd
> >> >> >     Licence: This work is licensed under the terms of the GNU GPL,
> >> >> >              version 2 or later. See the COPYING file in the top-level
> >> >> >              directory.
> >> >> >  @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
> >> >> >   *back-end*. The *front-end* is the application that shares its virtqueues, in
> >> >> >   our case QEMU. The *back-end* is the consumer of the virtqueues.
> >> >> >
> >> >> >  -In the current implementation QEMU is the *front-end*, and the *back-end*
> >> >> >  -is the external process consuming the virtio queues, for example a
> >> >> >  -software Ethernet switch running in user space, such as Snabbswitch,
> >> >> >  -or a block device back-end processing read & write to a virtual
> >> >> >  -disk. In order to facilitate interoperability between various back-end
> >> >> >  -implementations, it is recommended to follow the :ref:`Backend program
> >> >> >  -conventions <backend_conventions>`.
> >> >> >  +In the current implementation a Virtual Machine Manager (VMM) such as
> >> >> >  +QEMU is the *front-end*, and the *back-end* is the external process
> >> >> >  +consuming the virtio queues, for example a software Ethernet switch
> >> >> >  +running in user space, such as Snabbswitch, or a block device back-end
> >> >> >  +processing read & write to a virtual disk. In order to facilitate
> >> >> >  +interoperability between various back-end implementations, it is
> >> >> >  +recommended to follow the :ref:`Backend program conventions
> >> >> >  +<backend_conventions>`.
> >> >> >
> >> >> >   The *front-end* and *back-end* can be either a client (i.e. connecting) or
> >> >> >   server (listening) in the socket communication.
> >> >> >
> >> >> >  +Probing device details
> >> >> >  +----------------------
> >> >> >  +
> >> >> >  +Traditionally the vhost-user daemon *back-end* shares configuration
> >> >> >  +responsibilities with the VMM *front-end* which needs to know certain
> >> >> >  +key bits of information about the device. This means the VMM needs to
> >> >> >  +define at least a minimal stub for each VirtIO device it wants to
> >> >> >  +support. If the daemon supports the right set of protocol features the
> >> >> >  +VMM can probe the daemon for the information it needs to setup the
> >> >> >  +device. See :ref:`Probing features for standalone daemons
> >> >> >  +<probing_features>` for more details.
> >> >> >  +
> >> >> >  +
> >> >> >   Support for platforms other than Linux
> >> >> >   --------------------------------------
> >> >> >
> >> >> >  @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> >> >> >   * ``VHOST_USER_GET_VRING_BASE``
> >> >> >   * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> >> >> >   * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> >> >> >  +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >> >> >
> >> >> >   .. seealso::
> >> >> >
> >> >> >  @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
> >> >> >   Multiple queue support
> >> >> >   ----------------------
> >> >> >
> >> >> >  -Many devices have a fixed number of virtqueues.  In this case the front-end
> >> >> >  -already knows the number of available virtqueues without communicating with the
> >> >> >  -back-end.
> >> >> >  +Many devices have a fixed number of virtqueues. In this case the
> >> >> >  +*front-end* usually already knows the number of available virtqueues
> >> >> >  +without communicating with the back-end. For standalone daemons this
> >> >> >  +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
> >> >> >
> >> >> >   Some devices do not have a fixed number of virtqueues.  Instead the maximum
> >> >> >   number of virtqueues is chosen by the back-end.  The number can depend on host
> >> >> >  @@ -885,6 +902,23 @@ Protocol features
> >> >> >     #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS  15
> >> >> >     #define VHOST_USER_PROTOCOL_F_STATUS               16
> >> >> >     #define VHOST_USER_PROTOCOL_F_XEN_MMAP             17
> >> >> >  +  #define VHOST_USER_PROTOCOL_F_PROBE                18
> >> >> >  +
> >> >> >  +.. _probing_features:
> >> >> >  +
> >> >> >  +Probing features for standalone daemons
> >> >> >  +---------------------------------------
> >> >> >  +
> >> >> >  +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> >> >> >  +of additional messages which allow the *front-end* to probe details
> >> >> >  +about the VirtIO device from the *back-end*. However for a *back-end*
> >> >> >  +to be described as standalone it must also support:
> >> >> >  +
> >> >> >  +  * ``VHOST_USER_PROTOCOL_F_STATUS``
> >> >> >  +  * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> >> >> >  +
> >> >> >  +which are required to ensure the *back-end* daemon can operate
> >> >> >  +without the *front-end* managing some aspects of its configuration.
> >> >> >
> >> >> >   Front-end message types
> >> >> >   -----------------------
> >> >> >  @@ -1440,6 +1474,42 @@ Front-end message types
> >> >> >     query the back-end for its device status as defined in the Virtio
> >> >> >     specification.
> >> >> >
> >> >> >  +``VHOST_USER_GET_DEVICE_ID``
> >> >> >  +  :id: 41
> >> >> >  +  :request payload: N/A
> >> >> >  +  :reply payload: ``u32``
> >> >> >  +
> >> >> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >> >  +  successfully negotiated, this message is submitted by the front-end
> >> >> >  +  to query what VirtIO device the back-end support. This is intended
> >> >> >  +  to remove the need for the front-end to know ahead of time what the
> >> >> >  +  VirtIO device the backend emulates is.
> >> >> >  +
> >> >> >  +``VHOST_USER_GET_CONFIG_SIZE``
> >> >> >  +  :id: 42
> >> >> >  +  :request payload: N/A
> >> >> >  +  :reply payload: ``u32``
> >> >> >  +
> >> >> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >> >  +  successfully negotiated, this message is submitted by the front-end
> >> >> >  +  to query the size of the VirtIO device's config space. This is
> >> >> >  +  intended to remove the need for the front-end to know ahead of time
> >> >> >  +  what the size is. Replying with 0 when
> >> >> >  +  ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> >> >> >  +  an bug.
> >> >> >  +
> >> >> >  +``VHOST_USER_GET_MIN_VQ``
> >> >> >  +  :id: 43
> >> >> >  +  :request payload: N/A
> >> >> >  +  :reply payload: ``u32``
> >> >> >  +
> >> >> >  +  When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >> >  +  successfully negotiated, this message is submitted by the front-end to
> >> >> >  +  query minimum number of VQ's required to support the device. A
> >> >> >  +  device may support more than this number of VQ's if it advertises
> >> >> >  +  the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> >> >> >  +  number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> >> >> >  +  indicate a bug.
> >> >> >
> >> >> > Maybe I lack some background, but not sure what min_vq is here?
> >> >>
> >> >> There will be a minimum number of queues you need to support the device.
> >> >> For example the virtio-sound spec specifies you need four queues:
> >> >> control, event, tx, rx
> >> >
> >> > I don't understand why the front-end needs to know that? The backend
> >> > already reports the number of queues and not all of them need to be
> >> > initialized by the driver.
> >>
> >> But how many don't need to be initialised? We can't just skip:
> >>
> >>     /* Allocate queues */
> >>     vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
> >>     for (int i = 0; i < vub->num_vqs; i++) {
> >>         g_ptr_array_add(vub->vqs,
> >>                         virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
> >>     }
> >>
> >> Or are you saying just require probe-able backends to support
> >> VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
> >> of queues if it is not a MQ capable device?
> >
> > The front-end should prepare to allow the maximum number of virtqueues
> > returned by VHOST_USER_GET_QUEUE_NUM (VHOST_USER_PROTOCOL_F_MQ).
> >
> > VIRTIO Transports have a way to query the maximum number of queues but
> > not a way to query the minimum number of queues. Why is the minimum
> > necessary?
>
> It seems excessive to automatically create the maximum number of VQs. I
> guess for backends that don't support the MQ feature (i.e. a variable
> number of VQs) we could just say VHOST_USER_GET_QUEUE_NUM == min. But
> now we are overloading a different message originally added for
> something else.
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro

On Fri, Sep 08, 2023 at 01:03:26PM +0100, Alex Bennée wrote:
> 
> Stefan Hajnoczi <stefanha@redhat.com> writes:
> 
> > On Fri, Sep 01, 2023 at 12:00:18PM +0100, Alex Bennée wrote:
> >> Currently QEMU has to know some details about the VirtIO device
> >> supported by a vhost-user daemon to be able to setup the guest. This
> >> makes it hard for QEMU to add support for additional vhost-user
> >> daemons without adding specific stubs for each additional VirtIO
> >> device.
> >> 
> >> This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> >> which the back-end can advertise which allows a probe message to be
> >> sent to get all the details QEMU needs to know in one message.
> >> 
> >> Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> >> VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> >> daemons which are capable of handling all aspects of the VirtIO
> >> transactions with only a generic stub on the QEMU side. These daemons
> >> can also be used without QEMU in situations where there isn't a full
> >> VMM managing their setup.
> >> 
> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >
> > I think the mindset for this change should be "vhost-user is becoming a
> > VIRTIO Transport". VIRTIO Transports have a reasonably well-defined
> > feature set in the VIRTIO specification. The goal should be to cover
> > every VIRTIO Transport operation via vhost-user protocol messages so
> > that the VIRTIO device model can be fully conveyed over vhost-user.
> 
> Is it though? The transport is a guest visible construct whereas
> vhost-user is purely a backend implementation detail that should be
> invisible to the guest.

No, the transport is not necessarily guest-visible. The vhost-user model
is that the front-end emulates a VIRTIO device and some aspects of that
device are delegated to the vhost-user back-end.

In other words, the vhost-user device is not the same as the VIRTIO
device that the guest sees, but it's still important for the vhost-user
back-end to be a VIRTIO Transport because that's how we can be sure it
supports the VIRTIO device model properly.

> 
> Also the various backends do things a different set of ways. The
> differences between MMIO and PCI are mostly around where config space is
> and how IRQs are handled. For CCW we do actually have a set of commands
> we can look at:
> 
>   #define CCW_CMD_SET_VQ 0x13 
>   #define CCW_CMD_VDEV_RESET 0x33 
>   #define CCW_CMD_SET_IND 0x43 
>   #define CCW_CMD_SET_CONF_IND 0x53 
>   #define CCW_CMD_SET_IND_ADAPTER 0x73 
>   #define CCW_CMD_READ_FEAT 0x12 
>   #define CCW_CMD_WRITE_FEAT 0x11 
>   #define CCW_CMD_READ_CONF 0x22 
>   #define CCW_CMD_WRITE_CONF 0x21 
>   #define CCW_CMD_WRITE_STATUS 0x31 
>   #define CCW_CMD_READ_VQ_CONF 0x32 
>   #define CCW_CMD_SET_VIRTIO_REV 0x83 
>   #define CCW_CMD_READ_STATUS 0x72
> 
> which I think we already have mappings for.

Yes, there are differences between the transports. vhost-user uses
eventfds (callfd/kickfd) instead of interrupts.

> > Anything less is yet another ad-hoc protocol extension that will lead to
> > more bugs and hacks when it turns out some VIRTIO devices cannot be
> > expressed due to limitations in the protocol.
> 
> I agree we want to do this right.
> 
> > This requires going through the VIRTIO spec to find a correspondence
> > between virtio-pci/virtio-mmio/virtio-ccw's interfaces and vhost-user
> > protocol messages. In most cases vhost-user already offers messages and
> > your patch adds more of what is missing. I think this effort is already
> > very close but missing the final check that it really matches the VIRTIO
> > spec.
> >
> > Please do the comparison against the VIRTIO Transports and then adjust
> > this patch to make it clear that the back-end is becoming a full-fledged
> > VIRTIO Transport:
> > - The name of the patch series should reflect that.
> > - The vhost-user protocol feature should be named F_TRANSPORT.
> > - The messages added in this patch should have a 1:1 correspondence with
> >   the VIRTIO spec including using the same terminology for consistency.
> >
> > Sorry for the hassle, but I think this is a really crucial point where
> > we have the chance to make vhost-user work smoothly in the future...but
> > only if we can faithfully expose VIRTIO Transport semantics.
> 
> I wonder if first be handled by cleaning up the VirtIO spec to make it
> clear what capabilities each transport needs to support?

It's a fair point that the VIRTIO spec does not provide an interface
definition for the VIRTIO Transport or at least a definitive list of
requirements. The requirements are implicit (i.e. it is assumed that
very transport provides a way to set the virtqueue descriptor table
addresses) so it's necessary to review the existing transports to
understand their functionality.

If you want to create a list of the requirements for a VIRTIO Transport
and propose a patch to the VIRTIO spec then that would be great, but I
don't think that stops this patch series. It's possible to review
virtio-pci/virtio-mmio/virtio-ccw and check that there is equivalent
functionality in the vhost-user protocol.

Stefan