| Message ID | 20240416193319.778192-11-sakari.ailus@linux.intel.com |
|---|---|
| State | New |
| Headers | show |
| Series | Generic line based metadata support, internal pads | expand |
Hi Laurent, Thanks for the review! On Fri, Apr 19, 2024 at 09:49:26PM +0300, Laurent Pinchart wrote: > Hi Sakari, > > Thank you for the patch. > > On Tue, Apr 16, 2024 at 10:32:43PM +0300, Sakari Ailus wrote: > > Document internal sink pads, pads that have both SINK and INTERNAL flags > > set. Use the IMX219 camera sensor as an example. > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com> > > Reviewed-by Julien Massot <julien.massot@collabora.com> > > --- > > .../userspace-api/media/v4l/dev-subdev.rst | 145 ++++++++++++++++++ > > 1 file changed, 145 insertions(+) > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst > > index b76e02e54512..d30dcb9e2537 100644 > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst > > @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the > > sub-device and a (pad, stream) pair. For sub-devices that do not support > > multiplexed streams the 'stream' field is always 0. > > > > +.. _v4l2-subdev-internal-source-pads: > > + > > +Internal sink pads and routing > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > + > > +Cases where a single sub-device source pad is traversed by multiple streams, one > > +or more of which originate from within the sub-device itself, are special as > > +there is no external sink pad for such routes. In those cases, the sources of > > +the internally generated streams are represented by internal sink pads, which > > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>` > > +pad flag set. > > + > > +Internal pads have all the properties of an external pad, including formats and > > +selections. The format in this case is the source format of the stream. An > > +internal pad always has a single stream only (0). > > + > > +Routes from an internal sink pad to an external source pad are typically not > > +modifiable but they can be activated and deactivated using the > > +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending > > +on driver capabilities. > > + > > Interaction between routes, streams, formats and selections > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > > @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps: > > the configurations along the stream towards the receiver, using > > :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each > > stream endpoint in each sub-device. > > + > > +Internal pads setup example > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > + > > +A simple example of a multiplexed stream setup might be as follows: > > + > > +- An IMX219 camera sensor source sub-device, with one source pad (0), one > > + internal sink pad (1) as the source of the image data and an internal sink > > + pad (2) as the source of the embedded data. There are two routes, one from the > > + internal sink pad 1 to the source 0 (image data) and another from the internal > > s/source 0/source pad 0/ Yes. > > > + sink pad 2 to the source pad 0 (embedded data). Both streams are always > > + active, i.e. there is no need to separately enable the embedded data > > + stream. The sensor uses the CSI-2 interface. > > + > > +- A CSI-2 receiver in the SoC. The receiver has a single sink pad (pad 0), > > + connected to the sensor, and two source pads (pads 1 and 2), to the DMA > > + engine. The receiver demultiplexes the incoming streams to the source pads. > > s/engine/engines/ > > maybe better, "to two DMA engines". "to the two DMA engines". > > > + > > +- DMA engines in the SoC, one for each stream. Each DMA engine is connected to a > > + single source pad of the receiver. > > + > > +The sensor and the receiver are modelled as V4L2 sub-devices, exposed to > > +userspace via /dev/v4l-subdevX device nodes. The DMA engines are modelled as > > +V4L2 devices, exposed to userspace via /dev/videoX nodes. > > + > > +To configure this pipeline, the userspace must take the following steps: > > + > > +1) Set up media links between entities: enable the links from the sensor to the > > + receiver and from the receiver to the DMA engines. This step does not differ > > + from normal non-multiplexed media controller setup. > > + > > +2) Configure routing > > + > > +.. flat-table:: Camera sensor. There are no configurable routes. > > + :header-rows: 1 > > + > > + * - Sink Pad/Stream > > + - Source Pad/Stream > > + - Routing Flags > > + - Comments > > + * - 1/0 > > + - 0/0 > > + - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > + - Pixel data stream from the sink pad > > "from the internal image sink pad" > > > + * - 2/0 > > + - 0/1 > > + - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > + - Metadata stream from the internal sink pad > > "internal embedded data sink pad" Yes... > > > + > > +.. flat-table:: Receiver routing table. Typically both routes need to be > > + explicitly set. > > + :header-rows: 1 > > + > > + * - Sink Pad/Stream > > + - Source Pad/Stream > > + - Routing Flags > > + - Comments > > + * - 0/0 > > + - 1/0 > > + - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > + - Pixel data stream from camera sensor > > + * - 0/1 > > + - 2/0 > > + - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > + - Metadata stream from camera sensor > > + > > +The options available in sensor's routing configuration are dictated by hardware > > s/in sensor/in the sensor/ Agreed. > > > +capabilities: typically camera sensors always produce an image data stream while > > +it may be possible to enable and disable the embedded data stream. > > Should this go after the first table ? I'll move it there. > > > + > > +3) Configure formats and selections > > + > > + This example assumes that the formats are propagated from sink pad to the > > + source pad as-is. The tables contain fields of both struct v4l2_subdev_format > > + and struct v4l2_mbus_framefmt. > > + > > +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are > > + static or propagated. The order is aligned with configured > > + routes. > > + :header-rows: 1 > > + :fill-cells: > > + > > + * - Sub-device > > + - Pad/Stream > > + - Width > > + - Height > > + - Code > > + * - :rspan:`3` IMX219 > > + - 1/0 > > + - 3296 > > + - 2480 > > + - MEDIA_BUS_FMT_SRGGB10 > > + * - 0/0 > > + - **3296** > > + - **2480** > > + - **MEDIA_BUS_FMT_SRGGB10** > > + * - 2/0 > > + - 3296 > > + - 2 > > + - MEDIA_BUS_FMT_IMX219_EMBEDDED > > We need a patch in this series to define this format. I'm waiting for your imx290 patches. :-) > > > + * - 1/1 > > I think this should be 0/1. Good catch. > > > + - 3296 > > + - 2 > > + - MEDIA_BUS_FMT_META_10 > > + * - :rspan:`3` CSI-2 receiver > > + - 0/0 > > + - **3296** > > + - **2480** > > + - **MEDIA_BUS_FMT_SRGGB10** > > + * - 1/0 > > + - 3296 > > + - 2480 > > + - MEDIA_BUS_FMT_SRGGB10 > > + * - 0/1 > > + - **3296** > > + - **2** > > + - **MEDIA_BUS_FMT_META_10** > > + * - 2/0 > > + - 3296 > > + - 2 > > + - MEDIA_BUS_FMT_META_10 > > + > > +The embedded data format does not need to be configured as the format is > > Do you mean the "format on the sensor's pads" ? It's a bit confusing if > you don't specify it. That's what was meant, I'll add that. > > > +dictated by the pixel data format in this case. >
Hi Sakari, On Tue, Apr 23, 2024 at 10:27:11AM +0000, Sakari Ailus wrote: > On Fri, Apr 19, 2024 at 09:49:26PM +0300, Laurent Pinchart wrote: > > On Tue, Apr 16, 2024 at 10:32:43PM +0300, Sakari Ailus wrote: > > > Document internal sink pads, pads that have both SINK and INTERNAL flags > > > set. Use the IMX219 camera sensor as an example. > > > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com> > > > Reviewed-by Julien Massot <julien.massot@collabora.com> > > > --- > > > .../userspace-api/media/v4l/dev-subdev.rst | 145 ++++++++++++++++++ > > > 1 file changed, 145 insertions(+) > > > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst > > > index b76e02e54512..d30dcb9e2537 100644 > > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst > > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst > > > @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the > > > sub-device and a (pad, stream) pair. For sub-devices that do not support > > > multiplexed streams the 'stream' field is always 0. > > > > > > +.. _v4l2-subdev-internal-source-pads: > > > + > > > +Internal sink pads and routing > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > + > > > +Cases where a single sub-device source pad is traversed by multiple streams, one > > > +or more of which originate from within the sub-device itself, are special as > > > +there is no external sink pad for such routes. In those cases, the sources of > > > +the internally generated streams are represented by internal sink pads, which > > > +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>` > > > +pad flag set. > > > + > > > +Internal pads have all the properties of an external pad, including formats and > > > +selections. The format in this case is the source format of the stream. An > > > +internal pad always has a single stream only (0). > > > + > > > +Routes from an internal sink pad to an external source pad are typically not > > > +modifiable but they can be activated and deactivated using the > > > +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending > > > +on driver capabilities. > > > + > > > Interaction between routes, streams, formats and selections > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > > > > @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps: > > > the configurations along the stream towards the receiver, using > > > :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each > > > stream endpoint in each sub-device. > > > + > > > +Internal pads setup example > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > + > > > +A simple example of a multiplexed stream setup might be as follows: > > > + > > > +- An IMX219 camera sensor source sub-device, with one source pad (0), one > > > + internal sink pad (1) as the source of the image data and an internal sink > > > + pad (2) as the source of the embedded data. There are two routes, one from the > > > + internal sink pad 1 to the source 0 (image data) and another from the internal > > > > s/source 0/source pad 0/ > > Yes. > > > > + sink pad 2 to the source pad 0 (embedded data). Both streams are always > > > + active, i.e. there is no need to separately enable the embedded data > > > + stream. The sensor uses the CSI-2 interface. > > > + > > > +- A CSI-2 receiver in the SoC. The receiver has a single sink pad (pad 0), > > > + connected to the sensor, and two source pads (pads 1 and 2), to the DMA > > > + engine. The receiver demultiplexes the incoming streams to the source pads. > > > > s/engine/engines/ > > > > maybe better, "to two DMA engines". > > "to the two DMA engines". > > > > + > > > +- DMA engines in the SoC, one for each stream. Each DMA engine is connected to a > > > + single source pad of the receiver. > > > + > > > +The sensor and the receiver are modelled as V4L2 sub-devices, exposed to > > > +userspace via /dev/v4l-subdevX device nodes. The DMA engines are modelled as > > > +V4L2 devices, exposed to userspace via /dev/videoX nodes. > > > + > > > +To configure this pipeline, the userspace must take the following steps: > > > + > > > +1) Set up media links between entities: enable the links from the sensor to the > > > + receiver and from the receiver to the DMA engines. This step does not differ > > > + from normal non-multiplexed media controller setup. > > > + > > > +2) Configure routing > > > + > > > +.. flat-table:: Camera sensor. There are no configurable routes. > > > + :header-rows: 1 > > > + > > > + * - Sink Pad/Stream > > > + - Source Pad/Stream > > > + - Routing Flags > > > + - Comments > > > + * - 1/0 > > > + - 0/0 > > > + - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > > + - Pixel data stream from the sink pad > > > > "from the internal image sink pad" > > > > > + * - 2/0 > > > + - 0/1 > > > + - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > > + - Metadata stream from the internal sink pad > > > > "internal embedded data sink pad" > > Yes... > > > > + > > > +.. flat-table:: Receiver routing table. Typically both routes need to be > > > + explicitly set. > > > + :header-rows: 1 > > > + > > > + * - Sink Pad/Stream > > > + - Source Pad/Stream > > > + - Routing Flags > > > + - Comments > > > + * - 0/0 > > > + - 1/0 > > > + - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > > + - Pixel data stream from camera sensor > > > + * - 0/1 > > > + - 2/0 > > > + - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > > + - Metadata stream from camera sensor > > > + > > > +The options available in sensor's routing configuration are dictated by hardware > > > > s/in sensor/in the sensor/ > > Agreed. > > > > +capabilities: typically camera sensors always produce an image data stream while > > > +it may be possible to enable and disable the embedded data stream. > > > > Should this go after the first table ? > > I'll move it there. > > > > + > > > +3) Configure formats and selections > > > + > > > + This example assumes that the formats are propagated from sink pad to the > > > + source pad as-is. The tables contain fields of both struct v4l2_subdev_format > > > + and struct v4l2_mbus_framefmt. > > > + > > > +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are > > > + static or propagated. The order is aligned with configured > > > + routes. > > > + :header-rows: 1 > > > + :fill-cells: > > > + > > > + * - Sub-device > > > + - Pad/Stream > > > + - Width > > > + - Height > > > + - Code > > > + * - :rspan:`3` IMX219 > > > + - 1/0 > > > + - 3296 > > > + - 2480 > > > + - MEDIA_BUS_FMT_SRGGB10 > > > + * - 0/0 > > > + - **3296** > > > + - **2480** > > > + - **MEDIA_BUS_FMT_SRGGB10** > > > + * - 2/0 > > > + - 3296 > > > + - 2 > > > + - MEDIA_BUS_FMT_IMX219_EMBEDDED > > > > We need a patch in this series to define this format. > > I'm waiting for your imx290 patches. :-) I assume you mean imx219 :-) They will come, but we need to discuss usage of internal pads for raw sensors first. That won't make it in time for v6.10, I hope we'll be in time for v6.11. > > > + * - 1/1 > > > > I think this should be 0/1. > > Good catch. > > > > + - 3296 > > > + - 2 > > > + - MEDIA_BUS_FMT_META_10 > > > + * - :rspan:`3` CSI-2 receiver > > > + - 0/0 > > > + - **3296** > > > + - **2480** > > > + - **MEDIA_BUS_FMT_SRGGB10** > > > + * - 1/0 > > > + - 3296 > > > + - 2480 > > > + - MEDIA_BUS_FMT_SRGGB10 > > > + * - 0/1 > > > + - **3296** > > > + - **2** > > > + - **MEDIA_BUS_FMT_META_10** > > > + * - 2/0 > > > + - 3296 > > > + - 2 > > > + - MEDIA_BUS_FMT_META_10 > > > + > > > +The embedded data format does not need to be configured as the format is > > > > Do you mean the "format on the sensor's pads" ? It's a bit confusing if > > you don't specify it. > > That's what was meant, I'll add that. > > > > +dictated by the pixel data format in this case.
Hi Laurent, On Tue, Apr 23, 2024 at 03:56:18PM +0300, Laurent Pinchart wrote: > > > > + - MEDIA_BUS_FMT_IMX219_EMBEDDED > > > > > > We need a patch in this series to define this format. > > > > I'm waiting for your imx290 patches. :-) > > I assume you mean imx219 :-) They will come, but we need to discuss > usage of internal pads for raw sensors first. That won't make it in time > for v6.10, I hope we'll be in time for v6.11. Sure. This patch should actually be located later in the set, after the INTERNAL pad flag patch. I'll move it there for v10.
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst index b76e02e54512..d30dcb9e2537 100644 --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst @@ -553,6 +553,27 @@ A stream at a specific point in the media pipeline is identified by the sub-device and a (pad, stream) pair. For sub-devices that do not support multiplexed streams the 'stream' field is always 0. +.. _v4l2-subdev-internal-source-pads: + +Internal sink pads and routing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Cases where a single sub-device source pad is traversed by multiple streams, one +or more of which originate from within the sub-device itself, are special as +there is no external sink pad for such routes. In those cases, the sources of +the internally generated streams are represented by internal sink pads, which +are sink pads that have the :ref:`MEDIA_PAD_FL_INTERNAL <MEDIA-PAD-FL-INTERNAL>` +pad flag set. + +Internal pads have all the properties of an external pad, including formats and +selections. The format in this case is the source format of the stream. An +internal pad always has a single stream only (0). + +Routes from an internal sink pad to an external source pad are typically not +modifiable but they can be activated and deactivated using the +:ref:`V4L2_SUBDEV_ROUTE_FL_ACTIVE <v4l2-subdev-routing-flags>` flag, depending +on driver capabilities. + Interaction between routes, streams, formats and selections ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -668,3 +689,127 @@ To configure this pipeline, the userspace must take the following steps: the configurations along the stream towards the receiver, using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each stream endpoint in each sub-device. + +Internal pads setup example +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A simple example of a multiplexed stream setup might be as follows: + +- An IMX219 camera sensor source sub-device, with one source pad (0), one + internal sink pad (1) as the source of the image data and an internal sink + pad (2) as the source of the embedded data. There are two routes, one from the + internal sink pad 1 to the source 0 (image data) and another from the internal + sink pad 2 to the source pad 0 (embedded data). Both streams are always + active, i.e. there is no need to separately enable the embedded data + stream. The sensor uses the CSI-2 interface. + +- A CSI-2 receiver in the SoC. The receiver has a single sink pad (pad 0), + connected to the sensor, and two source pads (pads 1 and 2), to the DMA + engine. The receiver demultiplexes the incoming streams to the source pads. + +- DMA engines in the SoC, one for each stream. Each DMA engine is connected to a + single source pad of the receiver. + +The sensor and the receiver are modelled as V4L2 sub-devices, exposed to +userspace via /dev/v4l-subdevX device nodes. The DMA engines are modelled as +V4L2 devices, exposed to userspace via /dev/videoX nodes. + +To configure this pipeline, the userspace must take the following steps: + +1) Set up media links between entities: enable the links from the sensor to the + receiver and from the receiver to the DMA engines. This step does not differ + from normal non-multiplexed media controller setup. + +2) Configure routing + +.. flat-table:: Camera sensor. There are no configurable routes. + :header-rows: 1 + + * - Sink Pad/Stream + - Source Pad/Stream + - Routing Flags + - Comments + * - 1/0 + - 0/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Pixel data stream from the sink pad + * - 2/0 + - 0/1 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Metadata stream from the internal sink pad + +.. flat-table:: Receiver routing table. Typically both routes need to be + explicitly set. + :header-rows: 1 + + * - Sink Pad/Stream + - Source Pad/Stream + - Routing Flags + - Comments + * - 0/0 + - 1/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Pixel data stream from camera sensor + * - 0/1 + - 2/0 + - V4L2_SUBDEV_ROUTE_FL_ACTIVE + - Metadata stream from camera sensor + +The options available in sensor's routing configuration are dictated by hardware +capabilities: typically camera sensors always produce an image data stream while +it may be possible to enable and disable the embedded data stream. + +3) Configure formats and selections + + This example assumes that the formats are propagated from sink pad to the + source pad as-is. The tables contain fields of both struct v4l2_subdev_format + and struct v4l2_mbus_framefmt. + +.. flat-table:: Formats set on the sub-devices. Bold values are set, others are + static or propagated. The order is aligned with configured + routes. + :header-rows: 1 + :fill-cells: + + * - Sub-device + - Pad/Stream + - Width + - Height + - Code + * - :rspan:`3` IMX219 + - 1/0 + - 3296 + - 2480 + - MEDIA_BUS_FMT_SRGGB10 + * - 0/0 + - **3296** + - **2480** + - **MEDIA_BUS_FMT_SRGGB10** + * - 2/0 + - 3296 + - 2 + - MEDIA_BUS_FMT_IMX219_EMBEDDED + * - 1/1 + - 3296 + - 2 + - MEDIA_BUS_FMT_META_10 + * - :rspan:`3` CSI-2 receiver + - 0/0 + - **3296** + - **2480** + - **MEDIA_BUS_FMT_SRGGB10** + * - 1/0 + - 3296 + - 2480 + - MEDIA_BUS_FMT_SRGGB10 + * - 0/1 + - **3296** + - **2** + - **MEDIA_BUS_FMT_META_10** + * - 2/0 + - 3296 + - 2 + - MEDIA_BUS_FMT_META_10 + +The embedded data format does not need to be configured as the format is +dictated by the pixel data format in this case.
Document internal sink pads, pads that have both SINK and INTERNAL flags set. Use the IMX219 camera sensor as an example. Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com> Reviewed-by Julien Massot <julien.massot@collabora.com> --- .../userspace-api/media/v4l/dev-subdev.rst | 145 ++++++++++++++++++ 1 file changed, 145 insertions(+)