[v2,00/11] Venus stateful Codec API

Message ID 20190628130002.24293-1-stanimir.varbanov@linaro.org
State New
Headers show

Commit Message

Stanimir Varbanov June 28, 2019, 12:59 p.m.
Hello,

Here is v2 of the Venus transition to stateful codec API
compliance. The v2 can be found at [1].

Changes since v1:
 * codec_state is now enum
 * dropped IS_OUT and IS_CAP macros and use vb2_start_streaming_called()
 * corrected g_fmt and reconfig logic
 * s/vdec_dst_buffers_done/vdec_cancel_dst_buffers
 * use v4l2_m2m_ioctl_try_decoder_cmd M2M helper
 * various fixes to make v4l2-compliance pass the streaming test

To test the streaming with --stream-from-hdr v4l2-compliance option I have
to make the following hack (it is needed because the size of decoder input
buffers (OUTPUT queue) is not enough for the h264 bitstream, i.e the driver
default resolution is 64x64 but the h264 stream is 320x240):

 
... and with it the output of the test is:

dragonboard-410c:~/v4l-utils$ v4l2-compliance --stream-from-hdr test-25fps.h264.hdr -s250 -d /dev/video1
v4l2-compliance SHA: 08fed4d0edb1492b91d9d1054c36fed95c372eaa, 64 bits                                                                                                                                             
                                                                                                                                                                                                                   
Compliance test for qcom-venus device /dev/video1:

Driver Info:
        Driver name      : qcom-venus
        Card type        : Qualcomm Venus video decoder
        Bus info         : platform:qcom-venus
        Driver version   : 5.2.0
        Capabilities     : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
        Detected Stateful Decoder

Required ioctls:
        test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
        test second /dev/video1 open: OK
        test VIDIOC_QUERYCAP: OK
        test VIDIOC_G/S_PRIORITY: OK
        test for unlimited opens: OK

Debug ioctls:
        test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
        test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
        test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
        test VIDIOC_ENUMAUDIO: OK (Not Supported)
        test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
        test VIDIOC_G/S_AUDIO: OK (Not Supported)
        Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
        test VIDIOC_G/S_MODULATOR: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_ENUMAUDOUT: OK (Not Supported)
        test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
        test VIDIOC_G/S_AUDOUT: OK (Not Supported)
        Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
        test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
        test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
        test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
        test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
        test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
        test VIDIOC_QUERYCTRL: OK
        test VIDIOC_G/S_CTRL: OK
        test VIDIOC_G/S/TRY_EXT_CTRLS: OK
        test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
        test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
        Standard Controls: 9 Private Controls: 0

Format ioctls:
        test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
        test VIDIOC_G/S_PARM: OK (Not Supported)
        test VIDIOC_G_FBUF: OK (Not Supported)
        test VIDIOC_G_FMT: OK
        test VIDIOC_TRY_FMT: OK
        test VIDIOC_S_FMT: OK
        test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
        test Cropping: OK
        test Composing: OK
        test Scaling: OK

Codec ioctls:
        test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
        test VIDIOC_G_ENC_INDEX: OK (Not Supported)
        test VIDIOC_(TRY_)DECODER_CMD: OK

Buffer ioctls:
        test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
        test VIDIOC_EXPBUF: OK
        test Requests: OK (Not Supported)

Test input 0:

Streaming ioctls:
        test read/write: OK (Not Supported)
        test blocking wait: OK
        Video Capture Multiplanar: Captured 251 buffers   
        test MMAP (select): OK
        Video Capture Multiplanar: Captured 251 buffers   
        test MMAP (epoll): OK
        test USERPTR (select): OK (Not Supported)
        test DMABUF: Cannot test, specify --expbuf-device

Total for qcom-venus device /dev/video1: 49, Succeeded: 49, Failed: 0, Warnings: 0

regards,
Stan

[1] https://lore.kernel.org/lkml/0fc547f46d2ff90a2a8adf957cca7297@codeaurora.org/T/

Malathi Gottam (1):
  venus: venc: amend buffer size for bitstream plane

Stanimir Varbanov (10):
  venus: helpers: export few helper functions
  venus: hfi: add type argument to hfi flush function
  venus: hfi: export few HFI functions
  venus: hfi: return an error if session_init is already called
  venus: helpers: add three more helper functions
  venus: vdec_ctrls: get real minimum buffers for capture
  venus: vdec: allow bigger sizeimage set by clients
  venus: make decoder compliant with stateful codec API
  venus: helpers: handle correctly vbuf field
  venus: dec: populate properly timestamps and flags for capture buffers

 drivers/media/platform/qcom/venus/core.h      |  32 +-
 drivers/media/platform/qcom/venus/helpers.c   | 199 ++++++-
 drivers/media/platform/qcom/venus/helpers.h   |  12 +
 drivers/media/platform/qcom/venus/hfi.c       |  11 +-
 drivers/media/platform/qcom/venus/hfi.h       |   2 +-
 drivers/media/platform/qcom/venus/vdec.c      | 537 ++++++++++++++----
 .../media/platform/qcom/venus/vdec_ctrls.c    |   7 +-
 drivers/media/platform/qcom/venus/venc.c      |  13 +-
 8 files changed, 656 insertions(+), 157 deletions(-)

-- 
2.17.1

Comments

Hans Verkuil June 28, 2019, 1:37 p.m. | #1
On 6/28/19 2:59 PM, Stanimir Varbanov wrote:
> Hello,

> 

> Here is v2 of the Venus transition to stateful codec API

> compliance. The v2 can be found at [1].

> 

> Changes since v1:

>  * codec_state is now enum

>  * dropped IS_OUT and IS_CAP macros and use vb2_start_streaming_called()

>  * corrected g_fmt and reconfig logic

>  * s/vdec_dst_buffers_done/vdec_cancel_dst_buffers

>  * use v4l2_m2m_ioctl_try_decoder_cmd M2M helper

>  * various fixes to make v4l2-compliance pass the streaming test

> 

> To test the streaming with --stream-from-hdr v4l2-compliance option I have

> to make the following hack (it is needed because the size of decoder input

> buffers (OUTPUT queue) is not enough for the h264 bitstream, i.e the driver

> default resolution is 64x64 but the h264 stream is 320x240):

> 

> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp

> index c71dcf65b721..dc0fcf20d3e4 100644

> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp

> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp

> @@ -1294,6 +1294,11 @@ int testMmap(struct node *node, unsigned frame_count, enum poll_mode pollmode)

>                                         fmt.s_sizeimage(fmt.g_sizeimage(p) * 2, p);

>                         }

>                         fail_on_test(q.create_bufs(node, 1, &fmt));

> +

> +                       for (unsigned p = 0; p < fmt.g_num_planes(); p++)

> +                               fmt.s_sizeimage(fmt.g_sizeimage(p) * 2, p);

> +                       node->s_fmt(fmt);

> +

>                         fail_on_test(q.reqbufs(node, 2));

>                 }

>                 if (v4l_type_is_output(type))


Does the venus driver set sizeimage based on the given output resolution?

E.g. if v4l2-compliance would first set the output resolution to 320x240,
is the returned sizeimage value OK in that case?

And this also means that the venus driver requires each buffer to have
a single compressed frame, right? I.e. it can't be spread over multiple
OUTPUT buffers.

We really need to let userspace know about such restrictions.

Stanimir, can you list the restrictions of the decoder for the various
codecs?

Regards,

	Hans
Stanimir Varbanov June 28, 2019, 2:23 p.m. | #2
Hi Hans,

On 6/28/19 4:37 PM, Hans Verkuil wrote:
> On 6/28/19 2:59 PM, Stanimir Varbanov wrote:

>> Hello,

>>

>> Here is v2 of the Venus transition to stateful codec API

>> compliance. The v2 can be found at [1].

>>

>> Changes since v1:

>>  * codec_state is now enum

>>  * dropped IS_OUT and IS_CAP macros and use vb2_start_streaming_called()

>>  * corrected g_fmt and reconfig logic

>>  * s/vdec_dst_buffers_done/vdec_cancel_dst_buffers

>>  * use v4l2_m2m_ioctl_try_decoder_cmd M2M helper

>>  * various fixes to make v4l2-compliance pass the streaming test

>>

>> To test the streaming with --stream-from-hdr v4l2-compliance option I have

>> to make the following hack (it is needed because the size of decoder input

>> buffers (OUTPUT queue) is not enough for the h264 bitstream, i.e the driver

>> default resolution is 64x64 but the h264 stream is 320x240):

>>

>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp

>> index c71dcf65b721..dc0fcf20d3e4 100644

>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp

>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp

>> @@ -1294,6 +1294,11 @@ int testMmap(struct node *node, unsigned frame_count, enum poll_mode pollmode)

>>                                         fmt.s_sizeimage(fmt.g_sizeimage(p) * 2, p);

>>                         }

>>                         fail_on_test(q.create_bufs(node, 1, &fmt));

>> +

>> +                       for (unsigned p = 0; p < fmt.g_num_planes(); p++)

>> +                               fmt.s_sizeimage(fmt.g_sizeimage(p) * 2, p);

>> +                       node->s_fmt(fmt);

>> +

>>                         fail_on_test(q.reqbufs(node, 2));

>>                 }

>>                 if (v4l_type_is_output(type))

> 

> Does the venus driver set sizeimage based on the given output resolution?


Yes.

> 

> E.g. if v4l2-compliance would first set the output resolution to 320x240,

> is the returned sizeimage value OK in that case?


Yes.

Here are few options to me:
 - set the correct resolution
 - set 0x0 and sizeimage at some arbitrary value (1 or 2MB). Despite if
the bitstream is 4K it will not be enough if the bitrate is huge.
 - invent some mechanism to trigger reconfiguration on the OUTPUT queue
as well (similar to the CAPTURE queue)

> 

> And this also means that the venus driver requires each buffer to have

> a single compressed frame, right? I.e. it can't be spread over multiple

> OUTPUT buffers.


I cannot say for sure but that is how all downstream cases uses it i.e.
one compressed frame per input buffer. I wonder if you fill input
decoder buffer with many compressed frames in one input decoder buffer
how you pass the timestamp for every packet?

> 

> We really need to let userspace know about such restrictions.

> 

> Stanimir, can you list the restrictions of the decoder for the various

> codecs?


What you mean? Restrictions like "one compressed frame per input buffer"?

-- 
regards,
Stan
Hans Verkuil June 28, 2019, 2:25 p.m. | #3
On 6/28/19 4:23 PM, Stanimir Varbanov wrote:
> Hi Hans,

> 

> On 6/28/19 4:37 PM, Hans Verkuil wrote:

>> On 6/28/19 2:59 PM, Stanimir Varbanov wrote:

>>> Hello,

>>>

>>> Here is v2 of the Venus transition to stateful codec API

>>> compliance. The v2 can be found at [1].

>>>

>>> Changes since v1:

>>>  * codec_state is now enum

>>>  * dropped IS_OUT and IS_CAP macros and use vb2_start_streaming_called()

>>>  * corrected g_fmt and reconfig logic

>>>  * s/vdec_dst_buffers_done/vdec_cancel_dst_buffers

>>>  * use v4l2_m2m_ioctl_try_decoder_cmd M2M helper

>>>  * various fixes to make v4l2-compliance pass the streaming test

>>>

>>> To test the streaming with --stream-from-hdr v4l2-compliance option I have

>>> to make the following hack (it is needed because the size of decoder input

>>> buffers (OUTPUT queue) is not enough for the h264 bitstream, i.e the driver

>>> default resolution is 64x64 but the h264 stream is 320x240):

>>>

>>> diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp

>>> index c71dcf65b721..dc0fcf20d3e4 100644

>>> --- a/utils/v4l2-compliance/v4l2-test-buffers.cpp

>>> +++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp

>>> @@ -1294,6 +1294,11 @@ int testMmap(struct node *node, unsigned frame_count, enum poll_mode pollmode)

>>>                                         fmt.s_sizeimage(fmt.g_sizeimage(p) * 2, p);

>>>                         }

>>>                         fail_on_test(q.create_bufs(node, 1, &fmt));

>>> +

>>> +                       for (unsigned p = 0; p < fmt.g_num_planes(); p++)

>>> +                               fmt.s_sizeimage(fmt.g_sizeimage(p) * 2, p);

>>> +                       node->s_fmt(fmt);

>>> +

>>>                         fail_on_test(q.reqbufs(node, 2));

>>>                 }

>>>                 if (v4l_type_is_output(type))

>>

>> Does the venus driver set sizeimage based on the given output resolution?

> 

> Yes.

> 

>>

>> E.g. if v4l2-compliance would first set the output resolution to 320x240,

>> is the returned sizeimage value OK in that case?

> 

> Yes.

> 

> Here are few options to me:

>  - set the correct resolution

>  - set 0x0 and sizeimage at some arbitrary value (1 or 2MB). Despite if

> the bitstream is 4K it will not be enough if the bitrate is huge.

>  - invent some mechanism to trigger reconfiguration on the OUTPUT queue

> as well (similar to the CAPTURE queue)

> 

>>

>> And this also means that the venus driver requires each buffer to have

>> a single compressed frame, right? I.e. it can't be spread over multiple

>> OUTPUT buffers.

> 

> I cannot say for sure but that is how all downstream cases uses it i.e.

> one compressed frame per input buffer. I wonder if you fill input

> decoder buffer with many compressed frames in one input decoder buffer

> how you pass the timestamp for every packet?

> 

>>

>> We really need to let userspace know about such restrictions.

>>

>> Stanimir, can you list the restrictions of the decoder for the various

>> codecs?

> 

> What you mean? Restrictions like "one compressed frame per input buffer"?

> 


Yes :-)

Regards,

	Hans

Patch

diff --git a/utils/v4l2-compliance/v4l2-test-buffers.cpp b/utils/v4l2-compliance/v4l2-test-buffers.cpp
index c71dcf65b721..dc0fcf20d3e4 100644
--- a/utils/v4l2-compliance/v4l2-test-buffers.cpp
+++ b/utils/v4l2-compliance/v4l2-test-buffers.cpp
@@ -1294,6 +1294,11 @@  int testMmap(struct node *node, unsigned frame_count, enum poll_mode pollmode)
                                        fmt.s_sizeimage(fmt.g_sizeimage(p) * 2, p);
                        }
                        fail_on_test(q.create_bufs(node, 1, &fmt));
+
+                       for (unsigned p = 0; p < fmt.g_num_planes(); p++)
+                               fmt.s_sizeimage(fmt.g_sizeimage(p) * 2, p);
+                       node->s_fmt(fmt);
+
                        fail_on_test(q.reqbufs(node, 2));
                }
                if (v4l_type_is_output(type))