| Message ID | 20241128145144.61475-1-isaac.scott@ideasonboard.com |
|---|---|
| Headers | show |
| Series | Fix Sonix Technology MJPEG streams | expand |
On Thu, 28 Nov 2024 at 15:53, Isaac Scott <isaac.scott@ideasonboard.com> wrote: > > Some cameras, such as the Sonix Technology Co. 292A, exhibit issues when > running two parallel streams, causing USB packets to be dropped when an > H.264 stream posts a keyframe while an MJPEG stream is running > simultaneously. This occasionally causes the driver to erroneously > output two consecutive JPEG images as a single frame. > > To fix this, we inspect the buffer, and trigger a new frame when we > find an SOI. > > Signed-off-by: Isaac Scott <isaac.scott@ideasonboard.com> > --- > drivers/media/usb/uvc/uvc_video.c | 27 ++++++++++++++++++++++++++- > drivers/media/usb/uvc/uvcvideo.h | 1 + > 2 files changed, 27 insertions(+), 1 deletion(-) > > diff --git a/drivers/media/usb/uvc/uvc_video.c b/drivers/media/usb/uvc/uvc_video.c > index e00f38dd07d9..6d800a099749 100644 > --- a/drivers/media/usb/uvc/uvc_video.c > +++ b/drivers/media/usb/uvc/uvc_video.c > @@ -20,6 +20,7 @@ > #include <linux/atomic.h> > #include <linux/unaligned.h> > > +#include <media/jpeg.h> > #include <media/v4l2-common.h> > > #include "uvcvideo.h" > @@ -1116,6 +1117,7 @@ static void uvc_video_stats_stop(struct uvc_streaming *stream) > static int uvc_video_decode_start(struct uvc_streaming *stream, > struct uvc_buffer *buf, const u8 *data, int len) > { > + u8 header_len; > u8 fid; > > /* > @@ -1129,6 +1131,7 @@ static int uvc_video_decode_start(struct uvc_streaming *stream, > return -EINVAL; > } > > + header_len = data[0]; > fid = data[1] & UVC_STREAM_FID; > > /* > @@ -1210,9 +1213,31 @@ static int uvc_video_decode_start(struct uvc_streaming *stream, > return -EAGAIN; > } > > + /* > + * Some cameras, when running two parallel streams (one MJPEG alongside > + * another non-MJPEG stream), are known to lose the EOF packet for a frame. > + * We can detect the end of a frame by checking for a new SOI marker, as > + * the SOI always lies on the packet boundary between two frames for > + * these devices. > + */ > + if (stream->dev->quirks & UVC_QUIRK_MJPEG_NO_EOF && > + (stream->cur_format->fcc == V4L2_PIX_FMT_MJPEG || > + stream->cur_format->fcc == V4L2_PIX_FMT_JPEG)) { > + const u8 *packet = data + header_len; > + > + if (len >= header_len + 2 && > + packet[0] == 0xff && packet[1] == JPEG_MARKER_SOI && > + buf->bytesused != 0) { nit: !buf->bytesused (please ignore if you prefer your way) > + buf->state = UVC_BUF_STATE_READY; > + buf->error = 1; I have a question. Lets say that you have two frames: A and B, each one has 4 packets: A1A2A3A4B1B2B3B4 The last package of A is lost because the device is non-compliant. A1A2A3B1B2B3B4 You detect this by inspecting every packet, and checking for the values 0xff, JPEG_MARKER_SOI at the beggining of the packet. Can't that value happen in the middle of the image, let's say in A2, A3, B2, B3... ? If that happens, won't you be losing frames? Also, If I get it right, the device not only loses the packet A4, but it sends the wrong fid for all the Bx packets? Maybe the device is not losing A4 but sending wrong fids? Have you tried not setting buf->error=1 and inspecting the "invalid" image? I am not saying that it is incorrect. I am just trying to understand the patch better. :) > + stream->last_fid ^= UVC_STREAM_FID; It would be nice to have uvc_dbg() here, in case we want to debug what is going on. > + return -EAGAIN; > + } > + } > + > stream->last_fid = fid; > > - return data[0]; > + return header_len; > } > > static inline enum dma_data_direction uvc_stream_dir( > diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h > index b7d24a853ce4..040073326a24 100644 > --- a/drivers/media/usb/uvc/uvcvideo.h > +++ b/drivers/media/usb/uvc/uvcvideo.h > @@ -76,6 +76,7 @@ > #define UVC_QUIRK_NO_RESET_RESUME 0x00004000 > #define UVC_QUIRK_DISABLE_AUTOSUSPEND 0x00008000 > #define UVC_QUIRK_INVALID_DEVICE_SOF 0x00010000 > +#define UVC_QUIRK_MJPEG_NO_EOF 0x00020000 > > /* Format flags */ > #define UVC_FMT_FLAG_COMPRESSED 0x00000001 > -- > 2.43.0 > >
Hi Ricardo, I hope you are well! On Thu, 2024-11-28 at 17:16 +0100, Ricardo Ribalda wrote: > On Thu, 28 Nov 2024 at 15:53, Isaac Scott > <isaac.scott@ideasonboard.com> wrote: > > > > Some cameras, such as the Sonix Technology Co. 292A, exhibit issues > > when > > running two parallel streams, causing USB packets to be dropped > > when an > > H.264 stream posts a keyframe while an MJPEG stream is running > > simultaneously. This occasionally causes the driver to erroneously > > output two consecutive JPEG images as a single frame. > > > > To fix this, we inspect the buffer, and trigger a new frame when we > > find an SOI. > > > > Signed-off-by: Isaac Scott <isaac.scott@ideasonboard.com> > > --- > > drivers/media/usb/uvc/uvc_video.c | 27 ++++++++++++++++++++++++++- > > drivers/media/usb/uvc/uvcvideo.h | 1 + > > 2 files changed, 27 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/media/usb/uvc/uvc_video.c > > b/drivers/media/usb/uvc/uvc_video.c > > index e00f38dd07d9..6d800a099749 100644 > > --- a/drivers/media/usb/uvc/uvc_video.c > > +++ b/drivers/media/usb/uvc/uvc_video.c > > @@ -20,6 +20,7 @@ > > #include <linux/atomic.h> > > #include <linux/unaligned.h> > > > > +#include <media/jpeg.h> > > #include <media/v4l2-common.h> > > > > #include "uvcvideo.h" > > @@ -1116,6 +1117,7 @@ static void uvc_video_stats_stop(struct > > uvc_streaming *stream) > > static int uvc_video_decode_start(struct uvc_streaming *stream, > > struct uvc_buffer *buf, const u8 *data, int len) > > { > > + u8 header_len; > > u8 fid; > > > > /* > > @@ -1129,6 +1131,7 @@ static int uvc_video_decode_start(struct > > uvc_streaming *stream, > > return -EINVAL; > > } > > > > + header_len = data[0]; > > fid = data[1] & UVC_STREAM_FID; > > > > /* > > @@ -1210,9 +1213,31 @@ static int uvc_video_decode_start(struct > > uvc_streaming *stream, > > return -EAGAIN; > > } > > > > + /* > > + * Some cameras, when running two parallel streams (one > > MJPEG alongside > > + * another non-MJPEG stream), are known to lose the EOF > > packet for a frame. > > + * We can detect the end of a frame by checking for a new > > SOI marker, as > > + * the SOI always lies on the packet boundary between two > > frames for > > + * these devices. > > + */ > > + if (stream->dev->quirks & UVC_QUIRK_MJPEG_NO_EOF && > > + (stream->cur_format->fcc == V4L2_PIX_FMT_MJPEG || > > + stream->cur_format->fcc == V4L2_PIX_FMT_JPEG)) { > > + const u8 *packet = data + header_len; > > + > > + if (len >= header_len + 2 && > > + packet[0] == 0xff && packet[1] == > > JPEG_MARKER_SOI && > > + buf->bytesused != 0) { > nit: !buf->bytesused (please ignore if you prefer your way) > > + buf->state = UVC_BUF_STATE_READY; > > + buf->error = 1; > > I have a question. Lets say that you have two frames: A and B, each > one has 4 packets: > A1A2A3A4B1B2B3B4 > The last package of A is lost because the device is non-compliant. > A1A2A3B1B2B3B4 > > You detect this by inspecting every packet, and checking for the > values 0xff, JPEG_MARKER_SOI at the beggining of the packet. > > Can't that value happen in the middle of the image, let's say in A2, > A3, B2, B3... ? If that happens, won't you be losing frames? > I have found that in MJPEG, it is required to have both an SOI (0xFFD8) and an EOI (0xFFD9) in every payload. Source: p.16, USB Device Class Definition for Video Devices: MJPEG Payload (https://usb.org/document-library/video-class-v15-document-set)) Furthermore, the JPEG standard also explicitly defines 0xFFD8 to be the start of image marker, meaning its usage outside that functionality would not adhere to the standard. If it appears in the middle of a payload, the payload should be marked as invalid. Source: p. 32, Digital Compression and Coding of Continuous-Tone Still Images - Requirements and Guidelines (https://www.w3.org/Graphics/JPEG/itu-t81.pdf) > Also, If I get it right, the device not only loses the packet A4, but > it sends the wrong fid for all the Bx packets? Before the patch, B would be joined into A, and gets delivered to user space as A1, A2, A3, A4, A5, A6, A7, A8, C1, C2, C3... > Maybe the device is not losing A4 but sending wrong fids? Have you > tried not setting buf->error=1 and inspecting the "invalid" image? > I saw during the diagnosis of the issue by analysing the USB packets sent by the camera that the packet containing the EOF does not get sent whatsoever when the two streams are running simultaneously. > I am not saying that it is incorrect. I am just trying to understand > the patch better. :) > > > > + stream->last_fid ^= UVC_STREAM_FID; > It would be nice to have uvc_dbg() here, in case we want to debug > what > is going on. > > + return -EAGAIN; > > + } > > + } > > + > > stream->last_fid = fid; > > > > - return data[0]; > > + return header_len; > > } > > > > static inline enum dma_data_direction uvc_stream_dir( > > diff --git a/drivers/media/usb/uvc/uvcvideo.h > > b/drivers/media/usb/uvc/uvcvideo.h > > index b7d24a853ce4..040073326a24 100644 > > --- a/drivers/media/usb/uvc/uvcvideo.h > > +++ b/drivers/media/usb/uvc/uvcvideo.h > > @@ -76,6 +76,7 @@ > > #define UVC_QUIRK_NO_RESET_RESUME 0x00004000 > > #define UVC_QUIRK_DISABLE_AUTOSUSPEND 0x00008000 > > #define UVC_QUIRK_INVALID_DEVICE_SOF 0x00010000 > > +#define UVC_QUIRK_MJPEG_NO_EOF 0x00020000 > > > > /* Format flags */ > > #define UVC_FMT_FLAG_COMPRESSED 0x00000001 > > -- > > 2.43.0 > > > > > > Best wishes, Isaac
Hi Isaac Reviewed-by: Ricardo Ribalda <ribalda@chromium.org> On Fri, 29 Nov 2024 at 11:36, Isaac Scott <isaac.scott@ideasonboard.com> wrote: > > Hi Ricardo, > > I hope you are well! > > On Thu, 2024-11-28 at 17:16 +0100, Ricardo Ribalda wrote: > > On Thu, 28 Nov 2024 at 15:53, Isaac Scott > > <isaac.scott@ideasonboard.com> wrote: > > > > > > Some cameras, such as the Sonix Technology Co. 292A, exhibit issues > > > when > > > running two parallel streams, causing USB packets to be dropped > > > when an > > > H.264 stream posts a keyframe while an MJPEG stream is running > > > simultaneously. This occasionally causes the driver to erroneously > > > output two consecutive JPEG images as a single frame. > > > > > > To fix this, we inspect the buffer, and trigger a new frame when we > > > find an SOI. > > > > > > Signed-off-by: Isaac Scott <isaac.scott@ideasonboard.com> > > > --- > > > drivers/media/usb/uvc/uvc_video.c | 27 ++++++++++++++++++++++++++- > > > drivers/media/usb/uvc/uvcvideo.h | 1 + > > > 2 files changed, 27 insertions(+), 1 deletion(-) > > > > > > diff --git a/drivers/media/usb/uvc/uvc_video.c > > > b/drivers/media/usb/uvc/uvc_video.c > > > index e00f38dd07d9..6d800a099749 100644 > > > --- a/drivers/media/usb/uvc/uvc_video.c > > > +++ b/drivers/media/usb/uvc/uvc_video.c > > > @@ -20,6 +20,7 @@ > > > #include <linux/atomic.h> > > > #include <linux/unaligned.h> > > > > > > +#include <media/jpeg.h> > > > #include <media/v4l2-common.h> > > > > > > #include "uvcvideo.h" > > > @@ -1116,6 +1117,7 @@ static void uvc_video_stats_stop(struct > > > uvc_streaming *stream) > > > static int uvc_video_decode_start(struct uvc_streaming *stream, > > > struct uvc_buffer *buf, const u8 *data, int len) > > > { > > > + u8 header_len; > > > u8 fid; > > > > > > /* > > > @@ -1129,6 +1131,7 @@ static int uvc_video_decode_start(struct > > > uvc_streaming *stream, > > > return -EINVAL; > > > } > > > > > > + header_len = data[0]; > > > fid = data[1] & UVC_STREAM_FID; > > > > > > /* > > > @@ -1210,9 +1213,31 @@ static int uvc_video_decode_start(struct > > > uvc_streaming *stream, > > > return -EAGAIN; > > > } > > > > > > + /* > > > + * Some cameras, when running two parallel streams (one > > > MJPEG alongside > > > + * another non-MJPEG stream), are known to lose the EOF > > > packet for a frame. > > > + * We can detect the end of a frame by checking for a new > > > SOI marker, as > > > + * the SOI always lies on the packet boundary between two > > > frames for > > > + * these devices. > > > + */ > > > + if (stream->dev->quirks & UVC_QUIRK_MJPEG_NO_EOF && > > > + (stream->cur_format->fcc == V4L2_PIX_FMT_MJPEG || > > > + stream->cur_format->fcc == V4L2_PIX_FMT_JPEG)) { > > > + const u8 *packet = data + header_len; > > > + > > > + if (len >= header_len + 2 && > > > + packet[0] == 0xff && packet[1] == > > > JPEG_MARKER_SOI && > > > + buf->bytesused != 0) { > > nit: !buf->bytesused (please ignore if you prefer your way) > > > + buf->state = UVC_BUF_STATE_READY; > > > + buf->error = 1; > > > > I have a question. Lets say that you have two frames: A and B, each > > one has 4 packets: > > A1A2A3A4B1B2B3B4 > > The last package of A is lost because the device is non-compliant. > > A1A2A3B1B2B3B4 > > > > You detect this by inspecting every packet, and checking for the > > values 0xff, JPEG_MARKER_SOI at the beggining of the packet. > > > > Can't that value happen in the middle of the image, let's say in A2, > > A3, B2, B3... ? If that happens, won't you be losing frames? > > > > I have found that in MJPEG, it is required to have both an SOI (0xFFD8) > and an EOI (0xFFD9) in every payload. Thanks a lot for checking it out. If you happen to make a new version, that would be a very nice info to add to the comment. > > Source: p.16, USB Device Class Definition for Video Devices: MJPEG > Payload > (https://usb.org/document-library/video-class-v15-document-set)) > > Furthermore, the JPEG standard also explicitly defines 0xFFD8 to be the > start of image marker, meaning its usage outside that functionality > would not adhere to the standard. If it appears in the middle of a > payload, the payload should be marked as invalid. > > Source: p. 32, Digital Compression and Coding of Continuous-Tone Still > Images - Requirements and Guidelines > (https://www.w3.org/Graphics/JPEG/itu-t81.pdf) > > > Also, If I get it right, the device not only loses the packet A4, but > > it sends the wrong fid for all the Bx packets? > > Before the patch, B would be joined into A, and gets delivered to user > space as A1, A2, A3, A4, A5, A6, A7, A8, C1, C2, C3... > So it seems like the fid value does not change during all the A... Thanks again and sorry for not raising the questions before. > > > Maybe the device is not losing A4 but sending wrong fids? Have you > > tried not setting buf->error=1 and inspecting the "invalid" image? > > > > I saw during the diagnosis of the issue by analysing the USB packets > sent by the camera that the packet containing the EOF does not get sent > whatsoever when the two streams are running simultaneously. > > > I am not saying that it is incorrect. I am just trying to understand > > the patch better. :) > > > > > > > + stream->last_fid ^= UVC_STREAM_FID; > > It would be nice to have uvc_dbg() here, in case we want to debug > > what > > is going on. > > > + return -EAGAIN; > > > + } > > > + } > > > + > > > stream->last_fid = fid; > > > > > > - return data[0]; > > > + return header_len; > > > } > > > > > > static inline enum dma_data_direction uvc_stream_dir( > > > diff --git a/drivers/media/usb/uvc/uvcvideo.h > > > b/drivers/media/usb/uvc/uvcvideo.h > > > index b7d24a853ce4..040073326a24 100644 > > > --- a/drivers/media/usb/uvc/uvcvideo.h > > > +++ b/drivers/media/usb/uvc/uvcvideo.h > > > @@ -76,6 +76,7 @@ > > > #define UVC_QUIRK_NO_RESET_RESUME 0x00004000 > > > #define UVC_QUIRK_DISABLE_AUTOSUSPEND 0x00008000 > > > #define UVC_QUIRK_INVALID_DEVICE_SOF 0x00010000 > > > +#define UVC_QUIRK_MJPEG_NO_EOF 0x00020000 > > > > > > /* Format flags */ > > > #define UVC_FMT_FLAG_COMPRESSED 0x00000001 > > > -- > > > 2.43.0 > > > > > > > > > > > > Best wishes, > > Isaac
Some cameras, such as the Sonix Technology Co. 292A (also presents as the Microdia Webcam Vitade AF), allow multiple streams to be sent through the same USB device. It was found during testing that occasionally, when one H.264 stream sends a new keyframe during an MJPEG stream, a URB buffer would be lost, and a frame is lost in the resulting capture. This was because some EOF markers in URB buffers are lost. This particular use case is not covered by the uvc_video driver, which causes two consecutive buffers to be erroneously considered to be as part of the same buffer, and lead to the whole erroneous buffer to be dropped. This can be seen when using the current driver, as frames occasionally are double the expected size, containing two images. Furthermore, the interval between the frames was double that of what it should be (~66ms as opposed to the usual ~33ms). Upon further investigation, it was found that the packets within the erroneous buffer were being dropped. We can detect the end of a frame by checking for a new SOI marker in the JPEG. We should only check for this in the first two bytes after the header, as the new packets are not corrupted and should be used at the start of the next frame as part of a new buffer. Check the buffer for a new SOI on JPEG streams and complete the preceding buffer using EAGAIN, and invert the FID to make sure the erroneous frame is not dropped. It should only do this check when there is data after the header, as some packets will contain only a header and no payload. The changes in this series supply a new quirk for the UVC Driver, which cause the buffers of MJPEG streams to be closed when a new JPEG SOI marker is discovered, and be treated as a new buffer. This works around the missing EOF marker and separates the buffers. lsusb output: Bus 001 Device 006: ID 0c45:6366 Microdia Webcam Vitade AF Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.00 bDeviceClass 239 Miscellaneous Device bDeviceSubClass 2 [unknown] bDeviceProtocol 1 Interface Association bMaxPacketSize0 64 idVendor 0x0c45 Microdia idProduct 0x6366 Webcam Vitade AF bcdDevice 1.00 iManufacturer 2 Sonix Technology Co., Ltd. iProduct 1 292A-IPC-AR0330 iSerial 3 SN0001 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 0x05b9 bNumInterfaces 3 bConfigurationValue 1 iConfiguration 0 bmAttributes 0x80 (Bus Powered) MaxPower 500mA Interface Association: bLength 8 bDescriptorType 11 bFirstInterface 0 bInterfaceCount 3 bFunctionClass 14 Video bFunctionSubClass 3 Video Interface Collection bFunctionProtocol 0 iFunction 5 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 1 Video Control bInterfaceProtocol 0 iInterface 5 VideoControl Interface Descriptor: bLength 14 bDescriptorType 36 bDescriptorSubtype 1 (HEADER) bcdUVC 1.00 wTotalLength 0x0074 dwClockFrequency 15.000000MHz bInCollection 2 baInterfaceNr( 0) 1 baInterfaceNr( 1) 2 VideoControl Interface Descriptor: bLength 9 bDescriptorType 36 bDescriptorSubtype 3 (OUTPUT_TERMINAL) bTerminalID 5 wTerminalType 0x0101 USB Streaming bAssocTerminal 0 bSourceID 4 iTerminal 0 VideoControl Interface Descriptor: bLength 28 bDescriptorType 36 bDescriptorSubtype 6 (EXTENSION_UNIT) bUnitID 3 guidExtensionCode {28f03370-6311-4a2e-ba2c-6890eb334016} bNumControls 24 bNrInPins 1 baSourceID( 0) 2 bControlSize 3 bmControls( 0) 0xff bmControls( 1) 0xff bmControls( 2) 0xff iExtension 0 VideoControl Interface Descriptor: bLength 27 bDescriptorType 36 bDescriptorSubtype 6 (EXTENSION_UNIT) bUnitID 4 guidExtensionCode {dddf7394-973e-4727-bed9-04ed6426dc67} bNumControls 16 bNrInPins 1 baSourceID( 0) 3 bControlSize 2 bmControls( 0) 0xff bmControls( 1) 0x01 iExtension 0 VideoControl Interface Descriptor: bLength 18 bDescriptorType 36 bDescriptorSubtype 2 (INPUT_TERMINAL) bTerminalID 1 wTerminalType 0x0201 Camera Sensor bAssocTerminal 0 iTerminal 0 wObjectiveFocalLengthMin 0 wObjectiveFocalLengthMax 0 wOcularFocalLength 0 bControlSize 3 bmControls 0x0000000e Auto-Exposure Mode Auto-Exposure Priority Exposure Time (Absolute) VideoControl Interface Descriptor: bLength 11 bDescriptorType 36 bDescriptorSubtype 5 (PROCESSING_UNIT) Warning: Descriptor too short bUnitID 2 bSourceID 1 wMaxMultiplier 0 bControlSize 2 bmControls 0x0000177f Brightness Contrast Hue Saturation Sharpness Gamma White Balance Temperature Backlight Compensation Gain Power Line Frequency White Balance Temperature, Auto iProcessing 0 bmVideoStandards 0x09 None SECAM - 625/50 VideoControl Interface Descriptor: bLength 9 bDescriptorType 36 bDescriptorSubtype 3 (OUTPUT_TERMINAL) bTerminalID 6 wTerminalType 0x0101 USB Streaming bAssocTerminal 0 bSourceID 4 iTerminal 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x83 EP 3 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0010 1x 16 bytes bInterval 6 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 VideoStreaming Interface Descriptor: bLength 15 bDescriptorType 36 bDescriptorSubtype 1 (INPUT_HEADER) bNumFormats 2 wTotalLength 0x02bf bEndpointAddress 0x81 EP 1 IN bmInfo 0 bTerminalLink 5 bStillCaptureMethod 0 bTriggerSupport 0 bTriggerUsage 0 bControlSize 1 bmaControls( 0) 0 bmaControls( 1) 0 VideoStreaming Interface Descriptor: bLength 11 bDescriptorType 36 bDescriptorSubtype 6 (FORMAT_MJPEG) bFormatIndex 1 bNumFrameDescriptors 7 bFlags 0 Fixed-size samples: No bDefaultFrameIndex 1 bAspectRatioX 0 bAspectRatioY 0 bmInterlaceFlags 0x00 Interlaced stream or variable: No Fields per frame: 1 fields Field 1 first: No Field pattern: Field 1 only bCopyProtect 0 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 7 (FRAME_MJPEG) bFrameIndex 1 bmCapabilities 0x00 Still image unsupported wWidth 1920 wHeight 1080 dwMinBitRate 82920440 dwMaxBitRate 497522640 dwMaxVideoFrameBufferSize 2073600 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 7 (FRAME_MJPEG) bFrameIndex 2 bmCapabilities 0x00 Still image unsupported wWidth 1280 wHeight 720 dwMinBitRate 73728000 dwMaxBitRate 442368000 dwMaxVideoFrameBufferSize 1843789 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 7 (FRAME_MJPEG) bFrameIndex 3 bmCapabilities 0x00 Still image unsupported wWidth 640 wHeight 480 dwMinBitRate 24576000 dwMaxBitRate 147456000 dwMaxVideoFrameBufferSize 614989 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 7 (FRAME_MJPEG) bFrameIndex 4 bmCapabilities 0x00 Still image unsupported wWidth 640 wHeight 360 dwMinBitRate 18432000 dwMaxBitRate 110592000 dwMaxVideoFrameBufferSize 461389 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 7 (FRAME_MJPEG) bFrameIndex 5 bmCapabilities 0x00 Still image unsupported wWidth 320 wHeight 240 dwMinBitRate 6144000 dwMaxBitRate 36864000 dwMaxVideoFrameBufferSize 154189 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 7 (FRAME_MJPEG) bFrameIndex 6 bmCapabilities 0x00 Still image unsupported wWidth 320 wHeight 180 dwMinBitRate 4608000 dwMaxBitRate 27648000 dwMaxVideoFrameBufferSize 115789 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 7 (FRAME_MJPEG) bFrameIndex 7 bmCapabilities 0x00 Still image unsupported wWidth 1920 wHeight 1080 dwMinBitRate 82920440 dwMaxBitRate 497522640 dwMaxVideoFrameBufferSize 2073600 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 27 bDescriptorType 36 bDescriptorSubtype 4 (FORMAT_UNCOMPRESSED) bFormatIndex 2 bNumFrameDescriptors 7 guidFormat {32595559-0000-0010-8000-00aa00389b71} bBitsPerPixel 16 bDefaultFrameIndex 1 bAspectRatioX 0 bAspectRatioY 0 bmInterlaceFlags 0x00 Interlaced stream or variable: No Fields per frame: 2 fields Field 1 first: No Field pattern: Field 1 only bCopyProtect 0 VideoStreaming Interface Descriptor: bLength 30 bDescriptorType 36 bDescriptorSubtype 5 (FRAME_UNCOMPRESSED) bFrameIndex 1 bmCapabilities 0x00 Still image unsupported wWidth 1920 wHeight 1080 dwMinBitRate 165888000 dwMaxBitRate 165888000 dwMaxVideoFrameBufferSize 4147200 dwDefaultFrameInterval 2000000 bFrameIntervalType 1 dwFrameInterval( 0) 2000000 VideoStreaming Interface Descriptor: bLength 34 bDescriptorType 36 bDescriptorSubtype 5 (FRAME_UNCOMPRESSED) bFrameIndex 2 bmCapabilities 0x00 Still image unsupported wWidth 1280 wHeight 720 dwMinBitRate 73728000 dwMaxBitRate 147456000 dwMaxVideoFrameBufferSize 1843200 dwDefaultFrameInterval 1000000 bFrameIntervalType 2 dwFrameInterval( 0) 1000000 dwFrameInterval( 1) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 5 (FRAME_UNCOMPRESSED) bFrameIndex 3 bmCapabilities 0x00 Still image unsupported wWidth 640 wHeight 480 dwMinBitRate 24576000 dwMaxBitRate 147456000 dwMaxVideoFrameBufferSize 614400 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 5 (FRAME_UNCOMPRESSED) bFrameIndex 4 bmCapabilities 0x00 Still image unsupported wWidth 640 wHeight 360 dwMinBitRate 18432000 dwMaxBitRate 110592000 dwMaxVideoFrameBufferSize 460800 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 5 (FRAME_UNCOMPRESSED) bFrameIndex 5 bmCapabilities 0x00 Still image unsupported wWidth 320 wHeight 240 dwMinBitRate 6144000 dwMaxBitRate 36864000 dwMaxVideoFrameBufferSize 153600 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 5 (FRAME_UNCOMPRESSED) bFrameIndex 6 bmCapabilities 0x00 Still image unsupported wWidth 320 wHeight 180 dwMinBitRate 4608000 dwMaxBitRate 27648000 dwMaxVideoFrameBufferSize 115200 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 30 bDescriptorType 36 bDescriptorSubtype 5 (FRAME_UNCOMPRESSED) bFrameIndex 7 bmCapabilities 0x00 Still image unsupported wWidth 1920 wHeight 1080 dwMinBitRate 165888000 dwMaxBitRate 165888000 dwMaxVideoFrameBufferSize 4147200 dwDefaultFrameInterval 2000000 bFrameIntervalType 1 dwFrameInterval( 0) 2000000 VideoStreaming Interface Descriptor: bLength 6 bDescriptorType 36 bDescriptorSubtype 13 (COLORFORMAT) bColorPrimaries 1 (BT.709,sRGB) bTransferCharacteristics 1 (BT.709) bMatrixCoefficients 4 (SMPTE 170M (BT.601)) Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 1 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x0080 1x 128 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 2 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x0100 1x 256 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 3 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x0320 1x 800 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 4 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x0b20 2x 800 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 5 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x1320 3x 800 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 6 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x1400 3x 1024 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 2 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 VideoStreaming Interface Descriptor: bLength 14 bDescriptorType 36 bDescriptorSubtype 1 (INPUT_HEADER) bNumFormats 1 wTotalLength 0x018e bEndpointAddress 0x82 EP 2 IN bmInfo 0 bTerminalLink 6 bStillCaptureMethod 0 bTriggerSupport 0 bTriggerUsage 0 bControlSize 1 bmaControls( 0) 0 VideoStreaming Interface Descriptor: bLength 28 bDescriptorType 36 bDescriptorSubtype 16 (FORMAT_FRAME_BASED) bFormatIndex 1 bNumFrameDescriptors 7 guidFormat {34363248-0000-0010-8000-00aa00389b71} bBitsPerPixel 16 bDefaultFrameIndex 1 bAspectRatioX 0 bAspectRatioY 0 bmInterlaceFlags 0x00 Interlaced stream or variable: No Fields per frame: 2 fields Field 1 first: No Field pattern: Field 1 only bCopyProtect 0 bVariableSize 1 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 17 (FRAME_FRAME_BASED) bFrameIndex 1 bmCapabilities 0x00 Still image unsupported wWidth 1920 wHeight 1080 dwMinBitRate 27648000 dwMaxBitRate 165888000 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwBytesPerLine 0 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 17 (FRAME_FRAME_BASED) bFrameIndex 2 bmCapabilities 0x00 Still image unsupported wWidth 1280 wHeight 720 dwMinBitRate 24576000 dwMaxBitRate 147456000 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwBytesPerLine 0 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 17 (FRAME_FRAME_BASED) bFrameIndex 3 bmCapabilities 0x00 Still image unsupported wWidth 640 wHeight 480 dwMinBitRate 8192000 dwMaxBitRate 49152000 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwBytesPerLine 0 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 17 (FRAME_FRAME_BASED) bFrameIndex 4 bmCapabilities 0x00 Still image unsupported wWidth 640 wHeight 360 dwMinBitRate 6144000 dwMaxBitRate 36864000 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwBytesPerLine 0 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 17 (FRAME_FRAME_BASED) bFrameIndex 5 bmCapabilities 0x00 Still image unsupported wWidth 320 wHeight 240 dwMinBitRate 2048000 dwMaxBitRate 12288000 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwBytesPerLine 0 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 17 (FRAME_FRAME_BASED) bFrameIndex 6 bmCapabilities 0x00 Still image unsupported wWidth 320 wHeight 180 dwMinBitRate 1536000 dwMaxBitRate 9216000 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwBytesPerLine 0 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 50 bDescriptorType 36 bDescriptorSubtype 17 (FRAME_FRAME_BASED) bFrameIndex 7 bmCapabilities 0x00 Still image unsupported wWidth 1920 wHeight 1080 dwMinBitRate 27648000 dwMaxBitRate 165888000 dwDefaultFrameInterval 333333 bFrameIntervalType 6 dwBytesPerLine 0 dwFrameInterval( 0) 333333 dwFrameInterval( 1) 400000 dwFrameInterval( 2) 500000 dwFrameInterval( 3) 666666 dwFrameInterval( 4) 1000000 dwFrameInterval( 5) 2000000 VideoStreaming Interface Descriptor: bLength 6 bDescriptorType 36 bDescriptorSubtype 13 (COLORFORMAT) bColorPrimaries 1 (BT.709,sRGB) bTransferCharacteristics 1 (BT.709) bMatrixCoefficients 4 (SMPTE 170M (BT.601)) Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 2 bAlternateSetting 1 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x82 EP 2 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x0080 1x 128 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 2 bAlternateSetting 2 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x82 EP 2 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x0100 1x 256 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 2 bAlternateSetting 3 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x82 EP 2 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x0320 1x 800 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 2 bAlternateSetting 4 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x82 EP 2 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x0b20 2x 800 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 2 bAlternateSetting 5 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x82 EP 2 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x1320 3x 800 bytes bInterval 1 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 2 bAlternateSetting 6 bNumEndpoints 1 bInterfaceClass 14 Video bInterfaceSubClass 2 Video Streaming bInterfaceProtocol 0 iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x82 EP 2 IN bmAttributes 5 Transfer Type Isochronous Synch Type Asynchronous Usage Type Data wMaxPacketSize 0x1400 3x 1024 bytes bInterval 1 We can see the effect of this patch in GStreamer: To reproduce this effect we need to launch two streams simultaneously like so: gst-launch-1.0 -e v4l2src device=$H264_DEVICE ! video/x-h264,width=1920,height=1080,framerate=30/1 ! fakesink & gst-launch-1.0 -e v4l2src device=$MJPEG_DEVICE ! image/jpeg,width=1920,height=1080,framerate=30/1 ! jpegdec ! videoconvert ! x264enc tune=zerolatency ! mp4mux ! filesink location=output2.mp4 & Post-patch: Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, smpte170m/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 1861 kb/s, 29.10 fps, 29.58 tbr, 3k tbn (default) Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, smpte170m/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 1880 kb/s, 29.36 fps, 29.58 tbr, 3k tbn (default) Pre-patch: Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, smpte170m/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 1819 kb/s, 29.09 fps, 29.58 tbr, 3k tbn (default) Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, smpte170m/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 1765 kb/s, 27.83 fps, 29.58 tbr, 3k tbn (default) We can see from the results that the second stream (the one with 1 H.264 stream and 1 MJPEG stream, gets lower FPS before the patch, indicating the lost frames). After the patch we get higher FPS (29.36 as opposed to 27.83). Isaac Scott (2): media: uvcvideo: Implement dual stream quirk to fix loss of usb packets media: uvcvideo: Add new quirk definition for the Sonix Technology Co. 292a camera drivers/media/usb/uvc/uvc_driver.c | 9 +++++++++ drivers/media/usb/uvc/uvc_video.c | 27 ++++++++++++++++++++++++++- drivers/media/usb/uvc/uvcvideo.h | 1 + 3 files changed, 36 insertions(+), 1 deletion(-)