V4L2架构分析（一）

原创已于 2023-03-30 19:30:17 修改 · 2.2k 阅读

17 ·

CC 4.0 BY-SA版权

文章标签：

#linux #V4L2

于 2021-07-13 15:16:01 首次发布

嵌入式系统同时被 3 个专栏收录

354 篇文章

订阅专栏

v4l2

11 篇文章

订阅专栏

USB

8 篇文章

订阅专栏

本文详细介绍了V4L2架构，包括V4L2_device的重要地位，video_device的结构，以及V4L2的ioctl、framebuffer管理和pipeline控制。文章通过实例探讨了V4L2_subdev_call的实现，验证了ISP与Parser通道连接的顺序敏感性，并揭示了V4L2设计模式的应用。

V4L2架构图：

用到的IOCTL如下：

VIDIOC_QUERYCAP 查询设备的属性
VIDIOC_ENUM_FMT 帧格式
VIDIOC_S_FMT 设置视频帧格式，对应struct v4l2_format
VIDIOC_G_FMT 获取视频帧格式等
VIDIOC_REQBUFS 请求/申请若干个帧缓冲区，一般为不少于3个
VIDIOC_QUERYBUF 查询帧缓冲区在内核空间的长度和偏移量
VIDIOC_QBUF 将申请到的帧缓冲区全部放入视频采集输出队列
VIDIOC_STREAMON 开始视频流数据的采集
VIDIOC_DQBUF 应用程序从视频采集输出队列中取出已含有采集数据的帧缓冲区
VIDIOC_STREAMOFF 应用程序将该帧缓冲区重新挂入输入队列

2.struct v4l2_device在v4l2_framework中具有超然的地位，实际上，它应该叫做v4l2_root更能体现它的位置.在一个实现中仅此一个，没有副本，一般是代表设备的的对象，比如 struct platform_device, uvc device 或者PCI device. 中嵌入定义，如下图所示：

平台设备注册：

uvc中的注册：

它的作用包括：

管理一堆的sub-devices列表
notify callback for sub-devices.
release callback for sub-devices.

注册流程有两种：

第一种：直接在设备探测的时候调用,通过平台接口调用v4l2_device_register/v4l2_device_register_subdev_nodes直接注册:

usb_interface_probe->uvc_driver(struct usb_driver)->uvc_probe->v4l2_device_register/v4l2_device_register_subdev_nodes->........

或者全志的vin驱动也是类似的做法：

platform_driver_probe->vin_probe->v4l2_device_register->.....

第二种通过struct v4l2_async_notifier_operations的complete注册

platform_device probe->v4l2_async_notifier_register -> .... > v4l2_device_register_subdev_nodes->complete->v4l2_device_register_subdev_nodes->......

uvc_driver其实就是usb_driver.

3.另一个重要结构 struct video_device：

表示video/vbi/ratio/v4l2_subdev device node in /dev.

以struct v4l2_file_operations 为文件操作表

以v4l2_ioctl_ops为ioctl operations.

通常也表示存在一个dma engine,这个dma engine通过vb2_queue 管理buffer.

4.关于V4L2 struct v4l2_ioctl_info v4l2_ioctls[]函数中的套壳定义.

4.从面向对象继承的角度看待V4L2的设备模型:

5：VB2帧队列管理模型：

6.V4L2框架的调用架构:

架构调用细节：

v4l2_subdev_call是一个宏，它的实现很有意思，隐藏了很多的东西：

v4l2_subdev_call是Linux原生定义的结构，而vin_pipeline_call就不同了，他是我们自定义的一个调用，目的是控制整个管线上的所有资源

7.V4L2的PipeLine拓扑图

关于这幅拓扑图，我做了一些试验简单进行了验证：

实验一，验证ISP和 Parser0链接的全连接性:

修改函数__vin_pipeline_s_stream代码，添加打印：

由于只添加了打印，测试用例功能正常，输入如下：

msh />sample_virvi2vo
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
__vin_pipeline_s_stream line 683, total_rx_ch 1, i 0, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 0.
__vin_pipeline_s_stream line 690, id 0, vipp_sel 0, isp_sel 0, isp_tx_ch 0.
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
msh />

可以看到,total_rx_ch只有1个，也就是说方案里面固定死了这个链接。

现在根据实现，修改isp_input数组的，修改ISP到paser0 channel的链接

原来是0,1,2,3的，修改为1,1,2,3，表示isp0直接和parser0的channel1链接。

编译测试失败，功能已经不正常了:

msh />sample_virvi2vo
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
__vin_pipeline_s_stream line 683, total_rx_ch 1, i 0, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 1.
__vin_pipeline_s_stream line 690, id 0, vipp_sel 0, isp_sel 0, isp_tx_ch 0.
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
events.c                       E  <events_loop:172>         [ISP_ERR]events_loop, line: 172,isp0 event select timeout

video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
videoInputHw.c                 E  <VideoInputHw_CapThread:4245> vipp[0]: timeout for [10]s when vipp GetFrame!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
videoInputHw.c                 E  <VideoInputHw_CapThread:4245> vipp[0]: timeout for [20]s when vipp GetFrame!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
msh />

疑问，基于全连接的猜测，既然可以全连接，那我随便让ISP连接Parser0的任何一个channel应该都可以，为什么跳过channel 0直接连接channel 1会失败呢？

带着疑问，继续修改程序，先把isp_input数组还原回原来的样子，直接修改设置channel连接的个数，从1个改成4个：paser0的四个channel都和isp连接一遍，最后确定连接第三个。

编译测试，功能正常:

msh />sample_virvi2vo
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
__vin_pipeline_s_stream line 684, total_rx_ch 1, i 0, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 0.
__vin_pipeline_s_stream line 684, total_rx_ch 1, i 1, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 1.
__vin_pipeline_s_stream line 684, total_rx_ch 1, i 2, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 2.
__vin_pipeline_s_stream line 684, total_rx_ch 1, i 3, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 3.
__vin_pipeline_s_stream line 691, id 0, vipp_sel 0, isp_sel 0, isp_tx_ch 0.
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
msh />

咦！直接设置和paser0 channel1连接不行，依次设置和0，1，2，3连接，最终敲定连接3却可以，莫非这个连接和设置顺序有关？比如只能从低channel开始设置，逐步递增？验证一下猜测：

在上面试验的基础上，别的不改，只修改第一个paser几个channel的顺序，循环一下:

从0，1，2，3变为3,0,1,2，重新编译看效果：

测试发现功能异常：

msh />sample_virvi2vo
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
__vin_pipeline_s_stream line 684, total_rx_ch 1, i 0, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 3.
__vin_pipeline_s_stream line 684, total_rx_ch 1, i 1, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 1.
__vin_pipeline_s_stream line 684, total_rx_ch 1, i 2, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 2.
__vin_pipeline_s_stream line 684, total_rx_ch 1, i 3, id 0, isp_sel 0, csi_sel 0.
csic_isp_input_select line 232, i 3.
__vin_pipeline_s_stream line 691, id 0, vipp_sel 0, isp_sel 0, isp_tx_ch 0.
[DBG]: [esFSYS_fread:0579]:                                 EOF of regular file.
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
events.c                       E  <events_loop:172>         [ISP_ERR]events_loop, line: 172,isp0 event select timeout

video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
videoInputHw.c                 E  <VideoInputHw_CapThread:4245> vipp[0]: timeout for [10]s when vipp GetFrame!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
videoInputHw.c                 E  <VideoInputHw_CapThread:4245> vipp[0]: timeout for [20]s when vipp GetFrame!
video.c                        E  <video_wait_buffer:354>   [ISP_ERR]video_wait_buffer, line: 354,video0 select timeout!
videoInputHw.c                 E  <VideoInputHw_CapThread:4238> fatal error! vipp[0] get frame fail, but all frames are release!
msh />

这个试验一定程度上验证了猜测，确实和paser channel的连接顺序有关，很有可能只能从低channel到高channel号逐级连接，不能直接连接高channel.

V4L2架构：