RK3288 Debian下OpenCV通过Gstreamer解码RTSP视频流

前言

在RK3288主板Debian 9.13系统上想调用CPU硬解进行网络摄像头视频流进行解码，本来尝试用FFmpeg+Mpp方式进行，但ffmpeg集成mpp的解码器，解码后的格式为AV_PIX_FMT_DRM_PRIME，也就是 DRM 帧数据，要进行图像识别还得通过CPU转码为RGB或BGR格式。按照这个过程下来，光解码CPU的占用率就很高。。。后来在Rockchip的wiki_Mpp上看到，对Mpp有如下的一段说明：

We offer the Gstreamer Rockchip, it is a standard Gstreamer plugin for the hardware decoder and encoder at Rockchip platform. I would suggest all the user in the Linux don’t develop the MPP directly unless you know what you are doing. Choose Gstreamer rocckchip in you convenience.

简而言之就是建议Linux下的用户不要直接使用MPP进行开发，推荐用标准的Gstreamer插件。了解到，原来OpenCV可以集成Gstreamer进行使用。运行Gstreamer的例子发现，RK3288解码4K画面是很流畅。

• Debian GNU/Linux 9.13 (stretch) armv7l
• OpenCV 3.4.11
• Gstreamer 1.10.4

Gstreamer解码RTSP命令

#!/bin/sh
gst-launch-1.0 rtspsrc location=rtsp://192.168.31.163:8554/ ! \
        ! rtph264depay ! h264parse ! mppvideodec ! rkximagesink sync=false

安装需要的依赖包

apt-get update
apt-get install -y libgstreamer-plugins-base1.0-dev \
		libpng16-16 \
        build-essential \
        cmake \
        git \
        pkg-config \
        libjpeg-dev \
		libgtk2.0-dev \
        libv4l-dev \
        libatlas-base-dev \
        gfortran \
        libhdf5-dev \
        libtiff5-dev \
		libtbb-dev \
		libeigen3-dev

编译OpenCV

cd到3.4.11.zip文件目录下，执行如下命令：

unzip 3.4.11.zip # 解压zip压缩包
cd opencv-3.4.11 # 切换到源码包目录
mkdir build && cd build # 创建build目录并切换进去
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D INSTALL_C_EXAMPLES=OFF -D WITH_GSTREAMER=ON -D WITH_GTK_2_X=ON -D WITH_GTHREAD=ON -D WITH_TBB=ON -D WITH_OPENGL=ON  .. # 配置opencv 此处注意有两个点

如果配置成功的话，应该会有如下输出：

--   OpenCV modules:
--     To be built:                 calib3d core dnn features2d flann highgui imgcodecs imgproc ml objdetect photo shape stitching superres ts video videoio videostab
--     Disabled:                    world
--     Disabled by dependency:      -
--     Unavailable:                 cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev java js python2 python3 viz
--     Applications:                tests perf_tests apps
--     Documentation:               NO
--     Non-free algorithms:         NO
-- 
--   GUI: 
--     GTK+:                        YES (ver 2.24.31) # 识别到GTK+ 2.0 如果没有会无法显示
--       GThread :                  YES (ver 2.50.3)
--       GtkGlExt:                  NO
--     OpenGL support:              NO
--     VTK support:                 NO
-- 
--   Media I/O: 
--     ZLib:                        /usr/lib/arm-linux-gnueabihf/libz.so (ver 1.2.8)
--     JPEG:                        /usr/lib/libjpeg.so (ver 62)
--     WEBP:                        build (ver encoder: 0x020f)
--     PNG:                         /usr/lib/arm-linux-gnueabihf/libpng.so (ver 1.6.28)
--     TIFF:                        /usr/lib/arm-linux-gnueabihf/libtiff.so (ver 42 / 4.0.8)
--     JPEG 2000:                   build (ver 1.900.1)
--     OpenEXR:                     build (ver 2.3.0)
--     HDR:                         YES
--     SUNRASTER:                   YES
--     PXM:                         YES
-- 
--   Video I/O:
--     DC1394:                      NO
--     FFMPEG:                      YES # 系统自带了FFMPEG 所以会开启
--       avcodec:                   YES (ver 58.35.100)
--       avformat:                  YES (ver 58.20.100)
--       avutil:                    YES (ver 56.22.100)
--       swscale:                   YES (ver 5.3.100)
--       avresample:                YES (ver 4.0.0)
--     GStreamer:                   YES # 识别到Gstreamer
--       base:                      YES (ver 1.10.4)
--       video:                     YES (ver 1.10.4)
--       app:                       YES (ver 1.10.4)
--       riff:                      YES (ver 1.10.4)
--       pbutils:                   YES (ver 1.10.4)
--     libv4l/libv4l2:              NO
--     v4l/v4l2:                    linux/videodev2.h
-- 
--   Parallel framework:            TBB (ver 4.3 interface 8006)
-- 
--   Trace:                         YES (with Intel ITT)
-- 
--   Other third-party libraries:
--     Lapack:                      NO
--     Eigen:                       YES (ver 3.3.2)
--     Custom HAL:                  NO
--     Protobuf:                    build (3.5.1)
-- 
--   OpenCL:                        YES (no extra features)
--     Include path:                /opencv-3.4.11/3rdparty/include/opencl/1.2
--     Link libraries:              Dynamic load
-- 
--   Python (for build):            /usr/bin/python2.7
-- 
--   Java:                          
--     ant:                         NO
--     JNI:                         NO
--     Java wrappers:               NO
--     Java tests:                  NO
-- 
--   Install to:                    /usr/local
-- -----------------------------------------------------------------
-- 
-- Configuring done
-- Generating done
-- Build files have been written to: */opencv-3.4.11/build

执行如下命令进行编译安装

make -j4 # 编译
make install # 安装opencv库到/usr/local下

配置OpenCV

cd /etc/ld.so.conf.d/ # 切换目录
touch opencv.conf # 新建opencv配置文件
echo /usr/local/lib/ > opencv.conf # 填写opencv编译后库所在的路径
sudo ldconfig # 使配置文件生效

测试代码

#include <iostream>
#include <opencv2/opencv.hpp>
#include <opencv2/imgproc/types_c.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/time.h>
#include <iconv.h>
#include <sstream>
#include <string>
#include <string.h>

using namespace std;
using namespace cv;

int main()
{
    string gsurl = "rtspsrc location=rtsp://192.168.31.163:8554/ latency=0 ! rtph264depay ! h264parse ! mppvideodec ! videoconvert ! video/x-raw,format=(string)BGR  ! appsink sync=false";  // 也可以使用 rgaconvert
    VideoCapture cap = VideoCapture(gsurl,cv::CAP_GSTREAMER);
    if(!cap.isOpened())
    {
        std::cout<<"cannot open captrue..."<<std::endl;
        return 0;
    }

    int fps = cap.get(5);
    cout<<"fps:"<<fps<<endl;
    Mat frame;
    bool readreturn = false;
    while(1)
    {  
        readreturn = cap.read(frame);

        imshow("RTSP",frame);
        if (cvWaitKey(30) == 27) 
        {
            cout << "Esc key is pressed by user" << endl;
            break;
        }
    }

    cap.release();
    return 0;
}

使用如下命令进行编译

g++ main.cpp `pkg-config --cflags --libs opencv`