部署并训练PaddleSeg的pp_liteseg到param，onnx，mnn，openvion，blob，rknn框架

基本思想：利用实例分割方式，选用paddleSeg的pp_liteseg完成项目。

一、 配置环境

需要先自行配置 openvion，cuda, cudnn, tensorRT环境

ubuntu@ubuntu:~$ python -m pip install paddlepaddle-gpu==2.3.2.post111 -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html
ubuntu@ubuntu:~$ git clone https://github.com/PaddlePaddle/PaddleSeg.git
Cloning into 'PaddleSeg'...
remote: Enumerating objects: 21959, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (12/12), done.
Receiving objects:  22% (4856/21959), 45.23 MiB | 757.00 KiB/s 
Receiving objects:  61% (13554/21959), 245.85 MiB | 299.00 KiB/s 
Receiving objects:  61% (13554/21959), 246.70 MiB | 432.00 KiB/s
remote: Total 21959 (delta 0), reused 6 (delta 0), pack-reused 21946
Receiving objects: 100% (21959/21959), 346.46 MiB | 341.00 KiB/s, done.
Resolving deltas: 100% (14409/14409), done.

二、数据集准备

1. 自用数据集，通过labelme制作好标签后，将jpg原图像和json文件放置在同一文件夹即可。

2. 格式转换，将labelme格式转成paddleSeg

ubuntu@ubuntu:~/PaddleSeg$ python tools/data/labelme2seg.py datasets/

├── annotations
│   ├── 000001.png
│   ├── 000002.png
│   ├── 000003.png
├── class_names.txt
├── 000001.jpg
├── 000002.jpg
└── 000003.jpg

3. 整理数据集，需要将以上格式安排整理以下格式

├── annotations
│   ├── 000001.png
│   ├── 000002.png
│   ├── 000003.png
├── class_names.txt
├── images
│   ├── 000001.jpg
│   ├── 000002.jpg
│   ├── 000003.jpg

4. 化分数据集

ubuntu@ubuntu:~/PaddleSeg$ python tools/data/split_dataset_list.py /home/ubuntu/PaddleSeg/af_datasets/ images annotations --split 0.7 0.3 0.1 jpg png

├── annotations
│   ├── 000001.png
│   ├── 000002.png
│   ├── 000003.png
├── class_names.txt
├── images
│   ├── 000001.jpg
│   ├── 000002.jpg
│   ├── 000003.jpg
├── test.txt
├── train.txt
└── val.txt

5. 配置文件使用修改pp_liteseg_stdc1_camvid_960x720_10k.yml

ubuntu@ubuntu:~/PaddleSeg/configs/pp_liteseg$ cp pp_liteseg_stdc1_camvid_960x720_10k.yml pp_liteseg_stdc1_camvid_640x640_10k.yml

batch_size: 6  # total: 4*6
iters: 100000
 
train_dataset:
  type: Dataset
  dataset_root: /home/ubuntu/PaddleSeg/datasets/train
  num_classes: 2 #backgroud+cup
  mode: train
  train_path: /home/ubuntu/PaddleSeg/datasets/train/train.txt
  transforms:
    - type: ResizeStepScaling
      min_scale_factor: 0.5
      max_scale_factor: 2.5
      scale_step_size: 0.25
    - type: RandomPaddingCrop
      crop_size: [300, 300]
    - type: RandomHorizontalFlip
    - type: RandomDistort
      brightness_range: 0.5
      contrast_range: 0.5
      saturation_range: 0.5
    - type: Normalize
 
val_dataset:
  type: Dataset
  dataset_root: /home/ubuntu/PaddleSeg/datasets/val
  num_classes: 2
  mode: val
  val_path: /home/ubuntu/PaddleSeg/datasets/val/val.txt
  transforms:
    - type: Normalize
 
optimizer:
  type: sgd
  momentum: 0.9
  weight_decay: 5.0e-4
 
lr_scheduler:
  type: PolynomialDecay
  learning_rate: 0.01
  end_lr: 0
  power: 0.9
  warmup_iters: 200
  warmup_start_lr: 1.0e-5
 
loss:
  types:
    - type: OhemCrossEntropyLoss
      min_kept: 250000   # batch_size * 300 * 300 // 16
    - type: OhemCrossEntropyLoss
      min_kept: 250000
    - type: OhemCrossEntropyLoss
      min_kept: 250000
  coef: [1, 1, 1]
 
model:
  type: PPLiteSeg
  backbone:
    type: STDC1
    pretrained: https://bj.bcebos.com/paddleseg/dygraph/PP_STDCNet1.tar.gz
  arm_out_chs: [32, 64, 128]
  seg_head_inter_chs: [32, 64, 64]

三、训练模型

1. 开始训练

ubuntu@ubuntu:~/PaddleSeg$ python train.py --config configs/pp_liteseg/pp_liteseg_stdc1_camvid_640x640_10k.yml --do_eval
2022-11-25 16:46:23 [INFO]	
------------Environment Information-------------
platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
Python: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
Paddle compiled with cuda: True
NVCC: Build cuda_11.1.TC455_06.29069683_0
cudnn: 8.2
GPUs used: 1
CUDA_VISIBLE_DEVICES: None
GPU: ['GPU 0: NVIDIA GeForce']
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PaddleSeg: 2.6.0
PaddlePaddle: 2.3.2
OpenCV: 4.6.0
------------------------------------------------
2022-11-25 16:46:23 [INFO]	
---------------Config Information---------------
batch_size: 6
iters: 10000
loss:
  coef:
  - 1
  - 1
  - 1
  types:
2022-11-25 16:54:19 [INFO]	[TRAIN] epoch: 1, iter: 10/10000, loss: 2.7239, lr: 0.000460, batch_cost: 0.2893, reader_cost: 0.01094, ips: 20.7363 samples/sec | ETA 00:48:10
2022-11-25 16:54:19 [INFO]	[TRAIN] epoch: 1, iter: 20/10000, loss: 2.3742, lr: 0.000959, batch_cost: 0.0511, reader_cost: 0.00009, ips: 117.4557 samples/sec | ETA 00:08:29
2022-11-25 16:54:20 [INFO]	[TRAIN] epoch: 1, iter: 30/10000, loss: 1.9726, lr: 0.001459, batch_cost: 0.0536, reader_cost: 0.00026, ips: 111.8903 samples/sec | ETA 00:08:54
2022-11-25 16:54:20 [INFO]	[TRAIN] epoch: 2, iter: 40/10000, loss: 1.7898, lr: 0.001958, batch_cost: 0.0576, reader_cost: 0.00709, ips: 104.1587 samples/sec | ETA 00:09:33
2022-11-25 16:54:21 [INFO]	[TRAIN] epoch: 2, iter: 50/10000, loss: 2.6318, lr: 0.002458, batch_cost: 0.0550, reader_cost: 0.00426, ips: 109.1434 samples/sec | ETA 00:09:06
2022-11-25 16:54:21 [INFO]	[TRAIN] epoch: 2, iter: 60/10000, loss: 2.1906, lr: 0.002957, batch_cost: 0.0566, reader_cost: 0.00435, ips: 106.0024 samples/sec | ETA 00:09:22
2022-11-25 16:54:22 [INFO]	[TRAIN] epoch: 2, iter: 70/10000, loss: 1.9887, lr: 0.003457, batch_cost: 0.0567, reader_cost: 0.00542, ips: 105.8548 samples/sec | ETA 00:09:22
2022-11-25 16:54:23 [INFO]	[TRAIN] epoch: 3, iter: 80/10000, loss: 2.3479, lr: 0.003956, batch_cost: 0.0611, reader_cost: 0.01129, ips: 98.2484 samples/sec | ETA 00:10:05
2022-11-25 16:54:23 [INFO]	[TRAIN] epoch: 3, iter: 90/10000, loss: 2.0537, lr: 0.004456, batch_cost: 0.0551, reader_cost: 0.00373, ips: 108.8724 samples/sec | ETA 00:09:06
2022-11-25 16:54:24 [INFO]	[TRAIN] epoch: 3, iter: 100/10000, loss: 2.0187, lr: 0.004955, batch_cost: 0.0539, reader_cost: 0.00411, ips: 111.2684 samples/sec | ETA 00:08:53
2022-11-25 16:54:24 [INFO]	[TRAIN] epoch: 3, iter: 110/10000, loss: 2.1657, lr: 0.005455, batch_cost: 0.0508, reader_cost: 0.00069, ips: 118.2217 samples/sec | ETA 00:08:21
 

2. 查看模型

ubuntu@ubuntu:~/PaddleSeg/output$ ls
best_model iter_10000  iter_6000  iter_7000  iter_8000  iter_9000 

3. 测试模型

ubuntu@ubuntu:~/PaddleSeg$ python3 predict.py --config /home/ubuntu/PaddleSeg/configs/pp_liteseg/pp_liteseg_stdc1_camvid_640x640_10k.yml --model_path /home/ubuntu/PaddleSeg/output/best_model/model.pdparams --image_path /home/ubuntu/PaddleSeg/datasets/val/JPEGImages/000000002157.jpg

四、转换模型

基本路线 .pdparams 到 .onnx，然后到 .mnn，再到 .openvino， 最后到 .blob

1. onnx转换 

修改 infer_onnx_trt.py，放开以下注释。

model = SavedSegmentationNet(model)  # add argmax to the last layer

执行转换，对比paddle.shape有问题不用在意，因为打开了argmax，直接使用.onnx模型即可

ubuntu@ubuntu:~/PaddleSeg$ python3 deploy/python/infer_onnx_trt.py --config /home/ubuntu/PaddleSeg/configs/pp_liteseg/pp_liteseg_stdc1_camvid_960x720_10k.yml --model_path /home/ubuntu/PaddleSeg/model.pdparams --save_dir ./saved --width 960 --height 720
/home/ubuntu/.local/lib/python3.8/site-packages/scipy/fft/__init__.py:97: DeprecationWarning: The module numpy.dual is deprecated.  Instead of using dual, use the functions directly from numpy or scipy.
  from numpy.dual import register_func
/home/ubuntu/.local/lib/python3.8/site-packages/scipy/sparse/sputils.py:17: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`.
  supported_dtypes = [np.typeDict[x] for x in supported_dtypes]
/home/ubuntu/.local/lib/python3.8/site-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
2022-11-22 14:00:41 [INFO]	Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/PP_STDCNet1.tar.gz
2022-11-22 14:00:41 [INFO]	There are 145/145 variables loaded into STDCNet.
2022-11-22 14:00:41 [INFO]	Loading pretrained model from /home/ubuntu/PaddleSeg/model.pdparams
2022-11-22 14:00:41 [INFO]	There are 247/247 variables loaded into PPLiteSeg.
2022-11-22 14:00:41 [INFO]	Loaded trained params of model successfully
==input shape: 720 960
out shape: (1, 1, 720, 960)
The paddle model has been predicted by PaddlePaddle.
 
2022-11-22 14:00:42 [INFO]	Static PaddlePaddle model saved in ./saved/paddle_model_static_onnx_temp_dir.
[Paddle2ONNX] Start to parse PaddlePaddle model...
[Paddle2ONNX] Model file path: ./saved/paddle_model_static_onnx_temp_dir/model.pdmodel
[Paddle2ONNX] Paramters file path: ./saved/paddle_model_static_onnx_temp_dir/model.pdiparams
[Paddle2ONNX] Start to parsing Paddle model...
[Paddle2ONNX] Use opset_version = 11 for ONNX export.
[Paddle2ONNX] PaddlePaddle model is exported as ONNX format now.
2022-11-22 14:00:43 [INFO]	ONNX model saved in ./saved/pp_liteseg_stdc1_camvid_960x720_10k_model.onnx.
Completed export onnx model.
 
The onnx model has been checked.
The onnx model has been predicted by ONNXRuntime.
(1, 720, 960)
/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, locale, imp, stat
Traceback (most recent call last):
  File "deploy/python/infer_onnx_trt.py", line 491, in <module>
    export_load_infer(args)
  File "deploy/python/infer_onnx_trt.py", line 439, in export_load_infer
    assert onnx_out.shape == paddle_out.shape
AssertionError

2. onnx转mnn模型，转ncnn模型暂时存在很多不支持的算子

ubuntu@ubuntu:~/MNN/build$ ./MNNConvert -f ONNX --modelFile /home/ubuntu/PaddleSeg/saved/pp_liteseg_stdc1_camvid_960x720_10k_model.onnx --MNNModel /home/ubuntu/PaddleSeg/saved/pp_liteseg_stdc1_camvid_960x720_10k_model.mnn --bizCode MNN
Start to Convert Other Model Format To MNN Model...
[14:00:51] /home/ubuntu/MNN/tools/converter/source/onnx/onnxConverter.cpp:40: ONNX Model ir version: 8
Start to Optimize the MNN Net...
inputTensors : [ x, ]
outputTensors: [ argmax_0.tmp_0, ]
Converted Success!

2.1 部署mnn测试使用

cmakelist.txt

cmake_minimum_required(VERSION 3.16)
project(test_mnn)

set(CMAKE_CXX_STANDARD 14)


include_directories(${CMAKE_SOURCE_DIR})
include_directories(${CMAKE_SOURCE_DIR}/include)
include_directories(${CMAKE_SOURCE_DIR}/include/MNN)
find_package(OpenCV REQUIRED)

#添加头文件
include_directories(${OpenCV_INCLUDE_DIRS})
#链接Opencv库
add_library(libmnn SHARED IMPORTED)
set_target_properties(libmnn PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/lib/libMNN.so)

add_executable(test_mnn main.cpp)

target_link_libraries(test_mnn ${OpenCV_LIBS} libmnn)

main.cpp

#include <iostream>
#include <algorithm>
#include <vector>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/opencv.hpp>
#include<MNN/Interpreter.hpp>
#include<MNN/ImageProcess.hpp>

using namespace std;
using namespace cv;
using namespace std::chrono;


int main(int argc, char **argv) {
    cv::VideoCapture capture("/home/ubuntu/inspection_robot/test_mnn/video/B1.mp4");
//    cv::Mat bgr = cv::imread("/home/ubuntu/inspection_robot/test_mnn/image/610.jpg");

    if (!capture.isOpened())
    {
        cout << "视频读取失败，请检查地址！！！" << endl;
        return -1;
    }


    float fps = 0;
    int counter = 0;
    auto startTime = steady_clock::now();

    cv::Mat bgr;

    while (true) {

        capture >> bgr;

        int orignal_width=bgr.cols;
        int orignal_height=bgr.rows;
        int target_width=640;
        int target_height=640;
        cv::Mat resize_img;
        cv::resize(bgr, resize_img, cv::Size(target_width, target_height));
        auto start = chrono::high_resolution_clock::now();    //开始时间

        // MNN inference
        auto mnnNet = std::shared_ptr<MNN::Interpreter>(
                MNN::Interpreter::createFromFile(
                        "/home/ubuntu/inspection_robot/test_mnn/model/pp_liteseg_stdc1_camvid_640x640_10k_model.mnn"));
        auto t1 = std::chrono::steady_clock::now();
        MNN::ScheduleConfig netConfig;
        netConfig.type = MNN_FORWARD_CPU;
        netConfig.numThread = 4;

        auto session = mnnNet->createSession(netConfig);
        auto input = mnnNet->getSessionInput(session, nullptr);

        mnnNet->resizeTensor(input, {1, 3, target_height, target_width});
        mnnNet->resizeSession(session);
        MNN::CV::ImageProcess::Config config;

        const float mean_vals[3] = {255 * 0.5f, 255 * 0.5f, 255 * 0.5f};

        const float norm_255[3] = {1.f / (255 * 0.5), 1.f / (255 * 0.5), 1.f / (255 * 0.5)};

        std::shared_ptr<MNN::CV::ImageProcess> pretreat(
                MNN::CV::ImageProcess::create(MNN::CV::BGR, MNN::CV::RGB, mean_vals, 3,
                                              norm_255, 3));

        pretreat->convert(resize_img.data, target_width, target_height, resize_img.step[0], input);

        mnnNet->runSession(session);

        auto Sessionscores = mnnNet->getSessionOutput(session, "argmax_0.tmp_0");

        MNN::Tensor scoresHost(Sessionscores, Sessionscores->getDimensionType());
        Sessionscores->copyToHostTensor(&scoresHost);

        int w = scoresHost.width();
        int h = scoresHost.height();
        int c = scoresHost.channel();
        int b = scoresHost.batch();
        //printf(" w=%d h=%d c=%d b=%d\n", w, h, c, b);
        std::vector<int> vec_host_scores;
        ///
        w=c;
        h=h;
        c=1;

        //printf("new_w=%d new_h=%d new_c=%d new_b=%d\n", w, h, c, b);
        for (int i = 0; i < scoresHost.elementSize(); i++) {
            vec_host_scores.emplace_back(scoresHost.host<int>()[i]);
        }
        auto end = chrono::high_resolution_clock::now();    //结束时间
        auto duration1 = (end - start).count();
//        cout << "程序运行时间：" << setprecision(10) << duration1 / 1000000000.0 << "s"
//             << "；  " << duration1 / 1000000.0 << "ms"
//             << "；  " << duration1 / 1000.0 << "us"
//             << endl;

        counter ++;
        auto currentTime = steady_clock::now();
        auto elapsed = duration_cast<duration<float>>(currentTime - startTime);
        if (elapsed > seconds(1)) {
            fps = counter / elapsed.count();
            counter = 0;
            startTime = currentTime;
        }

        std::stringstream fpsStr;
        fpsStr << "fps: " << std::fixed << std::setprecision(2) << fps;
         printf("fps %f\n", fps);


        int num_class = 3;
        vector<int> color_map(num_class * 3);
        for (int i = 0; i < num_class; i++) {
            int j = 0;
            int lab = i;
            while (lab) {
                color_map[i * 3] |= ((lab >> 0 & 1) << (7 - j));
                color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j));
                color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j));
                j += 1;
                lab >>= 3;
            }
        }
        cv::Mat pseudo_img(w, h, CV_8UC3, cv::Scalar(0, 0, 0));
        for (int r = 0; r < w; r++) {
            for (int c = 0; c < h; c++) {
                int idx = vec_host_scores[r*h  + c];
                pseudo_img.at<Vec3b>(r, c)[0] = color_map[idx * 3];
                pseudo_img.at<Vec3b>(r, c)[1] = color_map[idx * 3 + 1];
                pseudo_img.at<Vec3b>(r, c)[2] = color_map[idx * 3 + 2];
            }
        }
        cv::Mat result;
        cv::addWeighted(resize_img, 0.4, pseudo_img, 0.6, 0, result, 0);
//        cv::imshow("pseudo_img", pseudo_img);
//        cv::imwrite("pseudo_img.jpg", pseudo_img);
//        cv::imshow("bgr", resize_img);
//        cv::imwrite("resize_img.jpg", resize_img);
        cv::imshow("result", result);

//        cv::imwrite("result.jpg", result);
//        cv::putText(result, fpsStr.str(), cv::Point(10, 20), cv::FONT_HERSHEY_TRIPLEX, 0.7,cv::Scalar(0,0,255));
        cv::resize(result,result,cv::Size(orignal_width,orignal_height));
        int key = cv::waitKey(1);
        if (key == 'q' || key == 27) {
            cv::destroyAllWindows();
            return 0;
        }

        mnnNet->releaseModel();
        mnnNet->releaseSession(session);

    }




    return 0;
}

3. onnx转openvion模型

ubuntu@ubuntu:~$ ubuntu@ubuntu:~$ python /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo.py --input_model /home/ubuntu/PaddleSeg/saved/pp_liteseg_stdc1_camvid_640x640_10k_model.onnx --output_dir /home/ubuntu/PaddleSeg/saved/FP16 --input_shape [1,3,720,960] --data_type FP16 --scale_values [127.5,127.5,127.5] --mean_values [127.5,127.5,127.5]
 
[ SUCCESS ] Generated IR version 10 model.
[ SUCCESS ] XML file: /home/ubuntu/PaddleSeg/saved/FP16/pp_liteseg_stdc1_camvid_640x640_10k_model.xml
[ SUCCESS ] BIN file: /home/ubuntu/PaddleSeg/saved/FP16/pp_liteseg_stdc1_camvid_640x640_10k_model.bin
[ SUCCESS ] Total execution time: 8.82 seconds. 
[ SUCCESS ] Memory consumed: 243 MB. 
It's been a while, check for a new version of Intel(R) Distribution of OpenVINO(TM) toolkit here https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit/download.html?cid=other&source=prod&campid=ww_2021_bu_IOTG_OpenVINO-2021-4-LTS&content=upg_all&medium=organic or on the GitHub*

4. openvion转blob模型

ubuntu@ubuntu:/opt/intel/openvino_2021/deployment_tools/tools/compile_tool$  ./compile_tool -m /home/ubuntu/PaddleSeg/saved/FP16/pp_liteseg_stdc1_camvid_640x640_10k_model.xml -ip U8 -d MYRIAD -VPU_NUMBER_OF_SHAVES 4 -VPU_NUMBER_OF_CMX_SLICES 4
Inference Engine: 
	IE version ......... 2021.4.1
	Build ........... 2021.4.1-3926-14e67d86634-releases/2021/4
 
Network inputs:
    x : U8 / NCHW
Network outputs:
    argmax_0.tmp_0 : I32 / CHW
[Warning][VPU][Config] Deprecated option was used : VPU_MYRIAD_PLATFORM
Done. LoadNetwork time elapsed: 6655 ms
 
ubuntu@ubuntu:/opt/intel/openvino_2021/deployment_tools/tools/compile_tool$ cp pp_liteseg_stdc1_camvid_960x720_10k_model.blob /home/ubuntu/PaddleSeg/saved/FP16
 

4.1 部署oak测试使用

cmakelist.txt

cmake_minimum_required(VERSION 3.16)
project(test_oak)

set(CMAKE_CXX_STANDARD 14)


find_package(OpenCV REQUIRED)
#message(STATUS ${OpenCV_INCLUDE_DIRS})
#添加头文件
include_directories(${OpenCV_INCLUDE_DIRS})
include_directories(${CMAKE_SOURCE_DIR}/include)
include_directories(${CMAKE_SOURCE_DIR}/include/utility)
#链接Opencv库
find_package(depthai CONFIG REQUIRED)

aux_source_directory(./src SRC_LIST) # 搜索当前目录下的所有.cpp文件，存储在变量SRC_LIST中
add_executable(test_oak main.cpp include/utility/utility.cpp)

target_link_libraries(test_oak ${OpenCV_LIBS} depthai::opencv)

main.cpp

//#include <stdio.h>
//#include <string>
//#include <iostream>
//#include <opencv2/core/core.hpp>
//#include <opencv2/highgui/highgui.hpp>
//#include <opencv2/imgproc/imgproc.hpp>
//
//#include "utility.hpp"
//#include <vector>
//#include "depthai/depthai.hpp"
//using namespace std;
//using namespace std::chrono;
//using namespace cv;
//int post_process(std::vector<int> vec_host_scores,cv::Mat resize_img,cv::Mat &result, vector<int> color_map,int w,int h){
//
//    cv::Mat pseudo_img(w, h, CV_8UC3, cv::Scalar(0, 0, 0));
//    for (int r = 0; r < w; r++) {
//        for (int c = 0; c < h; c++) {
//            int idx = vec_host_scores[r*h  + c];
//            pseudo_img.at<Vec3b>(r, c)[0] = color_map[idx * 3];
//            pseudo_img.at<Vec3b>(r, c)[1] = color_map[idx * 3 + 1];
//            pseudo_img.at<Vec3b>(r, c)[2] = color_map[idx * 3 + 2];
//        }
//    }
//
//    cv::addWeighted(resize_img, 0.4, pseudo_img, 0.6, 0, result, 0);
//    //cv::imshow("pseudo_img", pseudo_img);
    cv::imwrite(".pseudo_img.jpg", pseudo_img);
//     //cv::imshow("bgr", resize_img);
    cv::imwrite("resize_img.jpg", resize_img);
//    //cv::imshow("result", result);
    cv::imwrite("result.jpg", result);
//    //cv::waitKey(0);
//    return 0;
//}
//
//
//int main(int argc, char **argv) {
//    int num_class = 256;
//    vector<int> color_map(num_class * 3);
//    for (int i = 0; i < num_class; i++) {
//        int j = 0;
//        int lab = i;
//        while (lab) {
//            color_map[i * 3] |= ((lab >> 0 & 1) << (7 - j));
//            color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j));
//            color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j));
//            j += 1;
//            lab >>= 3;
//        }
//    }
//    int target_width=640;
//    int target_height=640;
//    dai::Pipeline pipeline;
//    //定义
//    auto cam = pipeline.create<dai::node::XLinkIn>();
//    cam->setStreamName("inFrame");
//    auto net = pipeline.create<dai::node::NeuralNetwork>();
//    dai::OpenVINO::Blob blob("/home/ubuntu/inspection_robot/test_oak/model/pp_liteseg_stdc1_camvid_640x640_10k_model.blob");
//    net->setBlob(blob);
//    net->input.setBlocking(false);
//
//    //基本熟练明白oak的函数使用了　
//    cam->out.link(net->input);
//
//
//
//    //定义输出
//    auto xlinkParserOut = pipeline.create<dai::node::XLinkOut>();
//    xlinkParserOut->setStreamName("parseOut");
//    auto xlinkoutOut = pipeline.create<dai::node::XLinkOut>();
//    xlinkoutOut->setStreamName("out");
//
//    auto xlinkoutpassthroughOut = pipeline.create<dai::node::XLinkOut>();
//
//    xlinkoutpassthroughOut->setStreamName("passthrough");
//
//
//    net->out.link(xlinkParserOut->input);
//
//    net->passthrough.link(xlinkoutpassthroughOut->input);
//
//    //结构推送相机
//    dai::Device device(pipeline);
//    //取帧显示
//    auto inqueue = device.getInputQueue("inFrame");//maxsize 代表缓冲数据
//    auto detqueue = device.getOutputQueue("parseOut", 8, false);//maxsize 代表缓冲数据
//
//    bool printOutputLayersOnce=true;
//
//    cv::VideoCapture capture("/home/ubuntu/inspection_robot/test_mnn/video/B1.mp4");
//    cv::Mat frame;
    cv::Mat frame=cv::imread("/home/ubuntu/inspection_robot/test_oak/images/630.jpg");
//    while(true) {
//
//        capture >> frame;
//
//        if(frame.empty()) break;
//
//        auto img = std::make_shared<dai::ImgFrame>();
//        frame = resizeKeepAspectRatio(frame, cv::Size(target_height, target_width), cv::Scalar(0));
//        toPlanar(frame, img->getData());
//        img->setTimestamp(steady_clock::now());
//        img->setWidth(target_height);
//        img->setHeight(target_width);
//        inqueue->send(img);
//
//        auto inNN = detqueue->get<dai::NNData>();
//        if( printOutputLayersOnce&&inNN) {
//            std::cout << "Output layer names: ";
//            for(const auto& ten : inNN->getAllLayerNames()) {
//                std::cout << ten << ", ";
//            }
//            std::cout << std::endl;
//            printOutputLayersOnce = false;
//        }
//        cv::Mat result;
//        auto pred=inNN->getLayerInt32(inNN->getAllLayerNames()[0]);
//
//        post_process(pred,frame,result,color_map,target_width,target_height);
//        //cv::imshow("demo", frame);
//        cv::imshow("result", result);
//        //cv::imwrite("result.jpg",result);
//        int key = cv::waitKey(1);
//        if(key == 'q' || key == 'Q') return 0;
//    }
//
    while (true) {


        auto ImgFrame = outqueue->get<dai::ImgFrame>();
        auto frame = ImgFrame->getCvFrame();

        auto inNN = detqueue->get<dai::NNData>();
        if( printOutputLayersOnce&&inNN) {
            std::cout << "Output layer names: ";
            for(const auto& ten : inNN->getAllLayerNames()) {
                std::cout << ten << ", ";
            }
            std::cout << std::endl;
            printOutputLayersOnce = false;
        }
        cv::Mat result;
        auto pred=inNN->getLayerInt32(inNN->getAllLayerNames()[0]);

        post_process(pred,frame,result,color_map,target_width,target_height);
        cv::imshow("demo", frame);
        cv::imshow("result", result);
        cv::imwrite("result.jpg",result);
        cv::waitKey(1);


    }
//
//
//    return 0;
//}



#include <stdio.h>
#include <string>
#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

#include "utility.hpp"
#include <vector>
#include "depthai/depthai.hpp"

using namespace std;
using namespace std::chrono;
using namespace cv;

int post_process(std::vector<int> vec_host_scores, cv::Mat resize_img, cv::Mat &result,
                 vector<int> color_map, int w, int h) {
    cv::Mat pseudo_img(w, h, CV_8UC3, cv::Scalar(0, 0, 0));
    for (int r = 0; r < w; r++) {
        for (int c = 0; c < h; c++) {
            int idx = vec_host_scores[r * h + c];
            pseudo_img.at<Vec3b>(r, c)[0] = color_map[idx * 3];
            pseudo_img.at<Vec3b>(r, c)[1] = color_map[idx * 3 + 1];
            pseudo_img.at<Vec3b>(r, c)[2] = color_map[idx * 3 + 2];
        }
    }

    cv::addWeighted(resize_img, 0.4, pseudo_img, 0.6, 0, result, 0);
//    cv::imshow("pseudo_img", pseudo_img);
//    cv::imwrite(".pseudo_img.jpg", pseudo_img);
//    cv::imshow("bgr", resize_img);
//    cv::imwrite("resize_img.jpg", resize_img);
//    cv::imshow("result", result);
//    cv::imwrite("result.jpg", result);
//    cv::waitKey(0);
    return 0;
}


int main(int argc, char **argv) {
    int num_class = 256;
    vector<int> color_map(num_class * 3);
    for (int i = 0; i < num_class; i++) {
        int j = 0;
        int lab = i;
        while (lab) {
            color_map[i * 3] |= ((lab >> 0 & 1) << (7 - j));
            color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j));
            color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j));
            j += 1;
            lab >>= 3;
        }
    }
    int target_width = 640;
    int target_height = 640;
    dai::Pipeline pipeline;
    //定义
    auto cam = pipeline.create<dai::node::ColorCamera>();
    cam->setBoardSocket(dai::CameraBoardSocket::RGB);
    cam->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);
    cam->setPreviewSize(target_height, target_width);  // NN input
    cam->setInterleaved(false);

    auto net = pipeline.create<dai::node::NeuralNetwork>();
    dai::OpenVINO::Blob blob("/home/ubuntu/inspection_robot/test_oak/model/pp_liteseg_optic_disc_640x640_1k_model.blob");
    net->setBlob(blob);
    net->input.setBlocking(false);

    //基本熟练明白oak的函数使用了　
    cam->preview.link(net->input);

    //定义输出
    auto xlinkParserOut = pipeline.create<dai::node::XLinkOut>();
    xlinkParserOut->setStreamName("parseOut");
    auto xlinkoutOut = pipeline.create<dai::node::XLinkOut>();
    xlinkoutOut->setStreamName("out");

    auto xlinkoutpassthroughOut = pipeline.create<dai::node::XLinkOut>();

    xlinkoutpassthroughOut->setStreamName("passthrough");

    net->out.link(xlinkParserOut->input);

    net->passthrough.link(xlinkoutpassthroughOut->input);

    //结构推送相机
    dai::Device device(pipeline);
    //取帧显示
    auto outqueue = device.getOutputQueue("passthrough", 8, false);//maxsize 代表缓冲数据
    auto detqueue = device.getOutputQueue("parseOut", 8, false);//maxsize 代表缓冲数据

    bool printOutputLayersOnce = true;
    auto startTime = steady_clock::now();
    int counter = 0;
    float fps = 0;
    while (true) {

        auto ImgFrame = outqueue->get<dai::ImgFrame>();
        auto frame = ImgFrame->getCvFrame();

        auto inNN = detqueue->get<dai::NNData>();
        if (printOutputLayersOnce && inNN) {
//            std::cout << "Output layer names: ";
//            for (const auto &ten : inNN->getAllLayerNames()) {
//                std::cout << ten << ", ";
//            }
//            std::cout << std::endl;
            printOutputLayersOnce = false;
        }
        cv::Mat result;
        auto pred = inNN->getLayerInt32(inNN->getAllLayerNames()[0]);

        post_process(pred, frame, result, color_map, target_width, target_height);

        counter++;
        auto currentTime = steady_clock::now();
        auto elapsed = duration_cast<duration<float>>(currentTime - startTime);
        if (elapsed > seconds(1)) {
            fps = counter / elapsed.count();
            counter = 0;
            startTime = currentTime;
        }
        std::stringstream fpsStr;
        fpsStr << "NN fps: " << std::fixed << std::setprecision(2) << fps;
        cv::putText(result, fpsStr.str(), cv::Point(2, result.rows - 4), cv::FONT_HERSHEY_TRIPLEX, 0.4, cv::Scalar(0,255,0));
        //cv::imshow("demo", frame);
        cv::imshow("result", result);
        //cv::imwrite("result.jpg", result);
        int key = cv::waitKey(1);
        if (key == 'q' || key == 27) {
            cv::destroyAllWindows();
            return 0;
        }
    }

    return 0;
}

换上测距代码

#include <stdio.h>
#include <string>
#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

#include "utility.hpp"
#include <vector>
#include "depthai/depthai.hpp"

using namespace std;
using namespace std::chrono;
using namespace cv;

dai::Pipeline pipeline;

float target_width = 640;
float target_height = 640;
dai::Point2f topLeft;
dai::Point2f bottomRight;
dai::SpatialLocationCalculatorConfigData config;

static std::atomic<bool> newConfig{false};



int find_bound(cv::Mat gray_img, cv::Mat resize_img, vector<Rect> &ploy_rects_) {
    cvtColor(gray_img, gray_img, cv::COLOR_BGR2GRAY);
    std::vector<std::vector<cv::Point>> contours;
    findContours(gray_img, contours, cv::RETR_TREE, cv::CHAIN_APPROX_SIMPLE);

    vector<vector<Point>> contours_ploy(contours.size()); // 逼近多边形点
    vector<Rect> ploy_rects(contours.size());             // 多边形框
    for (size_t i = 0; i < contours.size(); i++) {
        approxPolyDP(Mat(contours[i]), contours_ploy[i], 3, true);
        ploy_rects[i] = boundingRect(contours_ploy[i]);
//        std::cout << ploy_rects.size() << std::endl;
    }

    RNG rng(1234);
    Point2f pts[4];
    for (size_t t = 0; t < contours.size(); t++) {
        Scalar color = Scalar(rng.uniform(0, 255), rng.uniform(0, 255), rng.uniform(0, 255));
        rectangle(resize_img, ploy_rects[t], color, 2, 8);
        if (contours_ploy[t].size() > 5) {
            for (int r = 0; r < 4; r++) {
                line(resize_img, pts[r], pts[(r + 1) % 4], color, 2, 8);
            }
        }
    }
    cv::drawContours(resize_img, contours, -1, (255, 0, 0), 2);
//    cv::imshow("resize_img", resize_img);
    ploy_rects_ = ploy_rects;
//    imshow("drawImg", resize_img);
//    cv::imwrite("dram.jpg",resize_img);

}

int post_process(std::vector<int> vec_host_scores, cv::Mat resize_img, cv::Mat &result, vector<int> color_map,
                 int w, int h, std::vector<Rect> &ploy_rects_) {
    cv::Mat pseudo_img(w, h, CV_8UC3, cv::Scalar(0, 0, 0));
    for (int r = 0; r < w; r++) {
        for (int c = 0; c < h; c++) {
            int idx = vec_host_scores[r * h + c];
            pseudo_img.at<Vec3b>(r, c)[0] = color_map[idx * 3];
            pseudo_img.at<Vec3b>(r, c)[1] = color_map[idx * 3 + 1];
            pseudo_img.at<Vec3b>(r, c)[2] = color_map[idx * 3 + 2];
        }
    }
//    cv::imshow("pseudo_img", pseudo_img);
    cv::addWeighted(resize_img, 0.4, pseudo_img, 0.6, 0, result, 0);
    find_bound(pseudo_img, resize_img, ploy_rects_);
    // cv::imshow("pseudo_img", pseudo_img);
    // cv::imwrite(".pseudo_img.jpg", pseudo_img);
    // cv::imshow("bgr", resize_img);
    // cv::imwrite("resize_img.jpg", resize_img);
    // cv::imshow("result", result);
    // cv::imwrite("result.jpg", result);
    // cv::waitKey(0);
    return 0;
}


int initial(const std::string& nnPath) {
    float fps = 50;
    // 定义rgb相机
    auto camRgb = pipeline.create<dai::node::ColorCamera>();
    camRgb->setBoardSocket(dai::CameraBoardSocket::RGB);
    camRgb->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);
    camRgb->setPreviewSize((int) target_height, (int) target_width);  // NN input
    camRgb->setInterleaved(false);
    camRgb->setPreviewKeepAspectRatio(false);

    // 定义网络推理
    auto net = pipeline.create<dai::node::NeuralNetwork>();
    dai::OpenVINO::Blob blob(nnPath);
    net->setBlob(blob);
    net->input.setBlocking(false);
    camRgb->preview.link(net->input);

    // 定义mono相机
    auto monoLeft = pipeline.create<dai::node::MonoCamera>();
    auto monoRight = pipeline.create<dai::node::MonoCamera>();
    monoLeft->setResolution(dai::MonoCameraProperties::SensorResolution::THE_720_P);
    monoLeft->setBoardSocket(dai::CameraBoardSocket::LEFT);
    monoRight->setResolution(dai::MonoCameraProperties::SensorResolution::THE_720_P);
    monoRight->setBoardSocket(dai::CameraBoardSocket::RIGHT);


    // 定义空间计算节点
    auto stereo = pipeline.create<dai::node::StereoDepth>();
    auto spatialDataCalculator = pipeline.create<dai::node::SpatialLocationCalculator>();
    auto xoutDepth = pipeline.create<dai::node::XLinkOut>();
    auto xoutSpatialData = pipeline.create<dai::node::XLinkOut>();
    auto xinSpatialCalcConfig = pipeline.create<dai::node::XLinkIn>();
    xoutDepth->setStreamName("depth");
    xoutSpatialData->setStreamName("spatialData");
    xinSpatialCalcConfig->setStreamName("spatialCalcConfig");

    stereo->setLeftRightCheck(true);
    stereo->setExtendedDisparity(true);
    stereo->setDepthAlign(dai::CameraBoardSocket::RGB);
    stereo->setDefaultProfilePreset(dai::node::StereoDepth::PresetMode::HIGH_ACCURACY);

//    dai::Point2f topLeft(0.4f, 0.4f);
//    dai::Point2f bottomRight(0.6f, 0.6f);


    config.depthThresholds.lowerThreshold = 100;
    config.depthThresholds.upperThreshold = 1000;
    config.roi = dai::Rect(topLeft, bottomRight);

    spatialDataCalculator->inputConfig.setWaitForMessage(false);
    spatialDataCalculator->initialConfig.addROI(config);

    // Linking
    monoLeft->out.link(stereo->left);
    monoRight->out.link(stereo->right);

    //定义输出
    auto xlinkParserOut = pipeline.create<dai::node::XLinkOut>();
    xlinkParserOut->setStreamName("parserOut");
    auto xlinkoutOut = pipeline.create<dai::node::XLinkOut>();
    xlinkoutOut->setStreamName("out");
    auto xlinkoutpassthroughOut = pipeline.create<dai::node::XLinkOut>();
    xlinkoutpassthroughOut->setStreamName("passthrough");

    spatialDataCalculator->passthroughDepth.link(xoutDepth->input);
    stereo->depth.link(spatialDataCalculator->inputDepth);

    spatialDataCalculator->out.link(xoutSpatialData->input);
    xinSpatialCalcConfig->out.link(spatialDataCalculator->inputConfig);

    net->out.link(xlinkParserOut->input);

    net->passthrough.link(xlinkoutpassthroughOut->input);


    return 0;
}

int main(int argc, char **argv) {
    int num_class = 256;
    vector<int> color_map(num_class * 3);
    for (int i = 0; i < num_class; i++) {
        int j = 0;
        int lab = i;
        while (lab) {
            color_map[i * 3] |= ((lab >> 0 & 1) << (7 - j));
            color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j));
            color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j));
            j += 1;
            lab >>= 3;
        }
    }

    std::string nnPath("../model/pp_liteseg_stdc1_camvid_640x640_10k_model.blob");
    int ok_init = initial(nnPath);
    if (ok_init != 0) printf("Oak init failed");

    // 结构推送相机
    dai::Device device;

    // 红外激光投影仪、红外泛光照明器
    device.setIrLaserDotProjectorBrightness(1000);
    device.setIrFloodLightBrightness(0);
    device.startPipeline(pipeline);


    // 取帧显示
    auto outqueue = device.getOutputQueue("passthrough", 4, false);//maxsize 代表缓冲数据
    auto detqueue = device.getOutputQueue("parserOut", 4, false);//maxsize 代表缓冲数据
    auto depthQueue = device.getOutputQueue("depth", 4, false);
    auto spatialCalcQueue = device.getOutputQueue("spatialData", 4, false);
    auto spatialCalcConfigInQueue = device.getInputQueue("spatialCalcConfig");

    bool printOutputLayersOnce = true;
    auto startTime = steady_clock::now();
    int counter = 0;
    float fps = 0;
    auto color = cv::Scalar(255, 255, 255);

    while (true) {
        auto inDepth = depthQueue->get<dai::ImgFrame>();
        auto ImgFrame = outqueue->get<dai::ImgFrame>();

        auto frame = ImgFrame->getCvFrame();
        target_width = frame.cols * 1.0;
        target_height = frame.rows * 1.0;

        auto inNN = detqueue->get<dai::NNData>();
        if (printOutputLayersOnce && inNN) {
//            std::cout << "Output layer names: ";
//            for (const auto &ten : inNN->getAllLayerNames()) {
//                std::cout << ten << ", ";
//            }
//            std::cout << std::endl;
            printOutputLayersOnce = false;
        }
        cv::Mat result;
        auto pred = inNN->getLayerInt32(inNN->getAllLayerNames()[0]);
        std::vector<Rect> ploy_rects_;
        post_process(pred, frame, result, color_map, target_width, target_height, ploy_rects_);

        for (auto &item:ploy_rects_) {
            newConfig = true;
            cv::Mat depthFrame = inDepth->getFrame();  // depthFrame values are in millimeters
//            std::cout << depthFrame.rows << " " << depthFrame.cols << " " << std::endl;
            cv::Mat depthFrameColor;

            cv::normalize(depthFrame, depthFrameColor, 255, 0, cv::NORM_INF, CV_8UC1);
            cv::equalizeHist(depthFrameColor, depthFrameColor);
            cv::applyColorMap(depthFrameColor, depthFrameColor, cv::COLORMAP_HOT);

            topLeft.x = item.x * depthFrame.cols / target_width / depthFrame.cols;
            topLeft.y = item.y * depthFrame.rows / target_height / depthFrame.rows;
            bottomRight.x = (item.x * depthFrame.cols / target_width + item.width * depthFrame.cols / target_width) /
                            depthFrame.cols;
            bottomRight.y = (item.y * depthFrame.rows / target_height + item.height * depthFrame.rows / target_height) /
                            depthFrame.rows;

            auto spatialData = spatialCalcQueue->get<dai::SpatialLocationCalculatorData>()->getSpatialLocations();
            for (const auto& depthData : spatialData) {
                auto roi = depthData.config.roi;
                roi = roi.denormalize(depthFrameColor.cols, depthFrameColor.rows);
                auto xmin = (int) roi.topLeft().x;
                auto ymin = (int) roi.topLeft().y;
                auto xmax = (int) roi.bottomRight().x;
                auto ymax = (int) roi.bottomRight().y;

                auto depthMin = depthData.depthMin;
                auto depthMax = depthData.depthMax;

                cv::rectangle(result, cv::Rect(cv::Point((int) item.x, (int) item.y),
                                               cv::Point((int) item.x + (int) item.width,
                                                         (int) item.y + (int) item.height)), color,
                              cv::FONT_HERSHEY_SIMPLEX);
                std::stringstream depthX;
                depthX << "X: " << (int) depthData.spatialCoordinates.x << " mm";
                cv::putText(result, depthX.str(), cv::Point((int) item.x + 10, (int) item.y + 20),
                            cv::FONT_HERSHEY_TRIPLEX, 0.5, color);
                std::stringstream depthY;
                depthY << "Y: " << (int) depthData.spatialCoordinates.y << " mm";
                cv::putText(result, depthY.str(), cv::Point((int) item.x + 10, (int) item.y + 35),
                            cv::FONT_HERSHEY_TRIPLEX, 0.5, color);
                std::stringstream depthZ;
                depthZ << "Z: " << (int) depthData.spatialCoordinates.z << " mm";
                cv::putText(result, depthZ.str(), cv::Point((int) item.x + 10, (int) item.y + 50),
                            cv::FONT_HERSHEY_TRIPLEX, 0.5, color);

                cv::rectangle(result, cv::Rect(cv::Point((int) item.x, (int) item.y),
                                               cv::Point((int) item.x + (int) item.width,
                                                         (int) item.y + (int) item.height)), color,
                              cv::FONT_HERSHEY_SIMPLEX);
                auto coords = depthData.spatialCoordinates;
                auto distance = std::sqrt(coords.x * coords.x + coords.y * coords.y + coords.z * coords.z);
                std::stringstream depthDistance;
                depthDistance.precision(2);
                depthDistance << fixed << static_cast<float>(distance / 1000.0f) << "m";
                auto fontType = cv::FONT_HERSHEY_TRIPLEX;
                cv::putText(result, depthDistance.str(), cv::Point(xmin + 10, ymin + 70), fontType, 0.5, color);


                cv::rectangle(depthFrameColor, cv::Rect(cv::Point(xmin, ymin), cv::Point(xmax, ymax)), color,
                              cv::FONT_HERSHEY_SIMPLEX);
                depthX << "X: " << (int) depthData.spatialCoordinates.x << " mm";
                cv::putText(depthFrameColor, depthX.str(), cv::Point(xmin + 10, ymin + 20), cv::FONT_HERSHEY_TRIPLEX,
                            0.5, color);

                depthY << "Y: " << (int) depthData.spatialCoordinates.y << " mm";
                cv::putText(depthFrameColor, depthY.str(), cv::Point(xmin + 10, ymin + 35), cv::FONT_HERSHEY_TRIPLEX,
                            0.5, color);

                depthZ << "Z: " << (int) depthData.spatialCoordinates.z << " mm";
                cv::putText(depthFrameColor, depthZ.str(), cv::Point(xmin + 10, ymin + 50), cv::FONT_HERSHEY_TRIPLEX,
                            0.5, color);
//                cv::imshow("depthFrameColor", depthFrameColor);

            }
            if (newConfig) {
                config.roi = dai::Rect(topLeft, bottomRight);
                dai::SpatialLocationCalculatorConfig cfg;
                cfg.addROI(config);
                spatialCalcConfigInQueue->send(cfg);
                newConfig = false;
            }
        }
        counter++;
        auto currentTime = steady_clock::now();
        auto elapsed = duration_cast<duration<float>>(currentTime - startTime);
        if (elapsed > seconds(1)) {
            fps = counter / elapsed.count();
            counter = 0;
            startTime = currentTime;
        }
        std::stringstream fpsStr;
        fpsStr << "NN fps: " << std::fixed << std::setprecision(2) << fps;
        cv::putText(result, fpsStr.str(), cv::Point(2, result.rows - 4), cv::FONT_HERSHEY_TRIPLEX, 0.4,
                    cv::Scalar(0, 255, 0));

        cv::imshow("result", result);
        int key = cv::waitKey(1);
        if (key == 'q' || key == 27) {
            cv::destroyAllWindows();
            return 0;
        }
    }

    return 0;
}

5. onnx转rknn模型，部署rk3399 pro使用

dataset.txt （新建txt，放一张图片路径即可）

test.jpg

onnx2rknn.py

from rknn.api import RKNN
 
ONNX_MODEL = '/home/ubuntu/PaddleSeg/saved/pp_liteseg_stdc1_camvid_640x640_10k_model.onnx'
RKNN_MODEL = '/home/ubuntu/PaddleSeg/saved/pp_liteseg_stdc1_camvid_640x640_10k_model.rknn'
 
if __name__ == '__main__':
 
    # Create RKNN object
    rknn = RKNN(verbose=True)
 
    # pre-process config
    print('--> config model')
    rknn.config(mean_values=[[127.5, 127.5, 127.5]], std_values=[[127.5, 127.5, 127.5]], reorder_channel='0 1 2',
                target_platform='rk3399pro',
                quantized_dtype='asymmetric_affine-u8', optimization_level=3, output_optimize=1)
    print('done')
 
    print('--> Loading model')
    ret = rknn.load_onnx(model=ONNX_MODEL)
    if ret != 0:
        print('Load model  failed!')
        exit(ret)
    print('done')
 
    # Build model
    print('--> Building model')
    ret = rknn.build(do_quantization=True, dataset='dataset.txt')  # ,pre_compile=True
    if ret != 0:
        print('Build pp_liteseg_stdc1_camvid_640x640_10k_model failed!')
        exit(ret)
    print('done')
 
    # Export rknn model
    print('--> Export RKNN model')
    ret = rknn.export_rknn(RKNN_MODEL)
    if ret != 0:
        print('Export pp_liteseg_stdc1_camvid_640x640_10k_model.rknn failed!')
        exit(ret)
    print('done')
 
    rknn.release()

转换成功！！

D Packing Conv_p2o.Conv.8_47 ...
D Quantize @Conv_p2o.Conv.8_47:bias to asymmetric_affine.
D Quantize @Conv_p2o.Conv.8_47:weight to asymmetric_affine.
D Packing Conv_p2o.Conv.9_48 ...
D Quantize @Conv_p2o.Conv.9_48:bias to asymmetric_affine.
D Quantize @Conv_p2o.Conv.9_48:weight to asymmetric_affine.
D Packing Mul_p2o.Mul.1_109_Add_p2o.Add.6_99 ...
D Packing Mul_p2o.Mul.5_54_Add_p2o.Add.9_45 ...
D Packing Mul_p2o.Mul.9_20_Add_p2o.Add.12_16 ...
D Disable rknn op statistic.
D output tensor id = 0, name = ArgMax_p2o.ArgMax.0/out0_0
D input tensor id = 1, name = x_195
I Build config finished.
done
--> Export RKNN model
done

python测试脚本

import os
import urllib
import traceback
import time
import sys
import warnings
 
import numpy as np
import cv2
from rknn.api import RKNN
 
RKNN_MODEL = "/home/ubuntu/PaddleSeg/saved/pp_liteseg_stdc1_camvid_960x720_10k_model.rknn"
IMG_PATH = "/home/ubuntu/PycharmProjects/untitled3/0001TP_007980.png"
 
QUANTIZE_ON = True
 
def visualize(image, result, color_map, save_dir=None, weight=0.6):
    """
    Convert predict result to color image, and save added image.
    Args:
        image (str): The path of origin image.
        result (np.ndarray): The predict result of image.
        color_map (list): The color used to save the prediction results.
        save_dir (str): The directory for saving visual image. Default: None.
        weight (float): The image weight of visual image, and the result weight is (1 - weight). Default: 0.6
    Returns:
        vis_result (np.ndarray): If `save_dir` is None, return the visualized result.
    """
 
    color_map = [color_map[i:i + 3] for i in range(0, len(color_map), 3)]
    color_map = np.array(color_map).astype("uint8")
    # Use OpenCV LUT for color mapping
    c1 = cv2.LUT(result, color_map[:, 0])
    c2 = cv2.LUT(result, color_map[:, 1])
    c3 = cv2.LUT(result, color_map[:, 2])
    pseudo_img = np.dstack((c3, c2, c1))
 
    im = image
    vis_result = cv2.addWeighted(im, weight, pseudo_img, 1 - weight, 0)
 
    if save_dir is not None:
        if not os.path.exists(save_dir):
            os.makedirs(save_dir)
        image_name = os.path.split(image)[-1]
        out_path = os.path.join(save_dir, image_name)
        cv2.imwrite(out_path, vis_result)
    else:
        return vis_result
 
def get_color_map_list(num_classes, custom_color=None):
    """
    Returns the color map for visualizing the segmentation mask,
    which can support arbitrary number of classes.
    Args:
        num_classes (int): Number of classes.
        custom_color (list, optional): Save images with a custom color map. Default: None, use paddleseg's default color map.
    Returns:
        (list). The color map.
    """
 
    num_classes += 1
    color_map = num_classes * [0, 0, 0]
    for i in range(0, num_classes):
        j = 0
        lab = i
        while lab:
            color_map[i * 3] |= (((lab >> 0) & 1) << (7 - j))
            color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j))
            color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j))
            j += 1
            lab >>= 3
    color_map = color_map[3:]
 
    if custom_color:
        color_map[:len(custom_color)] = custom_color
    return color_map
 
 
 
if __name__ == "__main__":
    class_num=256
    color_map=get_color_map_list(class_num)
    # Create RKNN object
    rknn = RKNN()
 
    if not os.path.exists(RKNN_MODEL):
        print("model not exist")
        exit(-1)
 
    # Load ONNX model
    print("--> Loading model")
    ret = rknn.load_rknn(RKNN_MODEL)
    if ret != 0:
        print("Load rknn model failed!")
        exit(ret)
    print("done")
    # init runtime environment
    print("--> Init runtime environment")
    ret = rknn.init_runtime()
    if ret != 0:
        print("Init runtime environment failed")
        exit(ret)
    print("done")
    image_size = (960, 720)
    src_img = cv2.imread(IMG_PATH)
    resize_img=cv2.resize(src_img,image_size)
    color_img = cv2.cvtColor(resize_img, cv2.COLOR_BGR2RGB)  # hwc rgb
    print("--> Running model")
    start = time.clock()
    pred = rknn.inference(inputs=[color_img])
    # 获取结束时间
    end = time.clock()
    # 计算运行时间
    runTime = end - start
    runTime_ms = runTime * 1000
    # 输出运行时间
    print("运行时间：", runTime_ms, "毫秒")
    pred = np.squeeze(pred).astype('uint8')
    print(pred)
    added_image =visualize(resize_img, pred, color_map, None,weight=0.6)
    cv2.imshow("added",added_image)
    cv2.imwrite("add.jpg",added_image)
    cv2.waitKey(0)
    rknn.release()

c++　rk3399 pro测试

cmakelist.txt

cmake_minimum_required(VERSION 3.16)
project(untitled10)
set(CMAKE_CXX_FLAGS "-std=c++11")
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fopenmp ")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fopenmp")
 
include_directories(${CMAKE_SOURCE_DIR})
include_directories(${CMAKE_SOURCE_DIR}/include)
find_package(OpenCV REQUIRED)
#message(STATUS ${OpenCV_INCLUDE_DIRS})
#添加头文件
include_directories(${OpenCV_INCLUDE_DIRS})
#链接Opencv库
add_library(librknn_api SHARED IMPORTED)
set_target_properties(librknn_api PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/lib64/librknn_api.so)
 
 
add_executable(untitled10 main.cpp)
target_link_libraries(untitled10 ${OpenCV_LIBS} librknn_api )

mian.cpp

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <queue>
#include "rknn_api.h"
#include "opencv2/core/core.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/highgui/highgui.hpp"
#include <chrono>
using namespace cv;
using namespace std;
void printRKNNTensor(rknn_tensor_attr *attr) {
    printf("index=%d name=%s n_dims=%d dims=[%d %d %d %d] n_elems=%d size=%d "
           "fmt=%d type=%d qnt_type=%d fl=%d zp=%d scale=%f\n",
           attr->index, attr->name, attr->n_dims, attr->dims[3], attr->dims[2],
           attr->dims[1], attr->dims[0], attr->n_elems, attr->size, 0, attr->type,
           attr->qnt_type, attr->fl, attr->zp, attr->scale);
}
 
int post_process_u8(float *input0,cv::Mat resize_img,int w,int h){
    std::vector<int> vec_host_scores;
    for(int i=0;i<w*h;i++){
        vec_host_scores.emplace_back(input0[i]);
    }
 
    int num_class = 256;//提取到外面　只执行一次即可，自己改吧
    vector<int> color_map(num_class * 3);
    for (int i = 0; i < num_class; i++) {
        int j = 0;
        int lab = i;
        while (lab) {
            color_map[i * 3] |= ((lab >> 0 & 1) << (7 - j));
            color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j));
            color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j));
            j += 1;
            lab >>= 3;
        }
    }
    cv::Mat pseudo_img(w, h, CV_8UC3, cv::Scalar(0, 0, 0));
    for (int r = 0; r < w; r++) {
        for (int c = 0; c < h; c++) {
            int idx = vec_host_scores[r*h  + c];
            pseudo_img.at<Vec3b>(r, c)[0] = color_map[idx * 3];
            pseudo_img.at<Vec3b>(r, c)[1] = color_map[idx * 3 + 1];
            pseudo_img.at<Vec3b>(r, c)[2] = color_map[idx * 3 + 2];
        }
    }
    cv::Mat result;
    cv::addWeighted(resize_img, 0.4, pseudo_img, 0.6, 0, result, 0);
    //cv::imshow("pseudo_img", pseudo_img);
    cv::imwrite(".pseudo_img.jpg", pseudo_img);
   // cv::imshow("bgr", resize_img);
    cv::imwrite("resize_img.jpg", resize_img);
    //cv::imshow("result", result);
    cv::imwrite("result.jpg", result);
     //cv::waitKey(0);
    return 0;
}
 
int main(int argc, char **argv) {
    const char *img_path = "../0001TP_007980.png";
    const char *model_path = "../pp_liteseg_stdc1_camvid_960x720_10k_model.rknn";
    const char *post_process_type = "fp";//fp
    const int target_width = 960;
    const int target_height = 720;
 
    // Load image
    cv::Mat bgr = cv::imread(img_path);
    if (!bgr.data) {
        printf("cv::imread %s fail!\n", img_path);
        return -1;
    }
    cv::Mat rgb;
    //BGR->RGB
    cv::cvtColor(bgr, rgb, cv::COLOR_BGR2RGB);
 
    cv::Mat img_resize;
    cv::resize(rgb,img_resize,cv::Size(target_width,target_height));
    int width=bgr.cols;
    int height=bgr.rows;
 
 
    // Load model
    FILE *fp = fopen(model_path, "rb");
    if (fp == NULL) {
        printf("fopen %s fail!\n", model_path);
        return -1;
    }
    fseek(fp, 0, SEEK_END);
    int model_len = ftell(fp);
    void *model = malloc(model_len);
    fseek(fp, 0, SEEK_SET);
    if (model_len != fread(model, 1, model_len, fp)) {
        printf("fread %s fail!\n", model_path);
        free(model);
        return -1;
    }
 
 
    rknn_context ctx = 0;
 
    int ret = rknn_init(&ctx, model, model_len, 0);
    if (ret < 0) {
        printf("rknn_init fail! ret=%d\n", ret);
        return -1;
    }
 
    /* Query sdk version */
    rknn_sdk_version version;
    ret = rknn_query(ctx, RKNN_QUERY_SDK_VERSION, &version,
                     sizeof(rknn_sdk_version));
    if (ret < 0) {
        printf("rknn_init error ret=%d\n", ret);
        return -1;
    }
    printf("sdk version: %s driver version: %s\n", version.api_version,
           version.drv_version);
 
 
    /* Get input,output attr */
    rknn_input_output_num io_num;
    ret = rknn_query(ctx, RKNN_QUERY_IN_OUT_NUM, &io_num, sizeof(io_num));
    if (ret < 0) {
        printf("rknn_init error ret=%d\n", ret);
        return -1;
    }
    printf("model input num: %d, output num: %d\n", io_num.n_input,
           io_num.n_output);
 
    rknn_tensor_attr input_attrs[io_num.n_input];
    memset(input_attrs, 0, sizeof(input_attrs));
    for (int i = 0; i < io_num.n_input; i++) {
        input_attrs[i].index = i;
        ret = rknn_query(ctx, RKNN_QUERY_INPUT_ATTR, &(input_attrs[i]),
                         sizeof(rknn_tensor_attr));
        if (ret < 0) {
            printf("rknn_init error ret=%d\n", ret);
            return -1;
        }
        printRKNNTensor(&(input_attrs[i]));
    }
 
    rknn_tensor_attr output_attrs[io_num.n_output];
    memset(output_attrs, 0, sizeof(output_attrs));
    for (int i = 0; i < io_num.n_output; i++) {
        output_attrs[i].index = i;
        ret = rknn_query(ctx, RKNN_QUERY_OUTPUT_ATTR, &(output_attrs[i]),
                         sizeof(rknn_tensor_attr));
        printRKNNTensor(&(output_attrs[i]));
    }
 
    int input_channel = 3;
    int input_width = 0;
    int input_height = 0;
    if (input_attrs[0].fmt == RKNN_TENSOR_NCHW) {
        printf("model is NCHW input fmt\n");
        input_width = input_attrs[0].dims[0];
        input_height = input_attrs[0].dims[1];
        printf("input_width=%d input_height=%d\n", input_width, input_height);
    } else {
        printf("model is NHWC input fmt\n");
        input_width = input_attrs[0].dims[1];
        input_height = input_attrs[0].dims[2];
        printf("input_width=%d input_height=%d\n", input_width, input_height);
    }
 
    printf("model input height=%d, width=%d, channel=%d\n", input_height, input_width,
           input_channel);
 
 
/* Init input tensor */
    rknn_input inputs[1];
    memset(inputs, 0, sizeof(inputs));
    inputs[0].index = 0;
    inputs[0].buf = img_resize.data;
    inputs[0].type = RKNN_TENSOR_UINT8;
    inputs[0].size = input_width * input_height * input_channel;
    inputs[0].fmt = RKNN_TENSOR_NHWC;
    inputs[0].pass_through = 0;
 
    /* Init output tensor */
    rknn_output outputs[io_num.n_output];
    memset(outputs, 0, sizeof(outputs));
    for (int i = 0; i < io_num.n_output; i++) {
            outputs[i].want_float = 1;
    }
    printf("img.cols: %d, img.rows: %d\n", img_resize.cols, img_resize.rows);
    auto t1=std::chrono::steady_clock::now();
    rknn_inputs_set(ctx, io_num.n_input, inputs);
    ret = rknn_run(ctx, NULL);
    if (ret < 0) {
        printf("ctx error ret=%d\n", ret);
        return -1;
    }
    ret = rknn_outputs_get(ctx, io_num.n_output, outputs, NULL);
    if (ret < 0) {
        printf("outputs error ret=%d\n", ret);
        return -1;
    }
    /* Post process */
    std::vector<float> out_scales;
    std::vector<uint8_t> out_zps;
    for (int i = 0; i < io_num.n_output; ++i) {
        out_scales.push_back(output_attrs[i].scale);
        out_zps.push_back(output_attrs[i].zp);
    }
 
    if (strcmp(post_process_type, "fp") == 0) {
        post_process_u8((float *) outputs[0].buf,img_resize,
                        target_height, target_width);
    }
//毫秒级
    auto t2=std::chrono::steady_clock::now();
    double dr_ms=std::chrono::duration<double,std::milli>(t2-t1).count();
    printf("%lf ms\n",dr_ms);
 
 
 
 
 
    ret = rknn_outputs_release(ctx, io_num.n_output, outputs);
 
    if (ret < 0) {
        printf("rknn_query fail! ret=%d\n", ret);
        goto Error;
    }
 
 
    Error:
    if (ctx > 0)
        rknn_destroy(ctx);
    if (model)
        free(model);
    if (fp)
        fclose(fp);
    return 0;
}