公开讲座系列 | 4D毫米波成像雷达的目标分类技术方法研究 讲座(白杰教授) 英文版笔记

本文为毫米波雷达点云的公开讲座笔记(英文版)，讲座信息如下：

• 题目：4D毫米波成像雷达的目标分类技术
• 主讲人：白杰 苏州豪米波技术有限公司董事长、浙大城市学院教授、国家高层次人才
• 会议(论坛)：2021 破壁，第九届汽车与环境创新论坛、第十三届全球汽车产业峰会
• 视频链接：4D毫米波成像雷达的目标分类技术方法研究 - 知乎
  https://www.zhihu.com/zvideo/1455660060366516224

---

内容在优快云、知乎和微信公众号同步更新

• 优快云博客
• 知乎
• 微信公众号
• qq邮箱: 981591477

• 本笔记Markdown源文件、中文版笔记均暂未公开，如有需要可联系邮箱
• 笔记难免存在问题，欢迎联系邮箱指正

---

---

1 Technical background of millimeter wave radar

• MM wave radar is a key sensor (in the multi-sensor fusion system) of assisted and driverless vehicles, which can be divided into, including

  • short-range radar SRR
  • medium range radar MRR
  • long range radar LRR

• Vision Sensors in Autonomous Driving (As shown below)

  • Among them, both MRR and LRR have short-range mode

  • i.e., SRR is implicit in MRR and LRR

  • Is it possible to combine SRR MRR LRR together?

    ❌ Insufficient technology, too high cost, difficult to productize

• Disadvantages of existing 2D radar point clouds:
  • Contains only position and speed information
  • Unable to distinguish target class
  • Low angular resolution

Why 4D mmWave Imaging Radar?

• The bottleneck of mainstream sensors in the market

  • If using a single sensor:

    ❌ Light changes are complex, the impact of bad weather

    ❌ The moving speed of the object is difficult to obtain directly

    ❌ Multi-target separation detection and tracking

    ❌ Easy to misdetect, miss detection, etc.

  • If using multi-sensor fusion (laser, mmWave, image)

    ❌ More development cost and complexity

    ❌ Varying performance of different types of sensors $\Rightarrow$ The fusion confidence is difficult to control and the arbitration is difficult arbitration 仲裁

    ❌ Viewpoint: Arbitration performance can not be improved $\Rightarrow$ multi-sensor is not as good as a single sensor $\Rightarrow$ Some manufacturers choose to use only Camera

---

PS: Why is there no problem with camera + millimeter wave multi-sensor fusion when making ADAS systems?

• 高级辅助驾驶系统（英语：Advanced Driver Assistance Systems，缩写成ADAS）是辅助进行汽车行驶及泊车的系统。当系统中含有人机交互接口时，它可以增加车辆安全和道路安全。

  • 车载导航系统，通常由GPS和TMC来提供实时交通信息。
  • 自适应巡航控制系统
  • 车道偏离警示系统
  • 自动变道系统
  • 防撞警示系统
  • 行人侦测

• Reason: ADAS and unmanned systems have different performance requirements, and there are essential differences

  • Unmanned System: Reliable, No Missing detection
  • ADAS: The driver is responsible and can miss some detections

---

• Advantages of 4D mmWave Radar
  • Completely changed the bottleneck problems such as the inability of traditional millimeter wave radar to give obstacle height information and object classification information
  • E.g., Separating objects by height (How to tell the difference between the viaducts and cars in the pic)

4D Millimeter Wave Imaging Radar Research Trends

• 4D Millimeter-Wave Imaging Radar Research and Development in Foreign Countries

  • The traditional radar supplier giants have been stepping up the research and development of a new generation of 4D mm wave radar products since 2020
  • 大陆集团
  • 安波福
  • 采埃孚
  • 博世

• Features of 4D mmWave Imaging Radar

  • The new generation of 4D millimeter wave imaging radar is not only to achieve 4D point cloud output

    🚩 Object classification + detection + tracking will also be done based on its point cloud images using neural networks

  • Chip cascade

    🚩 Implement dense point cloud and target detection + use radar internal DL algorithm to complete target classification

• Vision + 4D millimeter wave imaging radar can achieve raw data level fusion

  • 3D real-time modeling of the surrounding complex environment
  • Accurate detection of pedestrian, vehicle speed and distance
  • 4D millimeter wave radar makes up for the performance degradation of vision under heavy rain, dark sky and backlight
  • Vision + 4D millimeter wave imaging radar：Superior performance and reasonable price

Existing visualization results

• 16-Line LiDAR VS. 4D Millimeter-Wave Radar

• The difference:
  • mm wave: Data composed of multiple reflections of radio waves
  • LiDAR: Directly reflected data
  • $\Rightarrow$ millimeter wave radar target detection to be similar to lidar target detection, but there are also differences

2 Research on target classification of 4D imaging radar based on Suzhou Haomi wave

Main categories of traffic participants

• motor vehicle

  • Coaches
  • small car
  • motorcycle

• non-motor vehicle

  • bike

• pedestrian

• Classification standard: China’s traffic technical specifications and urban traffic status

4D radar point cloud visualization results for typical targets

• Some visualization results of five types of targets in 4D millimeter wave imaging radar point cloud

• Lidar point cloud

• Millimeter wave 4D point cloud object classification is difficult
  • Difficulty distinguishing between bicycles and motorcycles
  • (Because the angular resolution is not large enough?)

Classification experiment – data set production

• Produced 10,000 point cloud samples
  • 2000 samples per target
  • 1000 static, 1000 dynamic

Classification Experiments – Manual Feature Extraction Methods

Traffic participants Classification Based on 3D RADAR Point Clouds

• Classification algorithm flow based on manual feature extraction method

• Processing flow:

  • 1 Obtain a 3D directed bounding box
  • 2 Extract bounding box shape features, point cloud distribution density features, Doppler velocity features, and reflection intensity features
  • 3 Use point cloud feature set to filter out 20 feature vectors
  • 4 Use the extracted 20 feature vectors to complete the classification

• The experimental results on the five categories:

  • Conclusion 1: Dynamic object classification results are significantly better than static objects

    ✅ Due to Doppler velocity information

  • Conclusion 2: Bicycles and motorcycles are more likely to be confused

    ✅ The shape and Doppler velocity characteristics are relatively close

• Four classification experimental results

  • After merging motorcycles and bicycles, the classification accuracy is significantly improved (above 95%)

Classification Experiments – Deep Learning Methods

Radar Transformer: An Object Classification Network Based on 4DMMW Imaging Radar

• Transformer-based radar point cloud classification
  • Attention mechanism
• Classification results:
  • 5-category results: comparable (or even better) to the previous 4-category results

3 Summary & Outlook

• Suzhou Haomi wave radar products and planning
  • 24GHz
  • 77GHz
  • 79GHz
  • 2022: mass production