Domain Adaptive Object Detection for Autonomous Driving under FoggyWeather（翻）

Title：Domain Adaptive Object Detection for Autonomous Driving under FoggyWeather

雾天环境下自动驾驶领域自适应目标检测

Abstract：Most object detection methods for autonomous drivingusually assume a consistent feature distribution betweentraining and testing data,which is not always the case whenweathers differ significantly.The object detection modeltrained under clear weather might be not effective enoughon the foggy weather because of the domain gap.This paperproposes a novel domain adaptive object detection frame-work for autonomous driving under foggy weather.Ourmethod leverages both image-level and object-level adap-tation to diminish the domain discrepancy in image styleand object appearance.To further enhance the model’scapabilities under challenging samples,we also come upwith a new adversarial gradient reversal layer to performadversarial mining for the hard examples together with do-main adaptation.Moreover,we propose to generate anauxiliary domain by data augmentation to enforce a newdomain-level metric regularization.Experimental resultson public benchmarks show the effectiveness and accu-racy of the proposed method.The code is available athttps://github.com/jinlong17/DA-Detect.

摘要：大多数用于自动驾驶的目标检测方法通常假设训练和测试数据之间具有一致的特征分布，当天气差异较大时，情况并不总是如此。在晴朗天气下训练的目标检测模型在雾天天气下可能会因为域间隙而不够有效。本文提出了一种新的雾天环境下自动驾驶领域自适应目标检测框架。我们的方法同时利用图像级和对象级自适应来减少图像风格和对象外观的领域差异。为了进一步增强模型在挑战样本下的能力，我们还提出了一个新的对抗梯度反转层，结合域适应对困难样本进行对抗挖掘。此外，我们提出通过数据增强来生成辅助域，以执行新的域级度量正则化。在公共基准测试集上的实验结果表明了所提方法的有效性和准确性。代码可在https://github.com/jinlong17/DA- Detect获取。

1.Introduction：

Autonomous driving has wide applications for intelli-gent transportation systems,such as improving the efficiencyin the automatic 24/7 working manner,reducing the laborcosts,enhancing the comfortableness of customers,and soon[23,51].With the computer vision and artificial intel-ligence techniques,object detection plays a critical role inautonomous driving to understand the surrounding drivingscenarios[54,59].In some cases,the autonomous vehiclemight work in the complex residential and industry areas.The diverse weather conditions might make the object de-tection in these environments more difficult.For example,the usages of heating,gas,coal,and vehicle emissions inresidential and industry areas might be possible to generatemore frequent foggy or hazy weather,leading to a significant challenge to the object detection system installed on theautonomous vehicle.

1介绍：

无人驾驶在智能交通系统中有着广泛的应用，如提高自动24 / 7工作方式的效率、降低人工成本、增强客户的舒适性等。随着计算机视觉和人工智能技术的发展，目标检测在自动驾驶中发挥着至关重要的作用，以了解周围的驾驶场景。在某些情况下，自动驾驶汽车可能工作在复杂的居民区和工业区。多样的天气状况可能使得这些环境中的目标检测更加困难。例如，在居民区和工业区使用暖气、煤气、煤和车辆排放可能会产生更频繁的雾天或雾霾天气，这对安装在自动驾驶车辆上的目标检测系统提出了重大挑战。

Many deep learning models such as Faster R-CNN[37],YOLO[36]have demonstrated great success in autonomousdriving.However,most of these well-known methods as-sume that the feature distributions of training and testing dataare homogeneous.Such an assumption may fail when takingthe real-world diverse weather conditions into account[40].For example,as shown in Fig.1,the Faster R-CNN modeltrained on the clear-weather data(source domain)is capableof detecting objects accurately under good weather,but itsperformance drops significantly when it comes to the foggyweather(target domain).This degradation is caused by thefeature domain gap between divergent weather conditions,as the model is unfamiliar with the feature distribution onthe target domain,while the detection performance could beimproved under the foggy weather with domain adaptation.

许多深度学习模型如Faster R- CNN [ 37 ]、YOLO [ 36 ]等在自动驾驶领域取得了巨大的成功。然而，这些著名的方法大多假设训练和测试数据的特征分布是均匀的。当考虑到现实世界中不同的天气条件时，这种假设可能会失败[ 40 ]。例如，如图1所示，在晴朗天气数据(源域)上训练的Faster R - CNN模型能够在好天气下准确检测目标，但在有雾天气(目标域)上其性能明显下降。这种退化是由不同天气条件下的特征域差距造成的，因为模型不熟悉目标域上的特征分布，而在有雾天气下通过域自适应可以提高检测性能。

Domain adaptation,as a technique of transfer learning,isto reduce the domain shift between various weathers.Thispaper proposes a novel domain adaptation framework toachieve robust object detection performance in autonomousdriving under foggy weather.As manually annotating im-ages under adverse weathers is usually time-consuming,our design follows an unsupervised fashion same as thatin[5,26,43],where clear-weather images(source domain)are well labeled and foggy weather images(target domain)have no annotations.Inspired by[5,15],our method lever-ages both image-level and object-level adaptation to diminishthe domain discrepancy in image style and object appear-ance jointly,which is realized by involving image-level andobject-level domain classifiers to enable our convolutionalneural networks generating domain-invariant latent featurerepresentations.Specifically,the domain classifiers aim tomaximize the probability of distinguishing the features pro-duced by different domains,whereas the detection modelexpects to generate the domain-invariant features to confusethe classifiers.

域适应作为迁移学习的一种技术，是为了减少不同天气之间的域偏移。本文提出了一种新颖的域自适应框架，以实现雾天环境下自动驾驶的鲁棒目标检测性能。由于在恶劣天气下人工标注图像通常很耗时，我们的设计遵循了与[ 5、26、43]相同的无监督方式，其中晴朗天气图像(源域)有很好的标注，有雾天气图像(目标域)没有标注。受[ 5、15]的启发，我们的方法同时利用图像级和对象级的自适应来共同减少图像风格和对象外观的域差异，这是由图像级和对象级域分类器使我们的卷积神经网络生成域不变的潜在特征表示来实现的。具体来说，领域分类器旨在最大化区分不同领域产生的特征的概率，而检测模型则期望生成领域不变的特征来混淆分类器。

This paper also addresses two critical insights that areignored by previous domain adaptation methods[5,9,15,26,61]:1)Different training samples might have different challenging levels to be fully harnessed during the transferlearning,while existing works usually omit such diversity;2)Previous domain adaptation methods only consider thesource domain and target domain for transfer learning,whilethe domain-level feature metric distance to the third relateddomain might be neglected.However,embedding the min-ing for hard examples and involving an extra related domainmight potentially further enhance the model’s robust learningcapabilities,which has not been carefully explored before.To emphasize these two insights,we propose a new Ad-versarial Gradient Reversal Layer(AdvGRL)and generatean auxiliary domain by data augmentation.The AdvGRLperforms adversarial mining for the hard examples to en-hance the model learning on the challenging scenarios,andthe auxiliary domain enforces a new domain-level metricregularization during the transfer learning.Experimentalresults on the public benchmarks Cityscapes[7]and FoggyCityscapes[40]show the effectiveness of each proposedcomponent and the superior object detection performanceover the baseline and comparison methods.Overall,thecontributions of this paper are summarized as follows:

本文还提出了两个被先前的领域自适应方法[ 5、9、15、26、61]忽略的关键见解：1 )不同的训练样本在迁移学习过程中可能有不同的挑战水平需要充分利用，而现有的工作通常忽略了这种多样性；2 )以往的域适应方法仅考虑源域和目标域进行迁移学习，而域级特征度量到第三个相关域的距离可能被忽略。然而，嵌入对困难示例的挖掘和涉及额外的相关域可能会进一步增强模型的鲁棒学习能力，而这在以前还没有仔细研究过。为了强调这两个见解，我们提出了一个新的对抗梯度反转层( Ad-versarial Gradient Reversal Layer，AdvGRL )，并通过数据增强生成一个辅助域。AdvGRL对困难样本进行对抗挖掘以增强模型在挑战性场景下的学习，辅助域在迁移学习过程中执行新的域级度量正则化。在公共基准测试集Cityscapes [ 7 ]和FoggyCityscapes [ 40 ]上的实验结果表明了本文提出的每个组件的有效性以及优于基准和对比方法的目标检测性能。总体而言，本文的贡献总结如下：

•We propose a novel deep transfer learning baseddomain adaptive object detection framework for au-tonomous driving under foggy weather,including theimage-level and object-level adaptations,which istrained with labeled clear-weather data and unlabeledfoggy-weather data to enhance the generalization abilityof the deep learning based object detection model.

•We propose a new Adversarial Gradient Reversal Layer(AdvGRL)to perform adversarial mining for the hardexamples together with the domain adaptation to furtherenhance the model’s transfer learning capabilities underchallenging samples.

•We propose a new domain-level metric regularizationduring the transfer learning.By generating an auxiliarydomain with data augmentation,the domain-level met-ric constraint between source domain,auxiliary domain,and target domain is ensured as regularization duringthe transfer learning.

• 我们提出了一种新的基于深度迁移学习的雾天自动驾驶领域自适应目标检测框架，包括图像级和目标级自适应，分别使用有标记的晴空数据和无标记的雾天数据进行训练，以增强基于深度学习的目标检测模型的泛化能力。

• 我们提出了一种新的对抗梯度翻转层( Adversarial Gradient Reversal Layer，AdvGRL )对困难样本进行对抗挖掘，并结合领域自适应进一步提升模型在挑战样本下的迁移学习能力。

• 我们在迁移学习过程中提出了一种新的领域级度量正则化。通过生成带有数据增强的辅助域，在迁移学习过程中保证源域、辅助域和目标域之间的域级度量约束为正则化。

 

2.Related Work

2.1.Object detection for autonomous driving

Recent advancement in deep learning has brought out-standing progress in autonomous driving[6,25,33,53],andobject detection has been one of the most active topic underthis field[8,41,45,59].Regarding the network architecture,current object detection algorithms can be roughly split intotwo categories:two-stage methods and single-stage methods.Two-stage object detection algorithms typically composeof two processes:1)region proposal,2)object classifica-tion and localization refinement.R-CNN[14]is the firstwork for this kind of methods,it applies selective search forregional proposals and independent CNNs for each objectprediction.Fast R-CNN[13]improves R-CNN by obtainingobject features from the shared fe