MAPTR：在线矢量化高精地图构建的结构化建模与学习(2208)

MAPTR: STRUCTURED MODELING AND LEARNING FOR ONLINE VECTORIZED HD MAP CONSTRUCTION

MAPTR：在线矢量化高精地图构建的结构化建模与学习

ABSTRACT

High-definition (HD) map provides abundant and precise environmental information of the driving scene, serving as a fundamental and indispensable component for planning in autonomous driving system. We present MapTR, a structured end-to-end Transformer for efficient online vectorized HD map construction. We propose a unified permutation-equivalent modeling approach, i.e., modeling map element as a point set with a group of equivalent permutations, which accurately describes the shape of map element and stabilizes the learning process. We design a hierarchical query embedding scheme to flexibly encode structured map information and perform hierarchical bipartite matching for map element learning. MapTR achieves the best performance and efficiency with only camera input among existing vectorized map construction approaches on nuScenes dataset. In particular, MapTR-nano runs at real-time inference speed (25:1 FPS) on RTX 3090, 8× faster than the existing state-of-the-art camera-based method while achieving 5:0 higher mAP. Even compared with the existing stateof-the-art multi-modality method, MapTR-nano achieves 0:7 higher mAP , and MapTR-tiny achieves 13:5 higher mAP and 3× faster inference speed. Abundant qualitative results show that MapTR maintains stable and robust map construction quality in complex and various driving scenes. MapTR is of great application value in autonomous driving. Code and more demos are available at https://github.com/hustvl/MapTR.
高清（HD）地图提供了驾驶场景中丰富而精确的环境信息，是自动驾驶系统中规划部分的基础和不可或缺的组成部分。我们提出了MapTR，这是一个结构化的端到端Transformer，用于高效的在线矢量化高清地图构建。我们提出了一种统一的排列等价建模方法，即把地图元素建模为一组等价排列的点集，这能够准确描述地图元素的形状并稳定学习过程。我们设计了一种分层查询嵌入方案，以灵活编码结构化地图信息，并执行分层二分图匹配以进行地图元素学习。在nuScenes数据集上，MapTR仅使用摄像头输入就在现有的矢量化地图构建方法中实现了最佳性能和效率。特别是，MapTR-nano在RTX 3090上以实时推理速度（25.1 FPS）运行， 比现有的最先进的基于摄像头的方法快8倍 ，同时实现了5.0更高的平均精度（mAP）。即使与现有的最先进的多模态方法相比，MapTR-nano实现了0.7更高的mAP，而MapTR-tiny实现了13.5更高的mAP和3倍更快的推理速度。丰富的定性结果表明，MapTR在复杂多变的驾驶场景中保持了稳定和健壮的地图构建质量。MapTR在自动驾驶中具有重要的应用价值。代码和更多演示可在以下链接获取：https://github.com/hustvl/MapTR。

1 INTRODUCTION

High-definition (HD) map is the high-precision map specifically designed for autonomous driving, composed of instance-level vectorized representation of map elements (pedestrian crossing, lane divider, road boundaries, etc.). HD map contains rich semantic information of road topology and traffic rules, which is vital for the navigation of self-driving vehicle.
高清（HD）地图是专为自动驾驶设计的高精度地图，由地图元素的实例级矢量化表示（人行横道、车道分隔线、道路边界等）组成。高清地图包含了丰富的道路拓扑和交通规则的语义信息，这对于自动驾驶车辆的导航至关重要。
Conventionally HD map is constructed offline with SLAM-based methods (Zhang & Singh, 2014; Shan & Englot, 2018; Shan et al., 2020), incurring complicated pipeline and high maintaining cost. Recently, online HD map construction has attracted ever-increasing interests, which constructs map around ego-vehicle at runtime with vehicle-mounted sensors, getting rid of offline human efforts.
传统上，高清（HD）地图是通过基于SLAM（Simultaneous Localization and Mapping，即同时定位与地图构建）的方法离线构建的（Zhang & Singh, 2014; Shan & Englot, 2018; Shan et al., 2020），这涉及到复杂的流程和高昂的维护成本。最近，在线高清地图构建越来越受到关注，它通过车载传感器在运行时围绕自车构建地图，摆脱了线下人工的努力。
Early works (Chen et al., 2022a; Liu et al., 2021a; Can et al., 2021) leverage line-shape priors to perceive open-shape lanes based on the front-view image. They are restricted to single-view perception and can not cope with other map elements with arbitrary shapes. With the development of bird’s eye view (BEV) representation learning, recent works (Chen et al., 2022b; Zhou & Krahenb ¨ uhl ¨ , 2022; Hu et al., 2021; Li et al., 2022c) predict rasterized map by performing BEV semantic segmentation. However, the rasterized map lacks vectorized instance-level information, such as the lane structure, which is important for the downstream tasks (e.g., motion prediction and planning). To construct vectorized HD map, HDMapNet (Li et al., 2022a) groups pixel-wise segmentation results, which requires complicated and time-consuming post-processing. VectorMapNet (Liu et al., 2022a) represents each map element as a point sequence. It adopts a cascaded coarse-to-fine framework and utilizes auto-regressive decoder to predict points sequentially, leading to long inference time.
早期的工作（Chen et al., 2022a; Liu et al., 2021a; Can et al., 2021）利用线形先验，基于前视图图像感知开放形状的车道。它们受限于单视图感知，无法应对其他任意形状的地图元素。随着鸟瞰图（BEV）表示学习的发展，最近的工作（Chen et al., 2022b; Zhou & Krahenbühl, 2022; Hu et al., 2021; Li et al., 2022c）通过执行BEV语义分割来预测光栅化地图。然而，光栅化地图缺乏矢量化的实例级信息，例如车道结构，这对于下游任务（例如，运动预测和规划）是重要的。为了构建矢量化的高清地图，HDMapNet（Li et al., 2022a）将像素级分割结果进行分组，这需要复杂且耗时的后处理。VectorMapNet（Liu et al., 2022a）将每个地图元素表示为点序列。它采用了一个级联的粗到细框架，并利用自回归解码器顺序预测点，导致推理时间较长。
Current online vectorized HD map construction methods are restricted by the efficiency and not applicable in real-time scenarios. Recently, DETR (Carion et al., 2020) employs a simple and efficient encoder-decoder Transformer architecture and realizes end-to-end object detection.
当前在线矢量化高清地图构建方法受到效率限制，不适用于实时场景。最近，DETR（Carion et al., 2020）采用了简单高效的编码器-解码器Transformer架构，并实现了端到端的目标检测。
It is natural to ask a question: Can we design a DETR-like paradigm for efficient end-to-end vectorized HD map construction? We show that the answer is affirmative with our proposed Map TRansformer (MapTR)
很自然地，我们会问一个问题：我们能否设计一个类似DETR的范式来高效地进行端到端的矢量化高清地图构建？我们展示了一个肯定的答案，这就是我们提出的MapTRansformer（MapTR）。
Different from object detection in which objects can be easily geometrically abstracted as bounding box, vectorized map elements have more dynamic shapes. To accurately describe map elements, we propose a novel unified modeling method. We model each map element as a point set with a group of equivalent permutations. The point set determines the position of the map element. And the permutation group includes all the possible organization sequences of the point set corresponding to the same geometrical shape, avoiding the ambiguity of shape.
与目标检测不同，目标可以很容易地被几何抽象为边界框，矢量化地图元素具有更动态的形状。为了准确描述地图元素，我们提出了一种新颖的统一建模方法。我们将每个地图元素建模为一组具有等价排列的点集。点集决定了地图元素的位置。排列群包括所有可能的点集组织序列，这些序列对应于相同的几何形状，避免了形状的歧义。
Based on the permutation-equivalent modeling, we design a structured framework which takes as input images of vehicle-mounted cameras and outputs vectorized HD map. We streamline the on-line vectorized HD map construction as a parallel regression problem. Hierarchical query embeddings are proposed to flexibly encode instance-level and point-level information. All instances and all points of instance are simultaneously predicted with a unified Transformer structure. And the training pipeline is formulated as a hierarchical set prediction task, where we perform hierarchical bipartite matching to assign instances and points in turn. And we supervise the geometrical shape in both point and edge levels with the proposed point2point loss and edge direction loss.
基于排列等价建模，我们设计了一个结构化框架，它以车载摄像头拍摄的图像为输入，输出矢量化的高清地图。我们将在线矢量化高清地图构建简化为一个并行回归问题。提出了分层查询嵌入，以灵活编码实例级和点级信息。所有实例及其所有点同时通过统一的Transformer结构进行预测。训练流程被构建为一个分层集合预测任务，其中我们执行分层二分图匹配来依次分配实例和点。我们还提出了点对点损失和边缘方向损失，以在点和边缘两个层面上对几何形状进行监督。
With all the proposed designs, we present MapTR, an efficient end-to-end online vectorized HD map construction method with unified modeling and architecture. MapTR achieves the best performance and efficiency among existing vectorized map construction approaches on nuScenes (Caesar et al., 2020) dataset. In particular, MapTR-nano runs at real-time inference speed (25:1 FPS) on RTX 3090, 8× faster than the existing state-of-the-art camera-based method while achieving 5:0 higher mAP. Even compared with the existing state-of-the-art multi-modality method, MapTR-nano achieves 0:7 higher mAP and 8× faster inference speed, and MapTR-tiny achieves 13:5 higher mAP and 3× faster inference speed. As the visualization shows (Fig. 1), MapTR maintains stable and robust map construction quality in complex and various driving scenes.
结合所有提出的设计，我们呈现了MapTR，这是一种高效的端到端在线矢量化高清地图构建方法，具有统一的建模和架构。在nuScenes（Caesar et al., 2020）数据集上，MapTR在现有的矢量化地图构建方法中实现了最佳性能和效率。特别是，MapTR-nano在RTX 3090上以实时推理速度（25.1 FPS）运行，比现有的最先进的基于摄像头的方法快8倍，同时实现了5.0更高的平均精度（mAP）。即使与现有的最先进的多模态方法相比，MapTR-nano实现了0.7更高的mAP和8倍更快的推理速度，而MapTR-tiny实现了13.5更高的mAP和3倍更快的推理速度。正如可视化所示（图1），MapTR在复杂多变的驾驶场景中保持了稳定和健壮的地图构建质量。

Our contributions can be summarized as follows:
• We propose a unified permutation-equivalent modeling approach for map elements, i.e., modeling map element as a point set with a group of equivalent permutations, which accurately describes the shape of map element and stabilizes the learning process.
• Based on the novel modeling, we present MapTR, a structured end-to-end framework for efficient online vectorized HD map construction. We design a hierarchical query embedding scheme to flexibly encode instance-level and point-level information, perform hierarchical bipartite matching for map element learning, and supervise the geometrical shape in both point and edge levels with the proposed point2point loss and edge direction loss.
• MapTR is the first real-time and SOTA vectorized HD map construction approach with stable and robust performance in complex and various driving scenes.
我们的贡献可以总结如下：

• 我们提出了一种统一的排列等价建模方法来处理地图元素，即把地图元素建模为一组等价排列的点集，这能够准确描述地图元素的形状并稳定学习过程。
• 基于这种新颖的建模方法，我们提出了MapTR，这是一个结构化的端到端框架，用于高效的在线矢量化高清地图构建。我们设计了一种分层查询嵌入方案，以灵活编码实例级和点级信息，执行分层二分图匹配进行地图元素学习，并使用提出的点对点损失和边缘方向损失在点和边缘两个层面上对几何形状进行监督。
• MapTR是第一个实时且在复杂多变的驾驶场景中具有稳定和鲁棒性能的SOTA（State of the Art）矢量化高清地图构建方法。

2 RELATED WORK

HD Map Construction. Recently, with the development of 2D-to-BEV methods (Ma et al., 2022), HD map construction is formulated as a segmentation problem based on surround-view image data captured by vehicle-mounted cameras. Chen et al. (2022b); Zhou & Krahenb ¨ uhl ¨ (2022); Hu et al. (2021); Li et al. (2022c); Philion & Fidler (2020); Liu et al. (2022b) generate rasterized map by performing BEV semantic segmentation. To build vectorized HD map, HDMapNet (Li et al., 2022a) groups pixel-wise semantic segmentation results with heuristic and time-consuming post-processing to generate instances. VectorMapNet (Liu et al., 2022a) serves as the first end-to-end framework, which adopts a two-stage coarse-to-fine framework and utilizes auto-regressive decoder to predict points sequentially, leading to long inference time and the ambiguity about permutation. Different from VectorMapNet, MapTR introduces novel and unified modeling for map element, solving the ambiguity and stabilizing the learning process. And MapTR builds a structured and parallel onestage framework with much higher efficiency.
高清地图构建。最近，随着2D到鸟瞰图（BEV）方法（Ma et al., 2022）的发展，高清地图构建被构建为一个基于车载摄像头捕获的环视图像数据的分割问题。Chen et al. (2022b)；Zhou & Krahenbühl (2022)；Hu et al. (2021)；Li et al. (2022c)；Philion & Fidler (2020)；Liu et al. (2022b)通过执行BEV语义分割来生成光栅化地图。为了构建矢量化高清地图，HDMapNet（Li et al., 2022a）通过启发式且耗时的后处理将像素级语义分割结果分组以生成实例。VectorMapNet（Liu et al., 2022a）作为第一个端到端框架，采用两阶段的粗到细框架，并利用自回归解码器顺序预测点，导致推理时间长且排列存在歧义。与VectorMapNet不同，MapTR引入了地图元素的新颖且统一的建模，解决了歧义并稳定了学习过程。而且，MapTR构建了一个结构化且并行的单阶段框架，效率更高。
Lane Detection. Lane detection can be viewed as a sub task of HD map construction, which focuses on detecting lane elements in the road scenes. Since most datasets of lane detection only provide single view annotations and focus on open-shape elements, related methods are restricted to single view. LaneATT (Tabelini et al., 2021) utilizes anchor-based deep lane detection model to achieve good trade-off between accuracy and efficiency. LSTR (Liu et al., 2021a) adopts the Transformer architecture to directly output parameters of a lane shape model. GANet (Wang et al., 2022) formulates lane detection as a keypoint estimation and association problem and takes a bottom-up design. Feng et al. (2022) proposes parametric Bezier curve-based method for lane detection. Instead of detecting lane in the 2D image coordinate, Garnett et al. (2019) proposes 3D-LaneNet which performs 3D lane detection in BEV. STSU (Can et al., 2021) represents lanes as a directed graph in BEV coordinates and adopts curve-based Bezier method to predict lanes from monocular camera image. Persformer (Chen et al., 2022a) provides better BEV feature representation and optimizes anchor design to unify 2D and 3D lane detection simultaneously. Instead of only detecting lanes in the limited single view, MapTR can perceive various kinds of map elements of 360◦ horizontal FOV, with a unified modeling and learning framework.
车道检测。车道检测可以被视为高清地图构建的一个子任务，专注于在道路场景中检测车道元素。由于大多数车道检测数据集仅提供单视图注释并专注于开放形状元素，相关方法受限于单视图。LaneATT（Tabelini et al., 2021）利用基于锚点的深度车道检测模型，在准确性和效率之间取得了良好的平衡。LSTR（Liu et al., 2021a）采用Transformer架构直接输出车道形状模型的参数。GANet（Wang et al., 2022）将车道检测公式化为关键点估计和关联问题，并采用自下而上的设计理念。Feng et al.（2022）提出了基于参数贝塞尔曲线的车道检测方法。与在2D图像坐标中检测车道不同，Garnett et al.（2019）提出了3D-LaneNet，它在BEV中执行3D车道检测。STSU（Can et al., 2021）在BEV坐标中将车道表示为有向图，并采用基于曲线的贝塞尔方法从单目摄像头图像预测车道。Persformer（Chen et al., 2022a）提供了更好的BEV特征表示，并优化了锚点设计，以同时统一2D和3D车道检测。与仅在有限的单视图中检测车道不同，MapTR能够感知360°水平视场的各种地图元素，具有统一的建模和学习框架。
Contour-based Instance Segmentation. Another line of work related to MapTR is contour-based 2D instance segmentation (Zhu et al., 2022; Xie et al., 2020; Xu et al., 2019; Liu et al., 2021c). These methods reformulate 2D instance segmentation as object contour prediction task, and estimate the image coordinates of the contour vertices. CurveGCN (Ling et al., 2019) utilizes Graph Convolution Networks to predict polygonal boundaries. Lazarow et al. (2022); Liang et al. (2020); Li et al. (2021); Peng et al. (2020) rely on intermediate representations and adopt a two-stage paradigm, i.e., the first stage performs segmentation / detection to generate vertices and the second stage converts vertices to polygons. These works model contours of 2D instance masks as polygons. Their modeling methods cannot cope with line-shape map elements and are not applicable for map construction. Differently, MapTR is tailored for HD map construction and models various kinds of map elements in a unified manner. Besides, MapTR does not rely on intermediate representations and has an efficient and compact pipeline.
基于轮廓的实例分割。与MapTR相关的另一系列工作是基于轮廓的2D实例分割（Zhu et al., 2022; Xie et al., 2020; Xu et al., 2019; Liu et al., 2021c）。这些方法将2D实例分割重新定义为对象轮廓预测任务，并估计轮廓顶点的图像坐标。CurveGCN（Ling et al., 2019）利用图卷积网络来预测多边形边界。Lazarow et al. (2022); Liang et al. (2020); Li et al. (2021); Peng et al. (2020)依赖于中间表示，并采用两阶段范式，即第一阶段执行分割/检测以生成顶点，第二阶段将顶点转换为多边形。这些工作将2D实例掩码的轮廓建模为多边形。它们的建模方法无法应对线形地图元素，也不适用于地图构建。不同地，MapTR专为高清地图构建量身定制，以统一的方式建模各种类型的地图元素。此外，MapTR不依赖于中间表示，并且具有高效且紧凑的流程。

3 MAPTR

3.1 PERMUTATION-EQUIVALENT MODELING

MapTR aims at modeling and learning the HD map in a unified manner. HD map is a collection of vectorized static map elements, including pedestrian crossing, lane divider, road boundarie, etc. For structured modeling, MapTR geometrically abstracts map elements as closed shape (like pedestrian crossing) and open shape (like lane divider). Through sampling points sequentially along the shape boundary, closed-shape element is discretized into polygon while open-shape element is discretized into polyline.
MapTR旨在以统一的方式对高清地图进行建模和学习。高清地图是一系列矢量化静态地图元素的集合，包括人行横道、车道分隔线、道路边界等。为了结构化建模，MapTR几何抽象地将地图元素表示为封闭形状（如人行横道）和开放形状（如车道分隔线）。通过沿形状边界顺序采样点，封闭形状元素被离散化为多边形，而开放形状元素被离散化为折线。
Preliminarily, both polygon and polyline can be represented as an ordered point set V F = [v0; v1; : : : ; vNv−1] (see Fig. 3 (Vanilla)). Nv denotes the number of points. However, the permutation of the point set is not explicitly defined and not unique. There exist many equivalent permutations for polygon and polyline. For example, as illustrated in Fig. 2 (a), for the lane divider (polyline) between two opposite lanes, defining its direction is difficult. Both endpoints of the lane divider can be regarded as the start point and the point set can be organized in two directions. In Fig. 2 (b), for the pedestrian crossing (polygon), the point set can be organized in two opposite directions (counter-clockwise and clockwise). And circularly changing the permutation of point set has no influence on the geometrical shape of the polygon. Imposing a fixed permutation to the point set as supervision is not rational. The imposed fixed permutation contradicts with other equivalent permutations, hampering the learning process.
初步来看，多边形和折线都可以表示为有序点集 $V^F=[v_0,v_1,\dots ,v_{N_v-1}]$（见图3（Vanilla））。$N_v$ 表示点的数量。然而，点集的排列并没有明确定义，也不是唯一的。对于多边形和折线，存在许多等价的排列。例如，如 图2（a） 所示，对于两个相对车道之间的车道分隔线（折线），定义其方向是困难的。车道分隔线的两个端点都可以被视为起点，点集可以以两个方向组织。在图2（b）中，对于人行横道（多边形），点集可以以两个相反的方向（逆时针和顺时针）组织。并且，循环改变点集的排列对多边形的几何形状没有影响。将固定的排列强加给点集作为监督是不合理的。强加的固定排列与其他等价排列相矛盾，妨碍了学习过程。

图2. 典型案例，用以说明地图元素关于起点和方向的歧义性。
(a) 折线：对于两个相对车道之间的车道分隔线，定义其方向是困难的。车道分隔线的两个端点都可以被视为起点，点集可以以两个方向组织。
(b) 多边形：对于人行横道，多边形的每个点都可以被视为起点，并且多边形可以以两个相反的方向（逆时针和顺时针）连接。

图3. MapTR排列等价建模的说明。地图元素被几何抽象并离散化为折线和多边形。MapTR使用（V, Γ）对每个地图元素进行建模（一个点集 V 和一组等价排列 Γ），避免了歧义并稳定了学习过程。
To bridge this gap, MapTR models each map element with V = (V; Γ). V = fvjgN j=0 v−1 denotes the point set of the map element (Nv is the number of points). Γ = fγkg denotes a group of equivalent permutations of the point set V , covering all the possible organization sequences.
为了弥补这一差距，MapTR使用 $V=(V,\Gamma )$ 来对每个地图元素进行建模。 V = { v j } j = 0 N v − 1 V = \{v_j\}^{N_v-1}_{j=0} V={ vj​}j=0Nv​−1​ 表示地图元素的点集（$N_v$ 是点的数量）。 Γ = { γ k } Γ = \{γ^k\} Γ={ γk} 表示点集 $V$ 的一组等价排列，涵盖了所有可能的组织序列。
Specifically, for polyline element (see Fig. 3 (left)), Γ includes 2 kinds of equivalent permutations:
具体来说，对于折线元素（见图3（左）），Γ 包括 2 种等价排列：

For polygon element (see Fig. 3 (right)), Γ includes 2 × Nv kinds of equivalent permutations:
对于多边形元素（见图3（右）），Γ 包括 $2\times N_v$ 种等价排列：