【读论文】MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving

1. What

An autonomous driving simulator based on Nerf with three features: instance-aware, modular, and realistic.

2. Why

As for current autonomous driving algorithms, training the corner cases is helpful for their performance bottleneck.

Existing autonomous driving simulation methods have their own limitations, such as CARLA, AADS, and GeoSim.

Recently, Neural Scene Graph (NSG) decomposes dynamic scenes into learned scene graphs and learns latent representations for category-level objects. However, its multi-plane-based representation for background modeling cannot synthesize images under large viewpoint changes.

3. What

3.1 Inputs

The input to the system consists of a set of RGB-images { I i } N \{\mathcal{I}i\}^N { Ii}N, sensor poses { T i } N \{\mathcal{T}i\}^N { Ti}N (calculated using IMU/GPS signals), and object tracklets (including 3D bounding boxes { B i j } N × M \{\mathcal{B}_{ij}\}^{N\times M} { Bij​}N×M, categories { t y p e i j } N × M \{ \mathrm{type}_{ij}\} ^{N\times M} { typeij​}N×M, and instance IDs { i d x i j } N × M ) \{\mathrm{idx}_{ij}\}^{N\times M}) { idxij​}N×M). $N$ is the number of input frames and $M$ is the number of tracked instances { O j } M \{\mathcal{O}_j\}^M { Oj​}M across the whole sequence. An optional set of depth maps { D i } N \{\mathcal{D}_i\}^N { Di​}N and semantic segmentation masks.

3.2 Scene Representation

Architectures: It supports various NeRF backbones, which can be roughly categorized into two hyper-classes: MLP-based methods, or grid-based methods and this paper gives a formal exposition of grid-based methods(Instant-ngp).

Foreground Nodes: Similar to NSG, it exploits latent codes to encode
instance features and shared category-level decoders to encode class-wise priors.

3.3 Compositional Rendering

It uses the standard volume rendering process to render pixel-wise properties:

c ^ ( r ) = ∑ P i T i α i c i + ( 1 − a c c u m ) ⋅ c s k y , T i = exp ⁡ ( − ∑ k = 1 i − 1 σ k δ k ) d ^ ( r ) = ∑ P i T i α i t i + ( 1 − a c c u m ) ⋅ i n f s ^ ( r ) = ∑ P i T i α i s i + ( 1 − a c c