[论文精读] [NeRF] [SIGGRAPH 2022] Variable Bitrate Neural Fields

Abstract

Neural approximations of scalar and vector fields, such as signed distance functions and radiance fields, have emerged as accurate, high-quality representations. State-of-the-art results are obtained by conditioning a neural approximation with a lookup from trainable feature grids that take on part of the learning task and allow for smaller, more efficient neural networks. Unfortunately, these feature grids usually come at the cost of significantly increased memory consumption compared to stand-alone neural network models. We present a dictionary method for compressing such feature grids, reducing their memory consumption by up to 100x and permitting a multiresolution representation which can be useful for out-of-core streaming. We formulate the dictionary optimization as a vector-quantized auto-decoder problem which lets us learn end-to-end discrete neural representations in a space where no direct supervision is available and with dynamic topology and structure.

标量场和矢量场的神经近似，如带符号的距离函数和辐射场，已经成为准确的、高质量的表示。最先进的结果是通过从可训练的特征网格中查找神经近似来获得的，该网格承担了部分学习任务，并允许更小、更有效的神经网络。不幸的是，与独立的神经网络模型相比，这些特征网格通常以显著增加的内存消耗为代价。我们提出了一种压缩这种特征网格的字典方法，将其内存消耗减少了100倍，并允许使用多分辨率表示，这对外核流媒体是有用的。我们将字典优化表述为一个矢量量化的自动解码器问题，这让我们能够在一个没有直接监督的空间中学习端到端的离散神经表征，并且具有动态拓扑和结构。

(Top-left shows a baseline neural radiance field whose uncompressed feature grid weighs 15 207 kB. Our method, shown bottom right, compresses this by a factor of 60x, with minimal visual impact (PSNR shown relative to training images). In a streaming setting, a coarse LOD can be displayed after receiving only the first 10 kB of data. All sizes are without any additional entropy encoding of the bit-stream.)

Motivations

Feature grid methods are a special class of neural fields which have enabled state-of-the-art signal reconstruction quality whilst being able to render and train at interactive rates.

基于特征网格的方法是一类特殊的神经场，它实现了最先进的信号重建质量，同时能够以交互速率进行渲染和训练。

Since 𝜓_𝜃(𝑥, interp(𝑥, 𝑍)) ≈ 𝑢(𝑥) is a non-linear function, this approach has the potential to reconstruct signals with frequencies above the usual Nyquist limit. Thus coarser grids can be used, motivating their use in signal compression.

特征网格方法是一个非线性函数，这种方法有可能重建频率高于通常奈奎斯特极限¹的信号。因此，可以使用更粗的栅格，从而促进它们在信号压缩中的使用。
Nyquist Rate是信息论里面的一个概念，如果对一个连续信号进行采样，然后想要用采样之后的信号来恢复出原有信号的完整信息，那么采样率必须大于Nyquist Rate，而这个Rate是此连续信号中最高频分量频率的两倍²。

Compression

The feature grid can be represented as a matrix 𝑍 ∈ R^𝑚×𝑘 where 𝑚 is the number of grid points, and 𝑘 is the dimension of the feature vector at each grid point. Since 𝑚 ×𝑘 may be quite large compared to the size of the MLP, the feature vectors are by far the most memory hungry component.

但是grid与MLP的大小相比可能相当大，特征向量是到目前为止最耗费内存的分量。

These methods require high-resolution feature grids to achieve good quality. This makes them less practical for graphics systems which must operate within tight memory, storage, and bandwidth budgets.

这些方法需要高分辨率的特征栅格才能获得良好的质量。这使得它们不太适用于必须在紧张的内存、存储和带宽预算内运行的图形系统。

Multiresolution representation

Beyond compactness, it is also desirable for a shape representation to dynamically adapt to the spatially varying complexity of the data, the available bandwidth, and desired level of detail.

除了紧凑度之外，形状表示还需要动态地适应数据的空间变化的复杂性、可用带宽和所需的细节级别。

Contributions</