Abstract

Neural fields, which represent signals as a function parameterized by a neural network, are a promising alternative to traditional discrete vector or grid-based representations. Compared to discrete representations, neural representations both scale well with increasing resolution, are continuous, and can be many-times differentiable. However, given a dataset of signals that we would like to represent, having to optimize a separate neural field for each signal is inefficient, and cannot capitalize on shared information or structures among signals. Existing generalization methods view this as a meta-learning problem and employ gradient-based meta-learning to learn an initialization which is then fine-tuned with test-time optimization, or learn hypernetworks to produce the weights of a neural field. We instead propose a new paradigm that views the large-scale training of neural representations as a part of a partially-observed neural process framework, and leverage neural process algorithms to solve this task. We demonstrate that this approach outperforms both state-of-the-art gradient-based meta-learning approaches and hypernetwork approaches.

Results

Depicted below are neural fields representing 2D CT scans generated by our model from 1D partial observations:

Neural fields generated with our method are sharper than those generated with prior approaches such as gradient-based meta-learning (Reptile). Using a neural process-based framework also offers the opportunity to quantify uncertainty. Depicted below are preliminary results detailing what our model is uncertain about.

Citation

@article{gu2023generalizable,
  title={Generalizable Neural Fields as Partially Observed Neural Processes},
  author={Gu, Jeffrey and Wang, Kuan-Chieh and Yeung, Serena},
  journal={arXiv preprint arXiv:2309.06660},
  year={2023}
}

Acknowledgements

The website template was borrowed from ProbNeRF.