3D Novel View Synthesis with Diffusion Models (opens in new tab)

Interesting to compare this to DreamFusion[1] which came out of Google Research (which also did this paper) today.

In DreamFusion they do use a NeRF representation.

[1] https://dreamfusion3d.github.io/gallery.html

Oh, OpenAI does more or less release that much. People don't have issues implementing the models from their papers.

What they don't do is release the actual models and datasets, and it's very expensive to retrain those.

No, NeRFs are more interpretable because they directly model field densities which absorb and emit light. In this respect they are something akin to a neural version of photogrammetry. They don’t need to train on a large corpus of images, because they can reconstruct directly from a collection of posed images.

On the other hand, diffusion models can learn fairly arbitrary distributions of signals, so by exploiting this learned prior together with view consistency, they can be much more sample efficient than ordinary NeRFs. Without learning such a prior, 3D reconstruction from a single image is extremely ill-posed (much like monocular depth estimation).

Yes, the future is now. It’s still early, extracted model quality will undoubtedly improve dramatically over time.

https://blogs.nvidia.com/blog/2022/09/23/3d-generative-ai-re...

https://research.nvidia.com/publication/2021-11_extracting-t...

You're probably looking for multiview photogrammetry. Also known as "structure from motion." https://github.com/alicevision/Meshroom is a good free tool for this, but the most popular one is probably AGISoft MetaShape.