Atlas: End-to-End 3D Scene Reconstruction from Posed Images (opens in new tab)

I have no experience in this field at all and they note on the video that the sequence shown was not realtime but I wonder how far we're from having something like this running in realtime or how "realtime" it could be given fancy hardware to be used in the wild?

Yeah I have to agree. If this were a PhD thesis it would certainly deserve some praise, but given that this is the most exciting thing to come out of magic leap in years just barely puts them on par with SOTA...well I would be pretty pissed if I was an investor in them.

It is easy to have a billion dollar company if you have borrowed two billion. And haven't spent all the loot on salaries.

The company has a cool name and the product area is divisive. Some say it is vapourware and nobody wants Oculus Rift style VR. Others are gung-ho. It's like Bitcoin all over again.

Although this tech is being done with AI, it was being done with non-AI approach two decades ago for movies/TV. But it wasn't as if people ported this tech to their smart phones from the SGI desktop monsters of yesteryear.

Even more hijacking, I remember thinking medical applications were going to be the killer apps for VR. I was blown away by these demos almost half a decade ago https://youtu.be/MWGBRsV9omw?t=251

Did they ever make it into real life practice?

You're not the first to come up with that challenge ;) https://endovis.grand-challenge.org/

I believe random videos are too low of a quality. Like this, most of the stuff I've seen uses constrained videos.

I have seen random still images used for this kind of thing: https://nerf-w.github.io/

I haven't heard of any equivalent of EXIF for video. That goes a long way when trying to make sense of random video both for camera settings as well as GPS location if you're trying to correlate multiple videos.

Google will do this, and then sell the data to security institutions. We will be told about it later, or consent to it during a Terms&Conditions update.

Cool idea, but how would you keep it maintained? It's tricky enough to keep maps up to date. A 3D Mesh would be even more complex to maintain.

By default NeRF does not produce a mesh (but one could use marching cubes as does Atlas) and it requires training a neural network for each scene whereas Atlas (as far as I understand it) uses pretrained network to process new scenes.

NeRF would probably produce a much better final result but the Atlas approach (no need to train something from scratch) is the only one that can hope to be run in real time which is vital for some application.

NeRF has a potential to make all those classical methods obsolete, though it requires many input images and I am not sure how it handles rolling shutter and other distortions.

In theory this should work. I’ve been doing photogrammetry with spherical video and existing software packages often want to “dewarp” the image on to a plane, which works fine for narrow field of view but fails on spherical video. It would be interesting to see if atlas supports spherical input. Also 360 cameras have pretty low visual acuity. My 5.6k GoPro Fusion has to divide those pixels across the whole field of view, so images are less detailed. Still I think 360 video can be useful in photogrammetry with the right algorithms.

Worst case, you can sample the 360 frames to get images with a smaller field of view. However, the app takes in camera intrinsics and positional data so it seems like it would work out of the box.

i imagine a lot of unfortunate artefacts come out of stitching together the camera views that form a 360 or "spherical" image.