World Emulation via Neural Network (opens in new tab)

You can do it using the more classic technique of photogrammetry. There are commercial products used by real estate salesmen to produce high quality "games" where you walk around inside a house, but they're more like Google Streetview where you swoosh between points where a 360 degree photo was taken. All those things will be more faithful than neurally generating next frames based on previous frames and control input.

>I don't get this analogy at all. Instead of a human information flows through a neural network which alters the information.

These days most photos are also stored using lossy compression which alters the information.

You can think of this as a form of highly lossy compression of an image of this forest in time and space.

Most lossy compression is 'subtractive' in that detail is subtracted from the image in order to compress it, so the kind of alterations are limited. However there have been previous non-subtractive forms of compression (eg, fractal compression) that have been criticised on the basis of making up details, which is certainly something that a neural network will do. However if the network is only trained on this forest data, rather than being also trained on other data and then fine tuned, then in some sense it does only represent this forest rather than giving an 'informed impression' like a human artist would.

>It might also be there because the network inferred that it is there based on previous data.

There is no previous data. This network is exclusively trained on the data he collected from the scene.

I think https://diamond-wm.github.io is a reasonable place to start (they have public world-model training code, and people have successfully adapted their codebase to other games e.g. https://derewah.dev/projects/ai-mariokart). Most modern world models are essentially image generators with additional inputs (past-frames + controls) added on, so understanding how Diffusion/IADB/Flow Matching work would definitely help.

Yup, definitely similar! There are a lot of video-game-emulation World Models floating around now, https://worldarcade.gg had a list. In the self-driving & robotics literature there have also been many WMs created for policy training and evaluation. I don't remember a prior WM built on first-person cell-phone video, but it's a simple enough concept that someone has probably done it for a student project or something :)

NeRF is a more complex and constrained approach, based on a kind of ray tracing. But results are obviously similar.

Curious, which device/OS/browser? I did all my testing on 4-year old hardware (iPhone 13 Pro, M1 Pro MBP), and the model itself is extremely tiny (~1GFLOP) so I'm optimistic that performance issues would be solvable with a better software stack (e.g. native app).

Yeah, the similarities to psychedelics with some of this stuff is remarkable.

I also thought those wooden guard rails looked pretty spot on how they would look on 2C-B. The only thing that's missing is the overlay of geometric patterns on even surfaces.

For some reason, psilocybin causes me to randomly just lose consciousness, and LSD doesn't. Weird stuff.

or for a large dataset of math identities and have the user draw one side

Thanks! My favorite failure mode (not mentioned in the post - I think it was during the first round of upgrades?) was a "dry" form of soupification where the texture detail didn't fully disappear https://imgur.com/c7gVRG0

Yes please! I would love to try and use this on disappearing neighbourhoods, the results are so dreamlike, or like memories!

Yes! This was a solo project done in my free time :) to learn about WMs and get more practice training GANs.

The special aspect of NNs (in the context of simulating worlds) is that NNs can mimic entire worlds from videos alone, without access to the source code (in the case of pokemon) or even without the source code having existed (as is the case for the real-world forest trail mimicked in this post). They mimic the entire interactive behavior of the world, not just the geometry (note e.g. the not-programmed-in autoexposure that appears when you look at the sky).

Although the neural world in the post is a toy project, and quite far from generating photorealistic frames with "trees that bend in the wind, lilypads that bob in the rain, birds that sing to each other", I think getting better results is mostly a matter of scale. See e.g. the GAIA-2 results (https://wayve.ai/wp-content/uploads/2025/03/generalisation_0..., https://wayve.ai/wp-content/uploads/2025/03/unsafe_ego_01_le...) for an example of what WMs can do without the realtime-rendering-in-a-browser constraints :)

author is: https://x.com/madebyollin

The model seems to be viewable here:

https://netron.app/?url=https://madebyoll.in/posts/world_emu...