Newton: physics simulation engine built upon NVIDIA Warp (opens in new tab)

I think this is a step in the right direction, but I really dislike the Pythonification of everything. After using IsaacSim/IsaacLab for work, I'm convinced that Python is not the right tool for the job.

Developers inevitably write slow, error filled code when dealing with Python and working with the type annotations can be a pain.

Happy there's something to replace PhysX for robotics, and I do really like MuJoCo's API, but really wish we could get some good C/C++ APIs.

Apart from the language, NVIDIA doesn't seem to be great when dealing with software. IsaacSim and IsaacLab have so many bugs, are incredibly slow, and hard to debug. We spend so many hours on my team findings bugs for IsaacSim, it's just a pain. On version 5.0 and still feels like beta software.

Also IsaacSim's relience on USD to hold the scene structure and update prims makes it so hard to program for. USD isn't really performant when trying to generate a large amount of scenes. And the USD interface stops working completly when simulation starts on IsaacLab. I hope Newton goes a different route, and has less of a reliacne on USD. IMO USD should just be used as an interchange format, rather than how you actually represent the scene and properties internally. I much prefer that approach, which Unreal Engine seems to support.

Lastly, my god the names in this field are terrible. USD (Googling becomes a pain sometimes), Newton (Already another engine), Warp (literally the name of the architecture and a way to write Python GPU kernels, wtf).

I've been playing around with this for a few weeks. Newton is a pretty thin wrapper around mujoco-warp, which is trying to port mujoco, originally a CPU sim, over to warp on the GPU. There is also mujoco-mjx for this purpose, but using jax instead of warp. I think mjx/jax has the edge on performance because there are mature RL libraries for jax (brax) and big advantages to using Jax for everything, especially with its ability to "vmap" over each layer of abstraction. But I can see why nvidia wants to move away from IsaacLab using physX+pytorch because physX was made for games and interfacing with it through IsaacSim is a bit of a kludge. And apparently mjx isn't so accurate with collisions because of the way it has to be treated in jax. Pytorch RL works decently with newton/warp, at least they can share GPU buffers and you don't have to copy things back and forth to the CPU, however you can't optimize with cuda graphs past the newton/warp boundary because newton/warp have their own cuda graph capture scheme going on at the same time underneath.

They already have a newton branch of IsaacLab on github but its pretty early for it. I just came across a dope project today that is a different wrapper around mujoco-warp that already mimics IsaacLab's api and you can run some robot environments on it. Clean code too, very promising: https://github.com/mujocolab/mjlab.git

We're becoming too reliant on libraries made by our hardware vendors (vendor, singular, actually, to make it worse).

One feedback from someone interested in using this about the examples: I have looked at several and they seem too high level to get a sense of the actual API (i.e. the expected benefit of using this library vs the development complexity of using it).

For example, the cloth bending simulation is almost entirely: at __init__, call a function to add a cloth mesh to model builder obj, pass built model to initializer of a solver class; and at each timestep: call a collide model function, then call another function called solver.step. That's really it.

Is this related to the Newton Dynamics physics engine? https://newtondynamics.com/

Years ago there was PhysX... how does this compare?

How do they parallelize the sequential actions ?

What type of solver is mujoco?