What about Candle, made by Huggingface? Seems to at least allow the basics and has lots of examples, all of them run on both CPU and GPU. Haven't dived deeper into it, but played around with it a bit and found it good enough for embedding purposes at least.

I think the big value add of Mojo is that you are no longer writing GPU code that only runs on one particular GPU architecture.

In the same way that LLVM allows CPU code to target more than one CPU architecture, MLIR/Mojo allows GPU code to target multiple vendor's GPUs.

There is some effort required to write the backend for a new GPU architecture, and Lattner has discussed it taking about two months for them to bring up H100 support.