We built a 3D film tool called ArtCraft:
https://github.com/storytold/artcraft (our monorepo with the desktop app, server, and website code)
Our 3D virtual film set enables set decoration, location reuse, precision blocking (control over where things are), character posing, etc.
Sets and objects can be generated as meshes or gaussian splats.
Autoregressive multimodal models can treat input "previz" style scenes as ControlNets of sorts. Combined with reference images and a possibly a few follow-up rounds of editing, you can create extremely precise starting frames that reflect your vision as a director or creative almost exactly as you see it.
It gets rid of the "roulette wheel" of prompting in the places and times where you know you what you want. It's like a WYSIWYG editor.
Here's an example:
https://app.getartcraft.com/edit-3d/m_qa72baw3crghyfn2bbw52j... (Sci-Fi horror film, starring Garry Tan.)
I was just told that Google built a 3D tool similar to ours that they're showing off at Google IO. I don't view this as bad, but I do want to share ours before they get all the attention.
OpenAI's GPT Image is a lot better at this "3D previz to full render" task than Google's Nano Banana, and it's excelled at this workflow since GPT Image 1. (Unless Google shows off a brand new model for the task, I expect this to still be the case.)
Our plan is to build vertical specialized tools for animation, editing, dubbing and make the entire stack - including cloud API routers and model containers - open source.
Here's an example of timeline based animation (and we have the ability to edit animation curves for all objects, meshes, cameras, etc.):
Our tool has BYOK and subscriptions, so you can use lots of other API providers and even log in with the commercial versions of other models and aggregators. We don't care if we book the revenue directly, we want to build the rails and the UX and win mindshare.
We hit a pretty good growth spurt and have had folks start sending us PRs, which has been exciting.
Eager to see what Google's version of this looks like.
If you're a Rust developer interested in film and media, I'd love to say hi.
No comments yet.
