VToonify: Controllable high-resolution portrait video style transfer (opens in new tab)

Idk about the future of animation. It would require look-alike actors. Wouldn't it be easier to use already available tech to transfer gestures and facial expressions to cartoon models?

Can you confirm that was for inference? I thought that was only for training 55min on 8x v100

I missed the bit about using 8 of them to run it! Wow that’s a lot of GPU horsepower to do this. More efficient to just use a vtuber style pipeline using unreal engine and the metahumans or other avatars… only need one good GOU for that.

M1 performs better in some realtime use-cases because of the unified memory: the GPU and ML hardware can work on a camera framebuffer directly without any copy.

CUDA always requires sending data over the PCI bus, at least when it comes to realtime camera processing. GPUDirect exists but it's optimized for disks and NICs, I don't believe it's possible to use it with cameras.

No idea actually, I just find all sorts of odd benchmarks crop up for things where the Unified Memory Architecture on the M1/M2 give things surprising good performance due to the DMA transfer performance hit on other CPU/GPU combinations… it’s far from universal, but it’s just been surprising to see and this looked like the sort of thing that might be one of them between the camera decoding the ML & GPU processing and then “rendering” back out… where it might have had some benefits, hence my “wondering out loud”.

It’s been a while since I bothered looking at anything made by nVidia, I didn’t like their business practices and to be honest… Living in the Mac ecosystem for 90% of my work/life means I followed the paltry AMD rocM stuff ardently and was so happy when Apple made an effort up front to get the M1 and M2 chips supported by the major ML frameworks.

That’s a good point. I did briefly think about 24fps for “cinema” but as that’s a bit of a weird frame rate for a streaming webcam I dismissed the idea. But the video does actually have just enough frame rate to look smooth and I suppose that’s because it’s landing near enough to 12fps that combined with “looks like cartoon”, my brain is filing in the rest and barely noticing.