Carlos here, sole dev and product at the NIUS.TV.
We have taken a similar approach on NIUS.TV. We've even taken it further by synthesizing an AI anchor.
Check it out:
- https://twitter.com/nius_tv
- https://twitter.com/beyondearthtv
The way we synthesize our videos is a little different from Baidu in practice, but similar on a high level.
For video synthesis we use an event-driven architecture using K8s jobs, and a custom job scheduler built in-house that manages all the steps of the video creation process.
Additionally, because GPUs are expensive and knowing that we synthesize our stories in batches through the day, we built a k8s-cluster-builder, that spins up and down the k8s cluster based on pending stories. This saves us a tremendous amount of cost per month, as the cluster only runs when there is actual work for it. As a bootstrapped project, costs are really important.
Our current video synthesis pipeline is comprised of 20+ steps, this includes speech-synthesis, image generation, speech+text aligners (for captions), language models, text generation, among others.
From my experience building the product, and despite the fact that AI is expensive (training wise), it is feasible to build AI-powered products using off-the-shelf DL technology without having large resources.
