Stability AI introduces Stable Video 4D, its latest AI model for 3D video generation

Open generative artificial intelligence startup Stability AI Ltd., best known for its image generation tool Stable Diffusion, is working hard on developing AI models for 3D video.

Its newest model announced today can take a single video of an object from one angle and reproduce it from multiple angles. Stable Video 4D can transform a video of a 3D object into multiple-angle views of the identical object from eight different perspectives. Meaning it will possibly interpret what the thing looks like, including its movements, from the side it cannot see, to permit it to breed its movement and appearance from different angles.

The brand new model builds on the inspiration of Stability AI’s Stable Video Diffusion model, which the corporate released in November. The Stable Video model can take a still image and convert it right into a photorealistic video, including motion.

“The Stable Video 4D model takes a video as input and generates multiple novel-view videos from different perspectives,” the corporate said within the announcement. “This advancement represents a leap in our capabilities, moving from image-based video generation to full 3D dynamic video synthesis.”

This isn’t the primary time Stability AI has worked on 3D video. In March the corporate introduced Stable Video 3D, which cann take images of objects and produce rotating 3D videos of those objects based on the image.

Unlike SV3D, the brand new Stable Video 4D adds to its capabilities so it will possibly handle an object’s motion. Just like the SV3D model, SV4D must interpret the parts of the thing it cannot see to supply the essential additional perspectives. It also must reproduce invisible motion – resembling what is perhaps blocked from view — by understanding the thing and its components.

“The important thing features that enabled Stable Video 4D are that we combined the strengths of our previously-released Stable Video Diffusion and Stable Video 3D models, and fine-tuned it with a fastidiously curated dynamic 3D object dataset,” Varun Jampani, team lead of 3D Research at Stability AI, told VentureBeat in an interview.

In keeping with the researchers, SV4D is currently able to generating five-frame videos across eight perspectives in about 40 seconds, with all the optimization process taking around 20 to 25 minutes. The research team said that using a brand new approach to multiview diffusion by constructing on its previous work, it has produced a model that may faithfully reproduce 3D video across each frames and different perspectives.

Although the model remains to be within the research stages, Stability AI said, SV4D will probably be a major innovation for movie production, augmented reality, virtual reality, gaming and other industries where dynamic views of moving objects can be needed.

The model is currently available for developers and researchers to view and use on Hugging Face. It’s the corporate’s first video-to-video generation model, though it’s still under development as Stability AI continues to refine the model with higher optimization to handle a wider range of real-world videos beyond the synthetic datasets it was trained on.

Images: Stability AI