Meta advances generative AI video creation with Movie Gen

Meta Platforms Inc.’s artificial intelligence research team has showcased a brand new family of generative AI models for media that may generate and edit videos from easy text prompts.

Though the models are still a piece in progress, the corporate said they are going to provide the muse of latest video creation features set to look in Facebook, Instagram and WhatsApp next 12 months. The Meta Movie Gen models will enable users to create high-quality HD videos and pictures, edit those creations, generate audio and soundtracks, and even embed their very own likeness inside them, the corporate said.

In a blog post, Meta’s AI team explained that it’s aiming to usher in a brand new era of AI-generated content for creators on its platforms. The Meta Movie Gen models construct on the corporate’s earlier work in generative AI content creation, which began with its “Make-A-Scene” models that debuted in 2022, enabling users to create easy images and audio tracks, and later videos and 3D animations. Meta’s later Llama Image foundation models expanded on this work, introducing higher-quality images and videos, in addition to editing capabilities.

“Movie Gen is our third wave, combining all of those modalities and enabling further fine-grained control for the individuals who use the models in a way that’s never before been possible,” Meta’s AI team said in a blog post.

In accordance with Meta, the Movie Gen collection is made up of 4 models that enable video generation, personalized video generation, precise video editing and audio generation.

The primary of the models, Video Generation, is a 30 billion-parameter transformer model that’s in a position to generate videos of as much as 16 seconds in duration at 16 frames per second from prompts that might be easy text, images or a mix of the 2. Meta explained that it’s built on a joint model architecture that’s optimized for each text-to-image and text-to-video, and features advanced capabilities comparable to the power to reason about object motion, subject-object interactions and camera motion, so it could actually replicate more realistic motion within the videos it produces.

The Personalized Videos model is a bit different, because it’s specifically designed to take a picture of the user and create videos starring them, based on the user’s text prompts.

Meta explained that the identical foundational transformer model was used as the premise of its Precise Video Editing model. To make use of it, the user simply uploads the video they need to edit, together with a text input that describes how they need it to be edited, and the model will do the remainder.

It’s all about enabling more precision for creators, who can use it so as to add, remove or swap out specific elements of a video, comparable to the background, objects within the video, or style modifications, the corporate said. It does this while preserving the unique video content, targeting only the relevant pixels.

As for the Audio Generation tool, this is predicated on a 13 billion-parameter audio generation model that may take each video and text inputs to create high-fidelity soundtracks of as much as 45 seconds. It’s in a position to generate ambient sound, sound effects and instrumental background music, Meta said, and synchronize this with the content within the video.

Still a piece in progress

Meta hasn’t said anything about if or when it would make the Meta Movie Gen models available for others to make use of, but the corporate generally opts to open-source its AI innovations, comparable to its Llama models. So it’s likely that it won’t be long until developers will have the option to begin experimenting with them.

Once they do launch, Meta Movie Gen will go head-to-head with various other video generation models, comparable to Runway AI Inc.’s Gen-3 Alpha Turbo, OpenAI’s upcoming Sora, Google DeepMind’s Veo, Adobe Inc.’s Firefly, Luma AI Inc.’s Dream Machine and Captions LLC’s video editing tools.

The corporate is confident it could actually compete with those rivals. It individually published a research paper for many who need a more exhaustive deep dive into the inner workings of the Meta Movie Gen models. Within the paper, it claims various breakthroughs in model architecture, training objectives, data recipes, inference optimizations and evaluation protocols, and it believes these innovations enable Meta Movie Gen to significantly outperform its competitors.

That said, Meta concedes that there’s still quite a lot of room for improvement in its models, and it’s planning on making further optimizations to diminish inference time and improve the standard of the videos it generates.

Holger Mueller of Constellation Research Inc. said generative AI has already revolutionized the best way people write text, create images, understand documents and fix code, and the industry is now turning to the harder task of video creation.

“Creating film and video is a slow and expensive process that costs numerous money,” Mueller said. “Meta is promising to provide creators a faster and far more cost-effective alternative with Meta Movie Gen, and it could potentially democratize movie creation. If it does, it’ll likely send just a few shockwaves across the standard movie industry.”

Meta said the following steps involve working closely with filmmakers and other creators to integrate their feedback into the Meta Movie Gen models, with the goal being to give you a finished product that’s ultimately destined to look on platforms like Facebook and Instagram.

“Imagine animating a ‘day within the life’ video to share on Reels and editing it using text prompts, or making a customized animated birthday greeting for a friend and sending it to them on WhatsApp,” the corporate said. “With creativity and self-expression taking charge, the chances are infinite.”

Featured image: SiliconANGLE/Microsoft Designer