OpenAI’s video generation tool Sora took the AI community by surprise in February with fluid, realistic video that seems miles ahead of competitors. However the rigorously stage-managed debut overlooked quite a lot of details — details which have been filled in by a filmmaker given early access to create a brief using Sora.
Shy Kids is a digital production team based in Toronto that was picked by OpenAI as certainly one of a number of to supply short movies essentially for OpenAI promotional purposes, though they got considerable creative freedom in creating “air head.” In an interview with visual effects news outlet fxguide, post-production artist Patrick Cederberg described “actually using Sora” as a part of his work.
Perhaps a very powerful takeaway for many is solely this: While OpenAI’s post highlighting the shorts lets the reader assume they roughly emerged fully formed from Sora, the fact is that these were skilled productions, complete with robust storyboarding, editing, color correction, and post work like rotoscoping and VFX. Just as Apple says “shot on iPhone” but doesn’t show the studio setup, skilled lighting, and color work after the very fact, the Sora post only talks about what it lets people do, not how they really did it.
Cederberg’s interview is interesting and quite non-technical, so for those who’re interested in any respect, head over to fxguide and browse it. But listed here are some interesting nuggets about using Sora that tell us that, as impressive because it is, the model is maybe less of an enormous step forward than we thought.
Control continues to be the thing that’s essentially the most desirable and in addition essentially the most elusive at this point. … The closest we could get was just being hyper-descriptive in our prompts. Explaining wardrobe for characters, in addition to the form of balloon, was our way around consistency because shot to shot / generation to generation, there isn’t the feature set in place yet for full control over consistency.
In other words, matters which are easy in traditional filmmaking, like selecting the colour of a personality’s clothing, take elaborate workarounds and checks in a generative system, because each shot is created independent of the others. That would obviously change, however it is definitely far more laborious in the intervening time.
Sora outputs needed to be watched for unwanted elements as well: Cederberg described how the model would typically generate a face on the balloon that the principal character has for a head, or a string hanging down the front. These needed to be removed in post, one other time-consuming process, in the event that they couldn’t get the prompt to exclude them.
Precise timing and movements of characters or the camera aren’t really possible: “There’s just a little little bit of temporal control about where these different actions occur within the actual generation, however it’s not precise … it’s sort of a shot in the dead of night,” said Cederberg.
For instance, timing a gesture like a wave is a really approximate, suggestion-driven process, unlike manual animations. And a shot like a pan upward on the character’s body may or may not reflect what the filmmaker wants — so the team on this case rendered a shot composed in portrait orientation and did a crop pan in post. The generated clips were also often in slow motion for no particular reason.
Example of a shot because it got here out of Sora and the way it ended up within the short. Image Credits: Shy Kids
The truth is, using the on a regular basis language of filmmaking, like “panning right” or “tracking shot” were inconsistent generally, Cederberg said, which the team found pretty surprising.
“The researchers, before they approached artists to play with the tool, hadn’t really been considering like filmmakers,” he said.
Consequently, the team did lots of of generations, each 10 to twenty seconds, and ended up using only a handful. Cederberg estimated the ratio at 300:1 — but in fact we might probably all be surprised on the ratio on an peculiar shoot.
The team actually did just a little behind-the-scenes video explaining a number of the issues they bumped into, for those who’re curious. Like quite a lot of AI-adjacent content, the comments are pretty critical of the entire endeavor — though not quite as vituperative because the AI-assisted ad we saw pilloried recently.
The last interesting wrinkle pertains to copyright: Should you ask Sora to offer you a “Star Wars” clip, it’ll refuse. And for those who attempt to get around it with “robed man with a laser sword on a retro-futuristic spaceship,” it’ll also refuse, as by some mechanism it recognizes what you’re attempting to do. It also refused to do an “Aronofsky type shot” or a “Hitchcock zoom.”
On one hand, it makes perfect sense. However it does prompt the query: If Sora knows what these are, does that mean the model was trained on that content, the higher to acknowledge that it’s infringing? OpenAI, which keeps its training data cards near the vest — to the purpose of absurdity, as with CTO Mira Murati’s interview with Joanna Stern — will almost definitely never tell us.
As for Sora and its use in filmmaking, it’s clearly a strong and great tool as a substitute, but its place is just not “creating movies out of whole cloth.” Yet. As one other villain once famously said, “that comes later.”
