OpenAI’s Sora gave us a glimpse earlier this year of how generative AI is going to change video editing – and now Adobe has shown off how that’s going to play out by previewing of some fascinating new Premiere Pro tools.
The new AI-powered features, powered by Adobe Firefly, effectively bring the kinds of tricks we’ve seen from Google’s photo-focused Magic Editor – erasing unwanted objects, adding objects and extending scenes – to video. And while it isn’t the first piece of software to do that, seeing these tools in an industry standard app that’s used by professionals is significant.
For a glimpse of what’s coming “this year” to Premiere Pro and other video editing apps, check out the video below. In a new Generative panel, there’s a new ‘add object’ option that lets you type in an object you want to add to the scene. This appears to be for static objects, rather than things like a galloping horse, but it looks handy for b-roll and backgrounds.
Arguably even more helpful is ‘object removal’, which uses Firefly’s AI-based smart masking to help you quickly select an object to remove then make it vanish with a click. Alternatively, you can just combine the two tools to, for example, swap the watch that someone’s wearing for a non-branded alternative.
One of the most powerful new AI-powered features in photo editing is extending backgrounds – called Generative Fill in Photoshop – and Premiere Pro will soon have a similar feature for video. Rather than extending the frame’s size, Generative Extend will let you add frames to a video to help you, for example, pause on your character’s face for a little longer.
While Adobe hasn’t given these tools a firm release date, only revealing that they’re coming “later this year”, it certainly looks like they’ll change Premiere Pro workflows in a several major ways. But the bigger AI video change could be yet to come…
Will Adobe really plug into OpenAI’s Sora?
The biggest Premiere Pro announcement, and also the most nebulous one, was Adobe’s preview of third-party models for the editing app. In short, Adobe is planning to let you plug generative AI video tools including OpenAI‘s Sora, Runway and Pika Labs into Premiere Pro to sprinkle your videos with their effects.
In theory, that sounds great. Adobe showed an example of OpenAI’s Sora generating b-roll with a text-to-video prompt, and Pika powering Generative Extend. But these “early examples” of Adobe’s “research exploration” with its “friends” from the likes of OpenAI are still clouded in uncertainty.
Firstly, Adobe hasn’t committed to launching the third-party plug-ins in the same way as its own Firefly-powered tools. That shows it’s really only testing the waters with this part of the Premiere Pro preview. Also, the integration sits a little uneasily with Adobe’s current stance on generative AI tools.
Adobe has sought to set itself apart from the likes of Midjourney and Stable Diffusion by highlighting that Adobe Firefly is only trained on Adobe Stock image library, which is apparently free of commercial, branded and trademark imagery. “We’re using hundreds of millions of assets, all trained and moderated to have no IP,” Adobe’s VP of Generative AI, Alexandru Costin, told us earlier this year.
Yet a new report from Bloomberg claims that Firefly was partially trained on images generated by Midjourney (with Adobe suggesting that could account for 5% of Firefly’s training data). And these previews of new alliances with generative video AI models, which are similarly opaque when it comes to their training data, again sits uneasily with Adobe’s stance.
Adobe’s potential get-out here is Content Credentials, a kind of nutrition label that’s also coming to Premiere Pro and will add watermarks to clarify when AI was used in a video and with which model. Whether or not this is enough for Adobe to balance making a commercially-friendly pro video editor with keeping up in the AI race remains to be seen.