Adobe brings Firefly generative AI to video editing

Forward-looking: Ever since generative AI-based image creation took off, developers have played with the idea of using AI to generate videos. While AI can’t create coherent videos from scratch yet, Adobe is adding AI functions to its video editing tools to significantly change the production process.

Adobe is bringing its Firefly generative AI tools to Creative Cloud apps like Premiere Pro. Firefly adds multiple features that could help less experienced users create professional-looking videos.

Firefly can use text prompts to automatically edit a video based on what the AI sees in different shots. It can combine the feature with audio transcription (which it can use to automatically apply subtitles) to edit scenes according to spoken dialogue.

Adobe demonstrates AI-based editing with a clip showing a woman describing rock climbing preparation. The original clip is an unbroken shot of her talking in front of the camera, but Firefly picks up her mentions of putting shoes on and applying chalk and edits close shots of those actions on top of the corresponding dialogue.

Additionally, Firefly can use text prompts to change color grading. The demonstration shows phrases like “cool morning” and “golden hour” changing one scene to resemble two different times of the day. Another text prompt brightens a subject’s face.

Users can also add logos, graphics, fonts, or title cards with text prompts. Text and logos can be stylized and animated. Furthermore, Firefly can build royalty-free custom music and sounds to add to videos.

Perhaps most strikingly, Firefly can read a script to generate storyboards or edit B-rolls. Adobe’s video shows the tool creating simple drawn and CG visualizations based solely on the script’s text. Producers can re-arrange the text and quickly see the visual changes.

Adobe first revealed Firefly in late March as an AI image generator attached to apps like Adobe Express, Experience Manager, Photoshop, and Illustrator. The tool will eventually come to the Creative Cloud, Document Cloud, Experience Cloud, and Adobe Express workflows.

The company is taking multiple steps to avoid the controversy surrounding image generators like Stable Diffusion or Midjourney. Adobe is currently only training Firefly on Adobe Stock Photos, Public Domain content, and open-license material. Eventually, the company will give contributors an option to opt out of training its AI and will reveal a system to compensate stock photo makers.