Adobe is positioning itself at the forefront of AI-powered video content creation with its upcoming Firefly Video model. By the end of 2024, users will have the opportunity to experiment with this cutting-edge technology, marking a significant step forward for Adobe in the realm of generative AI. These features are expected to be accessible through the Premiere Pro beta app—a widely used video editing platform by professionals—and a free-to-use website, ensuring that both seasoned creators and casual users can explore its capabilities.
The Firefly Video model includes three major features: Generative Extend, Text to Video, and Image to Video. Currently undergoing private beta testing, these tools are designed to automate and streamline tasks that would traditionally take hours or even days.
Generative Extend
This feature allows users to effortlessly extend any video by two seconds. For creators, extending scenes is often a tedious task, requiring manual editing, precise splicing, or even additional footage. Generative Extend simplifies this process by using AI to predict and create seamless transitions. It analyzes the last few frames of a video and automatically generates extra footage that aligns with the original video’s style, movement, and background. This could be especially useful for content creators who need to lengthen a shot for pacing or to accommodate additional narrative elements.
Text to Video and Image to Video
These two features are built around the idea of generating videos from text prompts or still images. While previous AI models for video generation have struggled with consistency and quality, Adobe is aiming to address these issues by refining the output, especially in areas like word accuracy and visual coherence.
For instance, the Text to Video feature enables users to input a written prompt—such as “a sunset over the ocean”—and get a short video that visualizes this scene. Meanwhile, the Image to Video tool allows users to input a static image and have the AI create a short, dynamic video based on that image. These features hold significant potential for speeding up video creation, reducing the time and effort needed to generate content for industries like advertising, social media, and entertainment.
The Bigger Picture
Generative AI, including video generation, has become a transformative force in creative industries. Tools like Firefly Video have the potential to redefine video production workflows by significantly reducing manual labor. Instead of requiring a deep understanding of video editing software and animation techniques, users will be able to simply describe or provide an image, and the AI will take care of the rest.
Adobe’s development of Firefly Video is part of a larger response to the rapid advancement of generative AI models, like OpenAI’s DALL·E for images and Runway’s Gen-3 Alpha for video. While these models have impressed with their ability to create content quickly, their unpredictability has limited their use in professional settings. Adobe, however, sees an opportunity to provide more reliable and controllable tools, thus ensuring their Firefly AI features can become valuable assets in both casual and professional workflows.
By integrating these AI features into familiar platforms like Premiere Pro and offering them through a free website, Adobe is positioning itself to not only appeal to its core base of professional users but also to attract a new wave of creators who may not have the technical skills but are drawn to the potential of AI-generated content.
Later this year, Adobe will add a new feature called Generative Extend to the Premiere Pro beta app. This tool lets users easily add two more seconds to any video. Two other features, Text to Video and Image to Video, will allow users to create five-second videos using text prompts or images. These will be available on Adobe.firefly.com, and the five-second limit may increase in the future.
Adobe has been popular with creative professionals for years, but these new AI tools could change how content is made. Firefly is Adobe’s answer to the rise of generative AI, like OpenAI’s Sora and Runway’s Gen-3 Alpha, which have impressed people by quickly creating video clips. However, these early AI tools are not yet reliable enough for professional use.
Adobe aims to differentiate itself by providing greater control over its AI tools. Ely Greenfield, Chief Technology Officer of Digital Media at Adobe, highlights a significant demand for Firefly’s AI capabilities to optimize and accelerate existing workflows. He also points out that the generative fill feature, introduced in Adobe Photoshop, has quickly become one of the most widely used tools in recent years.
While pricing details for Adobe’s AI video features have not yet been disclosed, the company currently offers Creative Cloud customers a set number of “generative credits” for its Firefly tools. Higher-tier subscription plans provide access to more credits, reflecting the scalability of these features across various user needs.
Generative Extend uses Firefly’s Video model to predict and seamlessly add two seconds of footage to the end of a video. It also recreates background sounds like traffic or nature but avoids replicating voices or music due to licensing rules.
Greenfield showcased this feature by extending a video of an astronaut in space. While there was a subtle cue marking the extension, the overall scene and motion remained smooth, making it useful for extending abrupt scene endings.
The Text to Video and Image to Video features generate up to five seconds of video from a text or image prompt. These tools will be available on the Firefly website with unspecified usage limits.
Adobe emphasizes that its Text to Video feature excels at accurately handling text, a common challenge for AI models.
Adobe is playing it safe with its Firefly video models, which are set up to block content featuring nudity, drugs, or alcohol. They’ve also made sure the models aren’t trained on images of public figures like politicians or celebrities, unlike some of their competitors.
