A dAIrectorial debut

My dreams of being an artist have somewhat suffered because I can’t draw, paint, or direct a story, and I certainly do not have any of the tools to bring my ideas to life.

Luckily for me (and my fellow artistically impaired people out there), AI is finally here to do the heavy lifting.

From one of the startups that brought us text-to-image model Stable Diffusion comes Runway AI Gen-2.

Unlike its predecessor, which required preexisting data to generate style-transfer videos, Gen-2 allows users to create novel videos from scratch by typing in prompts.

Prompt: “The late afternoon sun peeking through the window of a New York City loft” / Video credit: Runway

This has opened up a myriad of use cases such as creating videos from images and text, making storyboards from mockups, and automatically modifying objects in clips with text prompts.

Its videos aren’t photorealistic yet and the most it can do are three-second loops. But filmmakers can mix and match their own clips with AI-generated ones to foster a new viewing experience. They can also use the tech as a simple starting point for more complex projects.

In any case, it seems that Runway is aware of how to market its tech’s storytelling potential – the company sponsored the inaugural AI Film Festival in the US.

Prompt: “Aerial drone footage of a mountain range” / Video credit: Runway

Runway isn’t the first company to attempt text-to-video. Meta’s Make-A-Video and Google’s Imagen Video were both announced last year though have not yet been publicly released.

Though Runway’s program isn’t available for access at the moment, users can join a waitlist via the company’s Discord channel. The firm assures that the program will be made public “in the coming weeks.”

Along with the benefits this may bring to novice and pro filmmakers, this tech could also be misused. The tool can be made to generate compromising imagery and spread disinformation.

For the time being, it seems that humans are closer than ever to visualizing their thoughts, and Gen-2 will undoubtedly open up more opportunities for the storytellers of the world.