Artists like Claude Monet, Pierre-Auguste Renoir, and Edgar Degas were ridiculed, their work described by critics as “base,” “unfinished,” and the worst thing to ever happen to art. A commercial flop, the exhibition saw 3,500 visitors, who mostly sauntered by to express horror at the plain frames and individual brushstrokes.
About a decade later, Georges Seurat would start A Sunday Afternoon on the Island of La Grande Jatte. Seven feet tall and ten feet wide, Sunday would become the most famous example of an Impressionist offshoot technique, pointillism.
Sunday’s central conceit was simple—one detailed image of a bustling afternoon at a Parisian park on the Seine. If you looked closely, you could see distinct spots of color and light that zoomed out into parasols, instruments, hats, humans, and a monkey on a leash. Each image could be unraveled, deconstructed into individual dots—the pixels of an analog age. And there’s a direct throughline between Seurat and the Impressionists and Total Pixel Space, the winning film at this month’s Runway AI Film Festival (AIFF).
“Pixels are the building blocks of digital images, tiny tiles forming a mosaic,” the film’s velvety voiceover says. “Each pixel is defined by numbers representing color and position. Therefore, any digital image can be represented as a sequence of numbers…Therefore, every photograph that could ever be taken exists as coordinates. Every frame of every possible film exists as coordinates. Every face that could ever be seen exists as coordinates. To deny this would be to deny the existence of numbers themselves.”
Jacob Adler, who made Total Pixel Space, is a classically-trained musician and composer, a multidisciplinary artist rendered a filmmaker by advances in AI. Adler worked on the film for more than a year, generating tens of thousands of images along the way, inspired by Jorge Luis Borges’s short story “The Library of Babel” and the miracle of making sense in a random, vast world.
“I was fascinated by the act of generating these images, and it spawned all these philosophical questions,” said Adler. “In this vast combinatorial space of language, the overwhelming majority of combinations of letters are gibberish and nonsense. So, apply that to digital imagery: How many images can possibly exist? And how many of these images are incomprehensible noise? I tried expressing this idea in other media, and it just failed. But it came together as a short AI film.”
Runway, the $3 billion AI video startup, has hosted the AIFF since 2023 to showcase short films made with AI. This year’s festival—won by Total Pixel Space—marked a major leap: from 300 submissions in small NYC theaters in 2023 to a sold-out Lincoln Center show with 6,000 submissions, drawing an international crowd. Runway didn’t pick the winner—a panel of judges, including directors Harmony Korine and Gaspar Noe, made the call—but Total Pixel Space reflects how Runway is thinking about its own future: AI-generated experiences that don’t just tell stories but build worlds.
“We’re going to have all these new forms of media that go beyond film and games, that exist in all the spaces in-between,” said Anastasis Germanidis, Runway CTO and cofounder. “Some of it might look more like immersive theater productions, where there’s a fixed storyline, but you can kind of move around, experiencing it from different perspectives.”
Germanidis added: “Imagine these models get really good at generating realistic depictions of reality, and you have a world where you can essentially simulate most of what we care about as we navigate the world. That’s going to be both a very important piece of solving problems.
Germanidis is thinking about world simulation as a principle more than anything else; one that could be applied not just to stories, but to biology, robotics, and physics. It’s distinctly about finding ways to mimic not just humans, but physics and biology.
“We want to be able to simulate pretty much every instruction you have in the physical world,” said Cristóbal Valenzuela, CEO and cofounder of Runway. “We know that’s coming…AI labs have been very obsessed with simulating the human mind. But I think that might be the wrong approach long-term. What you want to do is not simulate how humans work, but how the world works.”
We’re seeing the beginnings of this strategy play out this week, as Runway plans to launch an interactive gaming experience, marking a push into the gaming market. The product right now is text and image generation, but is expected to become increasingly visual over time. How this all will ultimately lead to world-building applications is hazy—and that’s part of the point.
“If you have a predetermined way of getting there, it’s too late and it’s obvious,” Valenzuela said. “For me, it goes back to how creative [something is]…. If you’re not involved in creative acts, you don’t understand. Most people who have any form of creative expression within their work know that when they start, they don’t exactly know where they’re heading. You’re putting yourself in a very vulnerable position to just explore everything. Then, eventually you’ll know by experience that you will have to land somewhere.”
Runway has no shortage of competition in AI video generation—including but not limited to OpenAI’s Sora, Stability AI, Moonvalley, and Pika Labs. And Runway is in the position where they must continue to distinguish themselves in order to compete. The company has raised more than $500 million to date from investors like General Atlantic, SoftBank, Nvidia, Salesforce Ventures, Felicis, and Coatue. Meta reportedly approached Runway in an attempt to acquire the company before dropping billions on Scale AI this summer.
The AI “wake up” call
The history of art is a history of technological disruption, from the invention of the printing press to the advent of “talkies” in the 1920s. Job displacement is, of course, part of that story—and always has been.
“Before the printing press, it was all monks and people who knew how to share specific stories,” said Valenzuela. “Then, with the printing press, more people could read and write, which was treated as an apocalyptic event.”
This is true: When the printing press was invented in 1440 and adoption of the technology spread, religious authorities worried about losing control, and guilds of scribes were displaced. But a world of people could now read, and stories could scale.
Valenzuela brings up another example, this one infused with a comically droll element:
“Before alarm clocks were invented, you’d hire a guy who came to your door, at the time you wanted, and throw up a stone to your window,” said Valenzuela. “That was a job. What else were you going to do if you didn’t have family around and needed to wake up?”
In 19th-century Britain and Ireland, these people were called “knocker-uppers.” They’d tap on windows with long sticks or shoot peas at windows to wake workers for shifts. Once alarm clocks were invented, it became natural for people to just, well, use alarm clocks. As AI comes tapping at Hollywood windows, a trend that Valenzuela is directly involved in, the industry reaction has been fraught—even as people secretly use it.
“It’s been a little dirty secret, because whether it’s Runway or, you know, he does have a little competition,” said Michael Burns, vice chair at Lionsgate onstage at AIFF, gesturing to Valenzuela. “We believe that this tool is being used by everybody that doesn’t talk about the fact that they’re using it.”
Runway’s Germanidis says there are three phases of technological art evolution: getting the technology to work, imitating existing art forms, and then creating unique forms. We’re just starting to “enter that third stage with, like generative generative models,” he said. That’s not to say, of course, that everything should be AI—for Adler, an artist whose practice has fundamentally expanded with AI, is very clear that some things (like surrealistic images and philosophical concepts) are well-suited to AI, whereas other material (like complex human interactions) isn’t.
“I look at [AI] as a tool, but I don’t know yet if I’m convinced that it’s a new genre,” said Adler. “There are things I can produce with cameras that are impossible with AI and vice versa—things I can do with AI that are impossible with cameras.”
That alone is an incredible phenomenon that speaks to excitement, and fear, that Runway and its video AI rivals are already causing throughout the worlds of art, media, and entertainment. For Runway’s founders however, the real payoff of their AI vision, if they can pull it off, will extend far beyond the screen, existing as something spectacular, immersive—and probably unrecognizable.
Politics8 years ago