A trip and a fall down memory lane, tracking the relationship of a couple from middle school into the afterlife. “The Hardest Part” is the winner of the AI in Art Award 2024.
Guest post by Paulo Trillo / Golden Nica AI in Art Award 2024
The music video for The Hardest Part by Washed Out is a groundbreaking project, being the first full generative video made with OpenAI’s SORA text-to-video model. The song is about moving on from a lost love, and I wanted to honor this theme while putting a unique spin on it. The video spans several decades, starting in the early 80s, and follows a young couple who meet in middle school. We see them age, go through changes, grow up together, fall in love, and eventually face the challenges of life not going as planned.
I aimed to lean into the hallucinatory, dream-like qualities of SORA rather than depict something entirely real. It’s about trying to preserve someone in time, with memories that are fleeting and hard to grasp. The surreal blends of environments and impossible transitions—from cars to buildings to landscapes—create a fluid drift through the subconscious, attempting to hold onto what’s real. AI has an ephemeral and familiar quality, allowing us to tap into a nostalgia for something that never happened, known as “anemoia”. I wanted to use AI from this conceptual standpoint, using technology to drum up dreams and memories of something we can never have.
AI models are like memory banks and time capsules, so the idea came from exploiting that aspect. Rather than being truly authentic to reality, AI creates something new—an uncanny reflection of our reality that deceives us into believing something that never happened. The aesthetic is not something you could capture with cameras or animate; it’s entirely new, resembling a dream or memory. At first glance, it appears real, but the details are murky and missing. Conceptually, tying the first SORA music video to this journey through memories felt appropriate.
The project was ambitious, traversing time and space in impossible ways. The idea of an infinite zoom through time following a couple in love came to me about 10 years ago, but I abandoned it, thinking it too ambitious for a music video. Between the aging characters, the numerous locations and time periods, and the visual effects, it wasn’t feasible for a music video budget. AI allows us to resurrect these ideas from the graveyard and breathe new life into them. In a way, the music video tells a similar story about a woman dreaming of her ex-lover who is no longer here. The lyrics suggest moving on from the past, letting go of nostalgia, and accepting that there is no going back. This mirrors the mixed feelings about embracing new technology and how things won’t be the same going forward. Also, the song’s theme of learning to let go of a loved one deeply resonated with me, and I infused my personal experiences of loss into the narrative.
When working with SORA or any AI tool, it’s important to retain your perspective and tone of voice. Setting constraints or formal stylistic rules for a film helps define that voice. It’s easy to let the machine dominate and dictate the aesthetic or the choices in the details. Working with AI is about controlling chaos, but all filmmaking is controlled chaos.
While I let the machine’s hallucinations deviate from reality and create a surreal atmosphere, I guided it with script page length prompts describing each shot’s beat, the character design, mood, lighting, retro aesthetic, camera motion, and pacing. I was deliberate with the camera motion and ensured that each cut represented a leap forward in time. The infinite camera push or zoom technique and transitions between environments are techniques I’ve used in my work for years. It created a connection to my previous work while allowing me to do something entirely new with it. Moving this synthetic camera through an imaginary latent space opened up entirely new visual impossibilities that gives a glimpse at what the future of filmmaking might look like.
The process involved a lot of trial and error and exploration to get to the final product. I wrote out the beats of the idea, turned them into prompts, generated the clips, edited them into a sequence, saw what worked and what didn’t, and then generated more clips. It was a fluid back-and-forth between “pre-production” and post-production. This opened up a whole multiverse of routes I could take the video. It felt intimate, as if I was learning about the characters with each generation and getting glimpses into their lives that no one had seen. It was fascinating to explore, but knowing when to stop can be a trap when working with AI. In total, I generated nearly 700 clips and cut them down to about 55 for the final edit. Each shot was between 15 to 20 seconds long, so I ended up generating about 230 minutes, which was cut down to 4 minutes. Much of this was about crafting the prompt and getting SORA to do what was in my head, finding the right combination of words to create the desired effect.
One of the biggest challenges in creating this video was ensuring the continuity of the zoom effect and maintaining a consistent aesthetic across different clips. This required meticulous attention to detail and constant adjustments to the prompts used to guide SORA. By editing the prompts to control the speed and direction of the camera movements, I was able to create a cohesive narrative flow that felt natural and immersive. It was like finding a formula of words that worked, though sometimes changing a single word caused a ripple effect of changes.
While this project is exciting for being the first of its kind, I hope it has merit beyond the technology alone. Now that we have the ability to conjure ambitious ideas and resurrect old ones, there is an opportunity to push filmmaking into new territories of the surreal and strange. Using AI technology allowed me to create something I would have never created, to realize an ambitious idea and bring it to life in a way that feels both familiar and entirely new. I acknowledge that the novel use of AI is temporary and that there are long term changes we’re facing. There are valid criticisms regarding how these models were built on stolen data and the impact of AI on the creative industries.
I believe that AI should be seen as a complementary tool to the traditional filmmaking process, to enhance human-led creativity rather than replacing it.
Transparency and ethical development of these technologies are crucial to creating a sustainable path forward.
Paul Trillo
Paul Trillo (US) is a director and video artist who challenges both his own curiosity and illusion with his visually inventive, conceptual, and technical films. His diverse body of work spans various genres and formats, infusing meaning and purpose into his experimental uses of technology and technique.