Hypercinema - Week 7 & 8 (24-10-29 & 24-11-05)

Does AI work with us? Or do we work with AI?

This synthetic media assignment challenged us to create a short film using AI generated content. This vague description allows us to explore different tools (generative text, text to image, text to video, image to video, etc...) and approach the assignment however we see fit for the story.
I experienced some mixed feelings when we first started this assignment, mostly rooted in hesitations surrounding the ethics of AI generated art and finding the balance between me as the creative vs the machine as the technician. 

An image of a "foggy city alleyway in the style of neon cyberpunk" created through Stable Diffusion

Process:

A "shaman prophet in the mountains in a muted hazy dream like environment" created through Stable Diffusion

During our brainstorm sessions, we agreed to a central concept of a dystopian world that is narrated by a prophet like figure, and is accompanied by visuals of a lucid dreamscape about a city in which we are not certain we want to live in anymore. 
​
Then we organized the AI technology that we were interested in, have experience with, and want to explore more, which brought us to the following list.
​
Music/Voice: Suno
Photos: Stable diffusion and MidJourney
Videos: Stable diffusion video and Runway

Our production process consisted of:

1. create a song
2. create a script
3. create stills from the script
4. create videos from the stills
5. create transition videos
6. editing in Premiere Pro

A more detailed breakdown of how the video generation worked is in the sketch to the right. This technique was primarily used in the beginning of the video, and as we progress through the song, we switched to fading in and out of various scenes to portray a dream-like state.

Song Lyrics

[Verse 1]
Nights alive under neon scars
Buildings whisper tales and bars
Shadows climb like broken dreams
City's pulse or so it seems

[Verse 2]
Skyrim crumbles in the haze
Lost in labyrinths and daze
Sirens wail like ghostly cries
Steel and glass under city skies

[Chorus]
Concrete jungle whispering fear
Echoed footsteps drawing near
Hollow alleys scream the truth
Eroding innocence and youth

[Bridge]
Fault lines crack beneath our feet
Heartbeat sync with subway beat
Glitter fades in morning light
Still we roam this endless night

[Verse 4]
Clocks don't stop
We live on time
Hear the bells of midnight chime
City breathes in shadows deep
In its arms we wake and sleep

[Verse 1 Repeat]
Nights alive under neon scars
Buildings whisper tales and bars
Shadows climb like broken dreams
City's pulse or so it seems

Music Video Script

[Opening Scene]

• Close-up of a digital clock: "11:59 PM." The numbers glitch as it strikes midnight, chiming once, ominously.
• The camera pulls back, revealing a sprawling cityscape - a fusion of neon lights, shattered glass, and flickering holograms. Sirens wail in the distance as a figure darts through a maze of empty alleys, wrapped in fog

[Scene 2: Establishing Mood]

• Overhead shot of the figure weaving through labyrinthine alleys, fog swirling at their feet.
• Holographic ads blink on and off, illuminating the figure’s face in eerie flashes. The ads glitch, showing fragments of cryptic messages: "The future is watching." "No escape."

[Scene 3: Suspense Build-Up]

• Quick cuts of the figure’s frantic movements, footsteps echoing as shadowy forms begin to close in, just beyond the edges of light.
• The walls glitch, flickering as if alive, displaying distorted graffiti: "No way out."
• The figure speeds up, breath shallow, darting down darker and darker alleys.

[Climax: Facing the Pursuers]

• The figure reaches a dead end and turns, back to the wall, facing the shadows closing in. The pursuers—digital phantoms—surround them, flickering in and out like lost souls.
• Figure (whispering): It’s all just code...
• They look down, pulling up their sleeve to reveal a countdown timer embedded in their arm: "00:00."
• They close their eyes, resigned. Silence. The screen fades to black as the final heartbeat syncs with the city’s pulse, then dies out.​​

[End Scene]
Visual and Sound Design
Colors: Neon blues, greens, and reds contrasted with shadowed grays.
Soundtrack: Haunting synth, low sirens, echoing footsteps, rhythmic heartbeat.
Final Cue: Last heartbeat echoes, then silence.

Footage generation (1 of the methods)

1. We took this image and put it through Runway...

...To create the following video.

2. Then with the last frame of the video...

...We stitched it with a different still.

This technique was used to generate transition videos in hope to create a seamless transition between the stills. We almost got a video going from a city avenue and turning in to an alleyway. 

A strong emphasis on the word "almost". It appears that Runway morphs the city streets into an alleyway, rather than creative a logical transition between states. Its as if Runway has interpreted "turning" as a transformation, like "turning into", rather than a rotational change in orientation, "turn towards".

Is this a limitation in the LLM? Or are we supposed to use more specific wording? We had discussed this in ICM - the computer does not interpret ambiguity, you have to be specific and concise.

So, should we be changing our prompting habits to adapt to the computer? Or should we be altering the program to better fit out linguistic usage?

Key takeaways from this project:

• The better quality the input, the better output you get from Runway. For us, the combination of Stable Diffusion as input content worked really well for us in generating decent and consistent footage from Runway.
• Unless you are really good at prompting AI, it is really hard to get exactly what you want from the generators. So we really leaned into the absurdity of what these LLMs can create.
• I have come to terms that AI art is just another art medium and that just like any other (paint, charcoal, digital graphics, etc...) it is a skill that needs to be honed. Except, instead of honing your control over a fixed medium, you are altering and adapting the medium, in code form, to meet your needs.