talking heads
An augmented reality experience that lets you talk music with the artists behind the records
.png)
Talking heads is an AI powered system for bringing record sleeves to life. Now more than ever the personalities behind the music play an important role in fans engagement. As fans look to connect on a personal level with the personalities behind the art, talking heads gives a new medium to here the stories behind the music as if from the artist themselves.
Records that have a portrait on the front of them can be scanned and their face brought to life and spoken to, letting fans hear the stories behind each album from the musicians themselves. Each record is trained on background information for each individual artist and the specific album in question, and the avatars voice emulates the artists own voice through partnerships and custom model training.
How it works
First audio from a user is fed into the pipeline, to begin the dialogue. This audio is translated into text and a response is generated as text by an LLM. The LLM is fine-tuned based on specific context about the album and artist. Any response text is then fed into a text-to-speech model which can then be input alongside the album artwork to a custom Comfyui pipeline that creates facial movement in correpsondence with audio. Finally this is all mediated through touch designer to allow a looping video to fill any buffer time while the user is asking their question. This is a prototype for an AR experience that would currently need to be run on the cloud to leverage the compute needed for the stable diffusion pipeline to run in near real-time.
