World Melody
A live instrument that transforms what a camera sees into music — the image itself becomes the spectrum, a local vision LLM interprets the scene like a poet, and a music model generates the score.
Live experiment
World Melody — live from the studio workstation
The picture itself, sounding.
World Melody runs live on the studio workstation — vision, language and music models, all local. Open the camera or drop in a video, and what the camera sees becomes what you hear.
The instrument below runs live on the studio workstation (RTX 4090 — vision, language and music models all local). Open the camera or drop in a video and the world starts to play. If it shows offline, the studio machine is asleep. Camera permission works best in its own tab:
The picture itself, sounding
World Melody is not a soundboard triggered by labels. The frame is treated as a spectrum and resynthesized continuously, about fifteen times a second: where the image is still you hear a quiet hum, and the exact region that moves is the region of the sound that blooms.
- Each image row is a frequency band — ground low, sky high
- Brightness sets a band's amplitude; the horizontal position of light places it in stereo
- Colour shapes timbre: warm and saturated regions ring richer
- Detected entities add resonance where they stand; a relation between two of them couples their bands so they breathe together
Three interpreters of one world
- Image field — the direct, continuous pixel-to-spectrum transformation
- Poet-composer — a local vision LLM watches, writes a one-line poetic reading of the scene through a selected worldview, and composes key, tempo and a voice for every visible thing; entities keep their themes across sessions
- Neural score — MusicGen generates music from the understood meaning the way an LLM generates sentences from tokens; the same understood world always returns the same music
Hands into Blender
The same perception stack drives a 3D control mode: MediaPipe hand tracking on a phone camera steers Blender over a WebSocket add-on. Point to place the cursor, tap to select, pinch to grab and carry an object (hand distance controls depth), open palm to orbit, two hands to zoom — with grab results reported back to the phone.
Everything runs locally on one workstation: YOLO26 segmentation and pose, a qwen3 vision LLM via Ollama, MusicGen for generation, and a WebAudio engine that mixes real sampled instruments with synthesis. A live-audience counter and a GPU guard keep multiple viewers from slowing each other down.