Milo — A Local-First Talking Character

A character, not an assistant

Milo is designed as a conversational character with a face, moods, opinions and persistent memory for each visitor. He listens through the microphone, answers with a natural voice and animates every reply with real-time facial expressions and phoneme-driven lip sync.

• Local Ollama language model by default
• Local Whisper speech recognition
• Local Kokoro neural voice generation
• Optional OpenAI-compatible cloud models with the visitor's own key
• WebGL 3D face with emotion and lip-sync animation
• Separate persistent memory for each browser

Technical architecture

The browser renders Milo with Three.js and WebGL from a rigged glTF character containing 51 facial morph targets. A local FastAPI service maintains the encrypted WebSocket session, coordinates language, speech and animation, and returns timed phoneme cues that drive real-time lip sync and expressions.

• Language: Ollama with gemma4:latest on the local workstation; optional DeepSeek deepseek-chat or OpenAI gpt-4o-mini using the visitor's session-only key
• Speech recognition: faster-whisper large-v3-turbo with CUDA FP16 acceleration and a CPU INT8 fallback
• Voice generation: Kokoro v1.0 ONNX with selectable voices; Edge TTS is used only as an automatic fallback
• Lip sync: Rhubarb phoneme timing mapped to the character's facial morph targets and emotion system
• Rendering: Three.js 0.160, WebGL, embedded glTF textures, HDR environment lighting and procedural eye reflections
• Infrastructure: local GPU workstation → restricted reverse SSH tunnel → Debian VPS/Nginx HTTPS gateway; AI models and conversation text never run on the VPS
• Memory: a random browser identifier keeps a separate local conversation history for up to 30 days; Clear memory removes the saved text and identifier
• Security: provider URLs are fixed server-side, requests are rate-limited, origins are restricted and cloud API keys remain only in the active encrypted WebSocket session

Model source and test-use notice

The current character head is derived from a MetaHuman test asset created with Epic Games' MetaHuman framework. This implementation is an experimental, non-commercial technology demonstration only. The character asset is not offered for sale, distributed as a downloadable product or included in any paid Blobidea package. MetaHuman, Unreal Engine and related marks belong to Epic Games. A future commercial release must use a separately verified and appropriately licensed production character.