Milo — A Google GNM Talking Character

A character, not an assistant

Milo is designed as a conversational character with a face, moods, opinions and persistent memory for each visitor. He listens through the microphone, answers with a natural voice and animates every reply with a neural, audio-driven facial performance.

• Local Ollama language model by default
• Local Whisper speech recognition
• Local Kokoro neural voice generation
• Optional OpenAI-compatible cloud models with the visitor's own key
• Google GNM Head v3 driven by NVIDIA Audio2Face 3D performance inference
• Separate persistent memory for each browser

Technical architecture

The browser renders Milo with Three.js and WebGL from Google GNM Head v3. GNM's 253-component identity space, all 383 native expression components, articulated eyes and internal anatomy are preserved in the production glTF. Each synthesized waveform is processed locally by NVIDIA Audio2Face 3D v2.3 Mark into a learned 30 FPS lower-face and tongue performance. A clean-slate GNM performance engine gives every anatomical region one owner: Audio2Face controls lips, jaw and tongue; the LLM-selected acting state controls brows, eyelids, cheeks and nose; an independent state machine controls anatomically timed blinks; and an event-based prosody track controls head delivery. The final pose is composed once in native 383-dimensional GNM space, preventing emotion or idle motion from reopening lip contacts or overriding the eyes.

• Language: Ollama with gemma4:latest on the local workstation; optional DeepSeek deepseek-chat or OpenAI gpt-4o-mini using the visitor's session-only key
• Speech recognition: faster-whisper large-v3-turbo with CUDA FP16 acceleration and a CPU INT8 fallback
• Voice generation: Kokoro v1.0 ONNX with selectable voices; Edge TTS is used only as an automatic fallback
• Facial performance: Audio2Face 3D v2.3 Mark generates a 30 FPS neural performance; only its lower-face and tongue channels own articulation, then the new performance engine composes them with LLM acting into all 383 native GNM expression components. Rhubarb remains an automatic fallback only
• Emotional acting: the language model selects one of 13 performance states—including playful, skeptical and disgusted. The new event-based actor converts that state and vocal stress into asymmetric brows, eyelids, cheeks, nose, gaze and head beats, while a separate blink state machine supplies full close-hold-open motion and Kokoro receives matching voice direction
• Rendering: Three.js 0.160, WebGL, Google GNM head/eyes/teeth/tongue geometry, surface-transferred skin and eye textures, and HDR environment lighting
• Infrastructure: local GPU workstation → restricted reverse SSH tunnel → Debian VPS/Nginx HTTPS gateway; AI models and conversation text never run on the VPS
• Memory: a random browser identifier keeps a separate local conversation history for up to 30 days; Clear memory removes the saved text and identifier
• Security: provider URLs are fixed server-side, requests are rate-limited and origins are restricted. Optional cloud API keys are encrypted in a local per-visitor vault, never returned to the browser and can be deleted from the interface.

Open model sources and licenses

The complete production head uses Google GNM Head v3: its topology, parametric identity, expression basis, articulated eyes, teeth, gums and tongue. The integration is pinned to an auditable upstream commit and the generated web asset records its provenance. GNM is Copyright 2026 Google LLC and is used under Apache License 2.0. Speech performance is inferred by NVIDIA Audio2Face 3D v2.3 Mark under the NVIDIA Open Model License; inference and retargeting run on the same private local workstation as the rest of Milo's AI stack.