Morph-Talk: Avatar-Based Video Conversation - A Browser-Based System Mapping Facial and Upper-Body Movements to VRM Avatars
Year: 2026
Type: article
Abstract: Video calls normally show the camera image of each participant. This paper presents Morph-Talk, a full-stack web application in which two users hold a video call while each appears to the other as an animated three-dimensional anime-style VRM avatar instead of the real camera feed. Each user's webcam video is used only locally: MediaPipe Holistic extracts face and body tracking information from it inside the browser, and this tracking data, not the video, is relayed to the other participant through Socket.IO. The receiving browser animates the corresponding avatar live with Three.js and three-vrm in a React.js interface, while the participants hear each other's voices during the call. Facial expressions are derived from geometric metrics measured against a calibrated neutral baseline and mapped to avatar expressions, while head orientation and arm movement are estimated from landmarks. A Node.js server with Socket.IO handles call signalling, with the call connection set up using WebRTC, and MongoDB is used for data persistence. We describe the architecture, the processing pipeline and the engineering problems met during implementation: differences between VRM versions, per-user calibration, secure-context browser requirements and audio device routing. The implemented features and the observations made during development are summarised. No quantitative accuracy, latency or frame-rate evaluation and no user study has been carried out; these are identified as future work.
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225465720