Work

Poster Unvoiced: Designing an Unvoiced User Interface using Earables and LLMs

Year: 2024

Type: conference-paper

Abstract: This poster presents the design and implementation of Unvoiced, a silent speech interaction system. Unvoiced transforms subtle jaw movements into rich speech spectrograms, enabling seamless and private device interaction. Our system captures low-frequency jaw motion signals using ear-worn IMUs and translates them into high-fidelity mel-spectrograms through cross-modal translation techniques. By incorporating phonetic, contextual, and syntactic information, Unvoiced generates high-fidelity spectrograms that existing speech recognition systems can process. In our evaluation with 19 users across four common tasks, Unvoiced achieved a remarkable >94% task completion rate and <9% Word Error Rate (WER) for over 90% of phrases, maintaining robust performance even in noisy conditions.

Source: ACM Conference on Embedded Networked Sensor Systems (SenSys)

DOI: 10.1145/3666025.3699413

Authors: Tanmay Srivastava, Prerna Khanna, Shijia Pan, Phuc Nguyen, Shubham Jain

Institutions: Stony Brook University, University of California, Merced, University of Massachusetts Amherst

Language: English

Topic: Speech and Audio Processing

Subfield: Signal Processing

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funders: National Science Foundation, Hellman Foundation

Awards: Collaborative Research: CSR:Small:Sustainable Wearable Sensing System for Continuous Tree Health Monitoring, Collaborative Research: CCSS: Continuous Facial Sensing and 3D Reconstruction via Single-ear Wearable Biosensors, CAREER: Closed-loop Health Behavior Interventions in Multi-device Environments