EchoSpace-AR

Overview

Detects surrounding sounds and renders a directional in-headset icon that moves toward the sound source to improve environmental awareness—especially for Deaf and hard-of-hearing users.

Problem

Deaf and hard-of-hearing users lack environmental audio awareness when using AR/VR headsets, missing important sounds like alarms, speech, or approaching vehicles.

Solution

An AR system that classifies surrounding sounds using a fine-tuned YAMNet model and renders directional icons in the headset. Speech-to-text is handled by OpenAI Whisper for live captions.

Highlights

• •Speech-to-text via OpenAI Whisper for live captions
• •Sound classification using fine-tuned Google YAMNet-based model
• •Evaluation pipeline: Python analysis for frequency content and classification metrics
• •Next goals: lower latency, multi-language captions, Apple Vision Pro support, portable form factors

Tech Stack