< Program

Special Session: Artificial Intelligence and Hearing Healthcare

Driving Hearing Devices Using Conversation and Communication Statistics
Christi Miller, PhD, CCC-A
Meta, Reality Labs Research, Seattle, WA

An Augmented Reality (AR) platform is a system of interdependent technologies (e.g., audio, eye-tracking, computer vision, etc.), which enable digital objects to be placed in our real-world surroundings. These digital objects may provide assistance by overlaying enhancements to natural auditory objects in the scene, but the classic hearing device problems of estimating listener effort and identifying signals-of-interest remains. An AR platform in the form of glasses could support a large number of widely spaced microphones, forward and eye-facing cameras, inertial measurement units and other motion tracking hardware, and many other sensors. These sensors could be used to shed light on what sounds a listener wishes to hear, and whether they are having difficulty hearing them, but only if this information is optimally combined with a deeper understanding of natural conversation behavior.

To this end our team has taken advantage of an AR glasses platform to create a number of egocentric datasets capturing conversation in difficult listening situations, utilizing similar types of data that future AR hearing devices could be able to capture. In a recent study, we used this approach to study the effects of noise level and hearing loss on communication behaviors. Communicators with and without hearing loss were recruited in groups (i.e., they were familiar with one another), and participated in a 1-hour conversation while background levels randomly varied in a mock restaurant space. A glasses research device, Aria, collected egocentric data with a variety of sensors (i.e., microphones, forward-facing cameras, eye-tracking cameras,inertial measurement units), combined with close-talk microphones.  Hypotheses were established a-priori about how behavior would change with increases in noise level and/or hearing loss, and regarded metrics from voice activity, head motion/position, and eye gaze. The data is being analyzed using human and automated annotations, combined with statistical and machine learning approaches with the eventual goal of leveraging these statistics to better understand what signals listeners wish to hear and how much difficulty they are having in understanding and communicating.

Christi Miller has spent her career improving the lives of individuals with hearing challenges, with roles spanning clinical practice and supervision, didactic teaching, and leading research programs in academia and industry. She is currently a senior research scientist at Meta on the Reality Labs Research - Audio team.