SurveyTalk: Voice-to-UI Mapping for In-Situ ESM via LLMs

Abstract

The Experience Sampling Method (ESM) allows researchers to collect ecologically valid data on users’ daily experiences. However, conventional mobile ESM systems rely on visual interfaces (e.g., checkboxes, text fields), which can burden users in mobile or hands-free contexts. Voice-based alternatives address this but are often limited by rigid, rule-based interactions. We present SurveyTalk, a multimodal ESM system powered by large language models (LLMs) that enables real-time semantic mapping between speech and visual survey interfaces. The system supports voice-driven selection, clarification, navigation, and response interpretation, reducing user friction and improving data quality. Our approach enhances the accessibility, adaptability, and robustness of ESM in dynamic real-world settings.