Lee, Hansoo Kim, Eunseo Lee, Jeongmin Joo, Hyunsoo Kwak, So na 2025-12-15T05:31:28Z 2025-12-15T05:31:28Z 2025-11-20 2025-09-27 https://pubs.kist.re.kr/handle/201004/153789 The Experience Sampling Method (ESM) allows researchers to collect ecologically valid data on users’ daily experiences. However, conventional mobile ESM systems rely on visual interfaces (e.g., checkboxes, text fields), which can burden users in mobile or hands-free contexts. Voice-based alternatives address this but are often limited by rigid, rule-based interactions. We present SurveyTalk, a multimodal ESM system powered by large language models (LLMs) that enables real-time semantic mapping between speech and visual survey interfaces. The system supports voice-driven selection, clarification, navigation, and response interpretation, reducing user friction and improving data quality. Our approach enhances the accessibility, adaptability, and robustness of ESM in dynamic real-world settings. English ACM SurveyTalk: Voice-to-UI Mapping for In-Situ ESM via LLMs Conference 10.1145/3746058.3758377 1 UIST '25: The 38th Annual ACM Symposium on User Interface Software and Technology, pp.1 - 3 UIST '25: The 38th Annual ACM Symposium on User Interface Software and Technology 1 3 KO Busan Republic of Korea 2025-09-28 Adjunct Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology