Emotion Recognition and Cognitive Load: Exploring Their Interconnected Dynamics in HCI

Abstract

Accurate assessment of human emotional states and cognitive load is essential for adaptive human-computer interaction in education, healthcare, and high-stress environments. However, existing studies have largely treated emotion recognition and cognitive load estimation as separate tasks, with limited research on their integrated analysis. This study proposes the Multimodal Emotion and Cognitive Load Interaction Framework (MECLIF), a unified system that models the cyclic interaction between emotion and cognitive load. MECLIF integrates deep learning-based emotion recognition with EEG-derived cognitive load estimation using a cyclic feedback loop, dynamically adjusting emotion recognition based on cognitive load fluctuations. The framework utilizes the IEMOCAP and CMU-MOSI datasets for text- and audio-based emotion analysis, while EEG signals from the SEED dataset are used for the estimation of cognitive load. Experimental results show that sustained emotional states increase cognitive load, which in turn degrades the accuracy of emotion recognition. A self-reinforcing feedback loop emerges, where recognition errors further amplify cognitive strain. These findings highlight the critical need to incorporate cognitive load considerations into emotion recognition systems, as neglecting their interaction can limit performance. Integrating cognitive load estimation into emotion recognition is essential to improve the accuracy and usability of human-computer interaction in education, healthcare, and high-stress decision-making environments. Future research will design dedicated experiments and construct new datasets to enable real-time monitoring and application of emotion-cognition interactions.