FACIAL RECOGNITION WITH ADVANCED EMOTIONAL STATES: A CNN-RF AND CNN-LSTM HYBRID FRAMEWORK FOR STRESS, FRUSTRATION, AND CONFIDENCE

Abstract

Emotional health is important in human communication, affecting social relationships, decision-making, and social interactions. This study explores facial emotion recognition (FER) using Paul Ekman’s basic emotion model, incorporating both the FER2013 dataset and an extended dataset, EmoFace (own dataset), which introduces "contempt" to analyze conflict resolution and social dynamics. This study leverages a Haar Cascade frontal facial classifier for efficient facial feature extraction, addressing the challenges posed by real-world images with varying orientations and backgrounds. The EmoNxtSeq hybrid fusion approach integrates CNN-RF to analyze confidence levels, whereas a CNN-LSTM fusion method is used to assess frustration and stress. The experimental results demonstrate that the model effectively identifies emotional states, highlighting the correlation between confidence, frustration, and stress during presentations. The findings offer insights for applications in mental health monitoring, public speaking training, and real-time feedback systems.