Lightweight Multimodal Emotion Recognition with Cross-Attention Fusion for In-Vehicle Edge Deployment

Abstract

To address the contradiction between limited computational resources and high real-time requirements in in-vehicle environments, this paper proposes a lightweight multimodal emotion recognition method optimized for CPU deployment, fusing speech and facial expressions. The speech branch uses a three-layer convolutional network to extract log-Mel spectrogram features, and introduces a multi-scale global average pooling to enhance the global representation. The facial branch uses a pre-trained MobileNetV3-Small to extract spatial features. A lightweight bidirectional cross-attention module, Cross-Attention Fusion, is designed to project speech (128-D) and facial (576-D) features into a common 256-D space for interactive fusion, followed by a two-layer fully connected classifier for six emotion categories. On the RAVDESS dataset, the model achieves 90.57% accuracy with only 1.85M parameters, 0.107G FLOPs, and 15.39ms inference time on CPU. After INT8 quantization, accuracy reaches 89.62%, inference time reduces to 10.33ms, and model size compresses to 5.47MB. Experiments show that our method achieves high accuracy while being lightweight, low-latency, and CPU-friendly, making it suitable for in-vehicle edge deployment.