Facial and Vocal Emotion Elicit Partially Shared Emotion-Sensitive Neurons in Multimodal Foundation Models

Emotion recognition is inherently multimodal, drawing on both facial expressions and vocal cues. While modern multimodal foundation models can process images, speech, and text within a unified interface, little is known about whether they represent facial and vocal emotion through shared internal mechanisms or through modality-specific pathways. We provide a mechanistic analysis of emotion-sensitive neurons in three multimodal foundation models: Gemma-4-12B-it, MiniCPM-o-4.5, and Qwen2.5-Omni-7B. Using facial expression recognition (FER) and speech emotion recognition (SER) as complementary probes, we identify visual and acoustic emotion-sensitive neurons in decoder MLPs from FER and SER activations, respectively. First, we show that FER-derived V-ESNs are sparse and causally meaningful: deactivation selectively degrades recognition of the associated facial emotion, whereas activation steering shifts model predictions toward the target emotion. Second, V-ESNs and A-ESNs exhibit a consistent emotion-matched overlap pattern across models, indicating partial alignment between visual and acoustic affective representations. Third, cross-modal interventions reveal bidirectional causal transfer: A-ESNs identified from speech affect facial emotion recognition, and V-ESNs identified from faces affect speech emotion recognition. Compared with random neuron controls, these transferred interventions produce stronger and more emotion-specific effects. Our findings suggest that multimodal foundation models contain sparse decoder-level affective components that are partially shared across face and voice, offering a mechanistic account of cross-modal emotion recognition and a training-free route for controlling affective behavior.