Before You Act: Benchmarking Interpretability in Vision-Language Models

Large vision-language models (VLMs) have achieved strong multimodal reasoning performance, yet their internal decision-making processes remain unclear. Mechanistic Interpretability (MI) aims to uncover the circuits and representations responsible for model behavior. Acting on these internals, whether to steer, edit, or audit a model, first requires knowing which MI methods recover genuine causal mechanisms rather than surface correlations, yet no such validation exists for multimodal systems. While benchmarks such as mechanistic interpretability benchmark (MIB) enable rigorous evaluation of MI methods in large language models, no standardized framework exists for multimodal architectures, so interpretability studies in VLMs remain difficult to compare and validate. In this work, we introduce Multimodal Mechanistic Interpretability Benchmark (MMIB), a benchmark for evaluating MI methods in VLMs. MMIB extends the dual-track framework of MIB to multimodal settings on two different models. The first track evaluates circuit localization using the Circuit Performance Ratio (CPR) and Circuit-Model Distance (CMD) metrics under modality-specific counterfactual intervention. The second track evaluates causal variable localization using interchange intervention accuracy (IIA). We also study a metric that computes the cross modal information flow. We introduce image- and text-based counterfactual inputs, enabling structured causal validation across both modalities. Across the settings we evaluated, the faithfulness metrics show promising consistency, demonstrating that mechanistic benchmarking extends beyond language-only models to heterogeneous multimodal systems. Beyond comparing methods, MMIB’s diagnostics expose deployment-relevant behavior, such as models that represent visual information yet let language priors override it under conflict, providing an actionable signal for when a VLM will fail to ground in its visual input.