When Metrics Fail: Detecting Semantic Blind Spots via Controlled Perturbations for Code Comment Evaluation

Automatic metrics and LLM judges are widely used for code comment evaluation, yet their ability to distinguish semantically meaningful changes remains poorly understood. In code comment evaluation, small deviations like polarity flips can preserve high lexical similarity while completely altering meaning. We introduce SCORE, a controlled stress-testing framework for diagnosing evaluation reliability through semantic perturbations that injects ten fault types into 2,750 instances from 250 Java methods, enabling fine-grained diagnosis of evaluation failures across semantic fault types. Applying SCORE reveals systematic blind spots: lexical metrics (BLEU, ROUGE, BERTScore) assign high scores to polarity flips (BLEU 0.68) and boundary errors (BERTScore 0.96) despite semantic incorrectness: 201 instances (7.3%) lie in the ``when metrics fail'' quadrant where BLEU > 0.7 but LLM judges score < 0.3. Under controlled semantic perturbations, protocol-diverse LLM judges provide better discrimination in this controlled setting, but protocol choice matters: evidence-constrained (J2) is more lenient, adversarial (J3) more conservative, and their disagreement identifies cases needing audit. A selective audit of the 10\% most problematic instances with GPT-4o revisits systematic overestimations (quantifier flips: 1.0 → 0.0)) at only 20% additional cost. By enabling controlled diagnosis of semantic evaluation failures, SCORE provides a reproducible diagnostic methodology for stress-testing evaluation methods and analyzing where automatic evaluation succeeds or fails under semantically meaningful perturbations.