TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

As AI agents are selected, trained, and monitored by explicit evaluators, verifier scores become measurement instruments rather than passive la- bels. This creates a validity problem: a fixed verifier may reward brit- tle tool use, decorative process, or proxy shortcuts once agents adapt to evaluator-visible signals. We introduce TRACE, a stress-testing proto- col that pairs original agent evaluations with controlled task, tool, rubric, proxy, and observation mutations, then reports trace-grounded robust- ness, transfer, proxy-divergence, hack-risk, and process-overfitting diag- nostics. In a deterministic synthetic ResearchOps environment with 25 tasks and 1,575 scripted-policy rollouts, TRACE recovers the intended separation between robust and shortcut behavior: the careful policy pre- serves mutated success and transfer (MSR/MTR 0.860), while the short- cut policy exposes proxy-true divergence on proxy-relevant tasks (PTD 0.932). Controlled verifier-optimization loops show the same failure mode under selection pressure: a naive visible objective selects proxy exploit, whereas robust TRACE feedback selects careful, and ablations show that removing proxy-divergence or tool-interface probes hides distinct failures. Model-driven evidence is secondary but consistent with the diagnostic framing: an induced proxy-exploit LLM prompt yields PTD 3.215, hack- risk 0.708, and zero MSR/MTR, and a partial EvilGenie hard-task snapshot is ingested as external alignment evidence, not a full benchmark reproduc- tion. TRACE’s claim is therefore methodological: before verifier scores are used as training, evaluation, or deployment signals, their robustness should be measured under controlled non-stationarity and optimization pressure.