API Benchmarks Do Not Reliably Predict Chatbot Behavior

Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems.We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, sycophancy, and hallu- cination. We find systematic API–interface differences in both accuracy and consistency. API models outperform their web-interface counterparts by an average of 3.5% across benchmarks, and interface models exhibit lower test–retest outcome agreement by 2.4%. For ChatGPT, this interface-API gap exceeds the difference between GPT 5.3 and GPT 5.4 in the API. In other words, switching from API to interface degrades performance more than an entire model generation. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a gap between API measurements and interface behaviors that complicates the use of API evaluations as proxies for deployed systems.