Public Benefits Bench: Can AI Help People Navigate SNAP Benefits?

For evaluating whether general-purpose LLMs can help with obtaining and keeping public benefits, standard benchmarks evaluating model outputs against one ground-truth answer are unsuitable. SNAP policy, in particular, breaks the usual recipe on three counts: the model must elicit, the governing policy is subject to change, and models can reach the same answer by drawing on different sources which vary in reliability. We build a benchmark of 459 expert-validated SNAP scenarios with 2,931 pass/fail criteria and evaluate twelve frontier models on this use case. This benchmark queries each model through a simulated user who discloses context only on request, and the model has access to web search with multi-turn dialogue capabilities. No model reaches 70\% (Claude Opus 4.8 leads at 67.7\%), and what distinguishes models is tool use rather than stored knowledge. Access to multi-turn capabilities only raises pass rates about $4.4\times$ as much as web search only, and neither query volume nor response length track accuracy. Evaluating such systems means studying the model's behavior, not only the answer it lands on.