ACID-Bench: Measurement Contracts for State-Changing Tool Agents

Evaluating state-changing tool agents is a measurement problem, not only a task-completion problem. When an agent calls external APIs, mutates persistent records, and observes misleading tool outcomes, the observations are interactive, non-i.i.d., and path-dependent: a later state can depend on whether a fault was actually exposed and how the agent recovered. This paper presents ACID-Bench as a measurement-contract framework for this setting. Each contract binds a user goal, mutable-state preconditions, a deterministic transactional fault, the agent-visible tool outcome, final-state predicates, prompt-condition metadata, and validity rules. We instantiate the framework on 52 checked-in retail and airline contracts covering partial commits, ghost-success retries, stale reads, crash restarts, and consequence obligations. The evaluator uses deterministic trace/state predicates rather than an LLM judge. The evidence is used to test construct validity, not to rank providers: a task-only control completes 52/52 user goals while violating integrity in 52/52 final states; on 780 manifest-reviewed model-run rows, Tau-style task success would credit 431 rows, but 241 of those fail the ACID contract; and the validity flow is 780 attempted to 688 valid to 554 fault-exposed to 234 strict-clean. These results show why task success, fault exposure, denominator movement, safe handoff, and state integrity must be reported separately for state-changing agents. The paper does not claim deployment certification, universal domain coverage, or a provider leaderboard.