Task Completion Is Not User Success: AMI-Bench for Personalized Lifelong Agents

Personal agents increasingly act for the same user across many interactions. Existing agent benchmarks measure whether an agent completes a task, but a task can still be completed wrongly for the specific user: for example, a dinner order is placed successfully but violates the user's dietary restriction. We argue that personal agents should instead be evaluated on personalized task completion: completing a task the way a specific user would want. We introduce AMI-Bench to measure it. In AMI-Bench, the agent serves an LLM-simulated user with a hidden persona. Tasks are underspecified requests in simulated consumer apps, the agent may ask the user clarifying questions at any time, and grading is done on the final application state. Because the same user persists across episodes, a good agent should learn preferences instead of re-asking, so AMI-Bench tracks the number of clarifying questions. Each episode is scored on two axes: completion (did the workflow reach a valid end state) and personalization (does that state match the user's hidden preferences). The two diverge sharply. Even though the agent can always ask, completion exceeds personalization by 26 to 57 points for every model and method we evaluate: GPT-5.5 with experience retrieval reaches 98.9% completion but only 44.9% personalization, and no configuration exceeds 45%. Memory reduces re-asking (a persistent profile cuts Claude Sonnet's clarification requests from 0.76 to 0.29 per episode) but does not reliably close the gap. The results show that while frontier LLMs can complete many tasks based on completion metrics, they can still fail on other metrics that correspond to personal agent satisfaction.