Why LLMs Choose Bad Plans: Miscalibrated Capability Beliefs and Their Cost for Planning

Large language models are increasingly deployed as autonomous agents that construct multi-step workflow plans. Despite this, they often choose poor plans even when better ones are available. We ask whether one underexamined cause is that agents lack calibrated estimates of their task-success probabilities. Using a grid navigation task as a controlled capability dimension with per-model difficulty calibration, we embed the task in two planning environments, a two-stage decision tree and a sequential MDP, and a two-phase experiment that separates how a task success probability estimate is formed from how it is used. Across frontier LLMs, we find that (1) all models are systematically overconfident about hard tasks, (2) this overconfidence drives suboptimal plan selection of an expected-value trap that is optimal only under inflated beliefs, (3) models update their beliefs based on experienced outcomes (decreasing their confidence as they experience failures), but only one model updates to the optimal plan, while the rest over-correct into excess caution. These findings demonstrate that miscalibration in LLM estimates of their own abilities is a critical hurdle in developing increasingly autonomous AI agents that can safely and reliably handle long sequences of tasks.