TrustTrajectory: Measuring When LLM Agents Begin to Hallucinate in Long-Trajectory Agentic Execution
Abstract
LLM agents deployed in reservation workflows have to hold state coherently across long multi-turn conversations, absorb changes to constraints mid-stream, and recover cleanly when parameters shift underneath them. Prior benchmarks such as τ-Bench and τ²-Bench score only final task completion, which leaves an important question unanswered: at what point does an agent's internal state actually stop being reliable, and what causes it? We introduce TrustTrajectory, a 46-scenario benchmark spanning restaurant and flight booking across five difficulty tiers (easy, medium, hard, conflicting, adversarial), producing trajectories of up to 30 turns through information dripping, mid-task complications, and a hard constraint pivot. Every turn is scored against a five-class hallucination taxonomy (Param Drift, Premature Commit, Constraint Ignore, Confident Confab, Memory Collapse), which gives us fine-grained trust tracking instead of a single pass/fail label. We also introduce two metrics: First Hallucination Turn (FHT), which pinpoints the exact onset of unreliability, and the Trust Decay Curve, which traces hallucination dynamics across trajectory depth. Evaluating five frontier models, we find divergent pre-pivot behavior but a consistent post-pivot collapse across all of them. Llama-4-Maverick stays low pre-pivot (0.021 [0.014–0.029]) and then spikes sharply after the pivot (0.240 [0.201–0.278]); Gemini 3.5 Flash, by contrast, is already showing elevated pre-pivot vulnerability (0.098 [0.077–0.120]) before its post-pivot rate climbs to near-total collapse (0.842 [0.804–0.879]). Constraint Ignore is the dominant post-pivot failure mode across every model we tested. A five-tier ablation further shows that it is the type of constraint, not the raw count, that governs when hallucination sets in: over-specified hard scenarios actually delay hallucination onset relative to medium ones, an effect we attribute to a contextually anchoring guided-hallucination effect.