When Computer-Use Agents Act on the Wrong Instruction: Characterizing and Intervening on Prompt-Injected Behavior at the Action Slot

Computer-use agents act on screens, documents, and emails they do not control, and a recurring unsafe behavior is that they follow an instruction they merely read rather than the one their user gave, leaking a credential, emailing an attacker, or running a dangerous command. We study this failure at the level where it manifests: the individual action slot, not the whole page or action. The same untrusted content can be legitimate in one field but a hijack in another; an invoice may set a date, but never a payment recipient. We present ASP-Guard (Action-Slot Predicate Guard), a training-free, model-agnostic runtime monitor that measures how much each screen region influenced each slot and authorizes that influence by the slot’s semantic type. Thus, untrusted content may shape a date but not a recipient or command, while unrecognized slots escalate to a human rather than silently executing. Across all four visual VPI-Bench surfaces (126 attack cases, 41 landed on gpt-4o-mini in the pinned reference run) and tool-calling AgentDojo, ASP-Guard blocks or escalates every landed attack with zero silent failures, where a catch is a hard BLOCK or human CONFIRM. On the browser set, this gives hard-block DSR 0.77 and caught-or-escalated rate 0.97; the only baseline that catches more blocks all benign traffic. Its benign hard-block false-positive rate is 0.08 on the same model, far below untyped global-suppression baselines at 0.56; counting escalations as false positives, the strict rate is 0.23. The gain comes from typing: an untyped rule that ties ASP-Guard on one attack family (DSR 1.00) collapses to 0.33 as the attack surface diversifies, while the typed predicate does not. Against a CausalArmor-style attribution baseline, per-slot typing also removes silent allows left by the untyped rule. Open-weights models confirm the pattern: the typed-vs-untyped gap reproduces on AgentDojo, most clearly on Llama-3.3-70B, and on VPI-Bench’s 26 landed open-VLM attacks, ASP-Guard silently allows 1 versus the untyped rule’s 5.