Legible Agents: A Design Space for Overseeing Computer Use Agents

Computer use agents (CUAs) can act across everyday graphical interfaces. As they complete multi-step workflows on users’ behalf, users need to oversee agents to catch failures, calibrate trust, and intervene when the agent needs guidance. To explore how current CUAs make their actions and internal reasonings legible to users, we conducted a formative analysis of 18 CUA interfaces. Most current CUAs combine chat-heavy updates with abrupt on-screen actions, which are difficult for users to follow. To identify broader opportunities for legible CUA design, we conducted a co-design workshop with 8 stakeholders, drawing inspiration from adjacent research where awareness of others’ actions is essential, such as multi-user collaboration and human-robot interaction. We synthesize these insights into 1) a design space for CUA legibility mechanisms, and instantiate underexplored opportunities through 2) three technical probes across different oversight contexts: in-situ, parallel, and replay. Together, this work offers a structured lens for designing CUAs that are easier to oversee.*We will demo our technical probes upon acceptance.