Evaluating Diffusion Language Models Beyond Benchmarks: User Perception of Generation Dynamics

Non-autoregressive models, such as diffusion-based language models (DLLMs), now match autoregressive large language models (ALLMs) on standard benchmarks and are transitioning into production systems, where they offer advantages through bidirectional reasoning and fast inference. However, much of the recent work on these language models has focused on likelihood, benchmark accuracy, and efficiency, while little research has explored how their unique text generation process influences user perception. In this work, we examine how the generation dynamics of LLMs affect perceived model trustworthiness and quality by conducting a randomized within-subjects study ($N = 39$) where users engage with a DLLM (LLaDA-8B) and a conventional ALLM (LLaMA 3.1-8B) of comparable benchmark performance in an interactive chatbot environment. We find that the DLLM chatbot elicits lower trust and quality ratings, while the ALLM elicits more anthropomorphic descriptions and is rated as more trustworthy and higher in quality. These findings suggest that comparable benchmark performance does not guarantee comparable user trust and perceived quality, and that DLLM's unique generation dynamic may slow its adoption. We provide the code on the project repository: anonymous.4open.science/r/LLaDA-vs-LLaMA-Code-8C87.