Evaluating the Impact of Personalization in Conversational Cybersecurity Assistants

Large language models are increasingly used as cybersecurity assistants, making personalization a promising mechanism for improving the effectiveness of generated advice. We investigate four personalization strategies, ranging from static user profiles to interaction-history-based personalization, using a corpus of 1,045 real-world cybersecurity questions and a 7-day deployment involving 57 participants and 1,066 user questions. Across both a large-scale simulation study and human evaluation, conversation-based personalization consistently improves perceived helpfulness and likelihood of following security advice. Importantly, the relative trends observed in simulation align with those obtained from human evaluation, suggesting that simulation-based evaluation can provide a scalable mechanism for comparing personalization strategies before costly user studies. These results indicate that behavior-driven personalization is a promising direction for LLM-powered cybersecurity assistants and highlight the value of combining simulation and human evaluation when studying personalized language-model systems.