Moral Hazard in Multi-Agent Language Models

Cooperation can fail when socially valuable effort is costly, weakly observable, and mainly benefits others. Drawing on Holmstr\"om's team moral-hazard model, we introduce the Dialogue Moral Hazard Game, a textual hidden-action game for testing whether language agents provide costly, other-benefiting effort. In each episode, an agent can preserve an immediate local reward or pay a query cost to reveal a hidden safety fact that primarily helps another agent choose a team-beneficial final action. We evaluate seven open-weight language models across base and intervention conditions. Base models show two main failure modes: many preserve local reward without achieving team success, while others query often but fail to communicate useful warnings or change downstream final choices. In other words, models rarely complete the intended sequence of querying hidden facts, posting useful warning notes, and using those notes to make team-beneficial final decisions. We then evaluate supervised fine-tuning, RL fine-tuning with RLOO, sequential SFT+RL, and GEPA prompt optimization. Interventions produce limited, model-specific gains: OLMo 7B improves most under LoRA training, while Qwen3 4B benefits most from prompt optimization, though not always through the intended query mechanism. Overall, costly other-benefiting effort is not a default behavior of current open-weight language agents, but can be partially induced.