Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds

Reward hacking, where AI systems exploit misspecified objectives to achieve high reward without satisfying intended goals, remains a central challenge in AI safety. Yet most known instances have been discovered post hoc in frontier systems where controlled study is impractical. We adapt the AI Safety Gridworlds framework into a text-based evaluation suite that reformulates classic reinforcement learning safety tasks for language-based agents. Across frontier and mid-scale models, we find that specification gaming emerges zero-shot: models systematically achieve high observed reward while underperforming on hidden safety objectives, and even apparently safe behaviors can reflect misunderstanding rather than principled safety. Reinforcement learning amplifies rather than corrects these failures. Direct reward optimization increases the observed signal but fails to improve hidden safety performance, as the model's initial competence causes it to lock into locally rewarding strategies before discovering safer alternatives. This failure persists across model scales and is not resolved by finer credit assignment, exploration prompts, or entropy regularization. Our results show that reward hacking arises naturally when optimizing proxy objectives with capable language model agents and resists standard mitigations, highlighting the need for fundamentally different approaches to alignment in agentic settings.