Learning to Learn from Language Feedback with Social Meta-Learning

While large language models (LLMs) excel at processing static knowledge, they often struggle to adapt their behaviour based on language feedback. Current training paradigms rely on static corpora and fully defined tasks, overlooking the interactive feedback loops essential for learning to adapt. In this work, we regard learning from language feedback as a capability to be meta-learned. We introduce Social Meta-Learning (SML), an algorithm that transforms single-trial verifiable tasks into multi-trial teacher-student interactions driven by information asymmetry. During meta-training, a teacher model is given privileged information, such as a ground truth solution, and prompted to provide language feedback to the student after an incorrect solution attempt. This partial observability from the perspective of the student incentivises adaptation between trials, allowing us to train models to iteratively refine their solutions by integrating external guidance. Our experiments demonstrate that online reinforcement learning performs better as the outer loop learning algorithm than offline supervised finetuning. Once learned, the ability to learn from language feedback transfers across domains. Furthermore, we identify asking questions as a cognitive behaviour that improves transfer to tasks that are initially underspecified at test time. By repurposing static tasks as interactive social learning environments, SML provides a scalable path towards LLMs capable of learning and improving on their raw parametric capabilities during deployment.