The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations

LLM-as-a-Judge has become a standard way to scale evaluation of natural language generation. However, when such judges are deployed in real production systems, they face challenges that go beyond static benchmarks. In practice, an LLM judge is an imperfect measurement instrument whose scores must be validated, monitored, and periodically recalibrated. We study an LLM judge as a measurement instrument under these conditions, using a production system that evaluates user-facing recommendation explanations at a major video streaming platform serving millions of users. Rather than validating the judge once and treating it as a fixed artifact, we treat it as an instrument with a lifecycle and organize that lifecycle into four phases, each tied to a problem in AI measurement science. (I) Birth builds validated benchmarks of human labels and rationales. (II) Training calibrates the judge with reasoning-aligned rubric tuning, using a meta-judge that checks whether the judge agrees with human rationales for the right reasons, not just on the label. (III) Deployment runs the judge as a quality gate and as the critic in a self-reflective revision loop, where the generator is optimized against its signal. (IV) Monitoring detects drift between the judge and humans and triggers recalibration as the catalog and users evolve. We report an ablation isolating the reasoning-alignment signal and a large-scale online A/B test, and we distill practical lessons for operating production LLM judges.