The Lifecycle of LLM-as-a-Judge: Learning, Aligning, and Evolving for Large-Scale Recommendation Explanations

LLM-as-a-Judge has become a standard paradigm for scaling evaluation of natural language generation. Most work, however, treats a judge as a static artifact, validated once at construction and then frozen. We argue that an LLM judge deployed in a production system is better understood as a lifelong agent: a persistent component that must keep learning from human feedback, stay aligned with human judgment for the right reasons, and evolve as the world it evaluates drifts over time. We study such an agent in production at a major video streaming platform, where it evaluates user-facing recommendation explanations served to millions of users, and organize its lifecycle into four phases. (I) Birth builds validated benchmarks of human labels and rationales. (II) Training aligns the judge with reasoning-aligned rubric tuning, using a meta-judge that checks whether the judge agrees with human rationales for the right reasons, not just on the label. (III) Deployment runs the judge as a quality gate and as the critic in a self-reflective revision loop, where the generator is optimized against its signal. (IV) Monitoring continuously audits judge–human agreement, detects alignment drift, and triggers re-tuning as the recommendation catalog and users evolve. We also report an ablation isolating the reasoning-alignment signal and a large-scale online A/B test, and distill practical lessons for operating LLM judges as long-lived, continually adapting agents.