Beyond the Weighted Sum: Rubric Aggregation for Multi-Criterion LLM Alignment

Post-training large language models (LLMs) with reinforcement learning from verifiable rewards (RLVR) has shown promising results in complex reasoning tasks. A typical post-training method is to combine scores of different criteria to form a linearly scalarized reward. Such a formulation has shown disadvantages in rubric-based tasks, where the trained LLMs under the scalar reward fail to align well with several criteria simultaneously, such as coverage, brevity, and safety. In other words, the trained LLMs often align with only a limited number or even one criterion rather than all. The key reason behind this failure is that a policy may achieve balanced average criterion scores by mixing single-criterion specialist responses, without producing responses that jointly satisfy all criteria. We address this with response-level aggregation, which applies a non-linear aggregation function to each response's criterion vector before averaging across samples. We instantiate this idea as Nash-GRPO and Egalitarian-GRPO at the reward level, and as Egalitarian-GRPO (adv.) at the advantage level. Across a controlled criterion bandit and GRPO-trained Qwen3 models, response-level aggregation prevents criterion collapse and improves worst-criterion performance on summarization, long-form QA, a four-criterion verifiable rubric, and a judge-scored medical rubric.