Measuring the Measurers: A Generalizability Theory Approach to LLM-as-a-Judge Reliability

LLM-as-a-Judge evaluation underpins most current AI leaderboards, but its reliability is typically reported only as average judge--human agreement. The closest prior treatment, CyclicJudge, decomposes benchmark variance into scenario, generation, judge, and residual components but treats each judge's bias as a single constant shared across all evaluated models - ruling out, by construction, the self-preference and architectural-family biases documented elsewhere. Treating LLM-judge score variance as a single measurement object, we extend this line of work by (i) treating judges as a random facet and explicitly estimating the model $\times$ judge interaction $\sigma^2_{mj}$, (ii) translating reliability into the operationally meaningful metric of pairwise rank-flip probability (RFP), and (iii) testing whether the variance-component structure replicates across three distinct multi-judge benchmark releases. Analysis of $\sim$113,000 LLM-judge scores from WildBench v2, Arena-Hard v0.1, and BiGGen-Bench, plus 2,704 paired human ratings, partially falsifies one pre-registered hypothesis, confirms another, and reveals that the third holds conditional on score format: same-format benchmark pairs replicate the variance-component ordering perfectly, but Likert-vs-pairwise pairs do not. The headline operational finding is that no design within the studied $(n_j, n_i)$ grid holds the maximum adjacent-pair rank-flip probability below 5\% on any of the three benchmarks: under current evaluation budgets, adjacent leaderboard ranks are not reliably distinguishable.