Psychometric Validation of LLM Benchmarks as Measurement Instruments Through Item Response Theory

We reconceptualize benchmarks as principled measurement instruments and introduce a psychometric-driven approach to benchmark optimization and validation during development. Our framework is geared to systematically identify and remove malfunctioning items, thereby enhancing the precision of ability estimation, benchmark reliability, and operational efficiency after deployment. To validate our framework, we analyzed data from 79 LLMs across 57 MMLU benchmarks through the item response theory model. Our results confirm that the framework reduces benchmark size by over 60% while maintaining high reliability, and ensures consistent measurement precision by stabilizing test information functions. After all, our proposed optimization framework simultaneously advances computational efficiency, evaluative rigor, and measurement reliability, providing a principled foundation for the development of psychometrically robust LLM benchmarks.