Safety Benchmarks as Measurement Instruments: A Crossed Evaluation of LLM Harm Benchmarks

Safety benchmarks increasingly shape how language models are compared, released, audited, and governed. These scores are often treated as model properties, even when the benchmarks producing them rely on different prompt distributions, judge models and rubrics. We test how much these operational choices matter by studying 5 popular safety benchmarks that all draw from major provider usage policies. Using a fully crossed 5 × 5 design across 13 open-source models, we recombine each benchmark’s prompt set with each scorer to separate prompt-side and scorer-side effects. We find that disagreement is structured by measurement protocol: scorer-side protocols introduce more variation than prompt sets, benchmark choice substantially shifts rankings for middle-ranked and domain fine-tuned models, and cross-benchmark validity varies sharply by harm category. Concrete harmful-use categories show stronger cross-benchmark validity, while contextual domains like privacy or political campaigning do not support straightforward comparison across tools. These findings illustrate that safety benchmark scores are protocol-dependent measurements rather than intrinsic model properties, motivating reporting, calibration, and governance practices that tie safety claims more tightly to the operational choices that produce them.