InfoSynth: Information-Guided Benchmark Synthesis for LLMs

Large language models (LLMs) have advanced significantly in reasoning and code generation, but efficiently creating new benchmarks to evaluate these capabilities remains challenging. Traditional benchmark creation relies on expensive, time-consuming human effort, and existing benchmarks often contaminate LLM training data, necessitating novel, diverse benchmarks to assess genuine capabilities. We introduce InfoSynth, a framework for automatically generating and evaluating reasoning benchmarks guided by information-theoretic principles. We propose KL-divergence- and entropy-based metrics to quantify benchmark novelty and diversity without costly model evaluations, and build an end-to-end pipeline that synthesizes robust Python coding problems from seed datasets via genetic algorithms and iterative code feedback. Our method generates accurate test cases and solutions 97\% of the time, and the synthesized benchmarks consistently exhibit higher difficulty than prior works while allowing control over the novelty/diversity and difficulty of generated problems. InfoSynth offers a scalable, self-verifying pipeline for constructing high-quality, challenging coding benchmarks for LLMs.