Code Migration Benchmark: Evaluating AI Models' Autonomous Code Migration Performance

Code migration, porting a working program from one language to another, is a high-value but slow and risky engineering task, spanning the modernization of legacy COBOL systems and translation between modern languages. We introduce \textbf{Code Migration}, a benchmark that evaluates whether AI models can autonomously reproduce an existing program's behavior in a new language. It comprises two splits: a modern-to-modern \textbf{CLI} split of $30$ open-source command-line applications, each migrated into four target languages among Python, Java, Kotlin, Rust, and C++, and a legacy-to-modern \textbf{COBOL} split of $10$ COBOL-to-Java migrations. Crucially, migrations are judged by \emph{behavior} rather than code similarity: a submission scores by the fraction of a hidden behavior-test suite that the rebuilt program passes when driven through its interface, and a separate anti-cheat checker zeroes submissions that wrap the original, copy reference artifacts, hard-code outputs, or use the wrong language. Across $23$ frontier and open-weight models run in a long-horizon agentic sandbox, we find autonomous migration far from solved: the strongest model reaches $55.1\%$ overall accuracy on hidden test cases, the two splits reward distinct skills, accuracy and cost diverge by orders of magnitude.