The Castle of Crossed Models: Benchmarking LLMs on Calvinian Narrative Crossings
Enrico Fonda
Abstract
In Italo Calvino's The Castle of Crossed Destinies, mute travellers tell their stories by laying tarot cards: the same card means one thing in one tale and another in the next. We turn this device into a benchmark, and the benchmark into an audit of its own instrument. A model writes one short passage in which two stories cross through a single object that means something genuinely different in each, joined by a grounded physical, causal, or temporal link; quality is scored by a blind cross-lab panel of LLM judges, so the same data that ranks twelve models also measures where the instrument ranking them breaks down. We deliver three things: trustworthy tier discrimination, a principled boundary past which the scores measure the judges rather than the models, and a suggestive but unadjudicated frontier ordering that further measurement could resolve. The instrument resolves capability tiers cleanly and stably (rank-stability $\rho = 0.97$--$0.99$) and bounds the leading models as near-equivalent: any real within-cluster gap is smaller than the $\sim$0.2-unit separation the panel demonstrably resolves, and a lineage-balanced panel (one judge per lab) reproduces the ranking, the tie, and the bound. A flagship that arrived after the roster locked lands at nominal first, but only one rival-lineage judge ranks it ahead, and the neutral judge does not. LLM-judge bias has a measurable \emph{lineage geometry}: self-preference, plus a same-lab agreement seam (ICC 0.92 vs 0.83 cross-lab) that inflates pooled reliability, so that grading the frontier with one lab's judges would have crowned that lab. A validity paradox follows. The structural dimensions saturate across strong models, so frontier differentiation could only live on the taste axis (originality); yet there the panel finds no model difference (frontier $\eta^2 \approx 0$), and the agreement it does show rides a shared training prior. Five instrument knobs all fail to resolve the leading cluster, and not because the instrument is too coarse: the judges share a prior, and no amount of sharpening escapes it.
Chat is not available.
Successful Page Load