Language models process and generate text sequentially in token units, and the tokenizer determines how much text each inference step covers. Under standard tokenization, a short English phrase such as “On the table.” is usually produced as four separate predictions for the preposition (on), article (the), noun (table), and period (.), affecting the overall context length, and accordingly—inference costs. We introduce CoBPE, a compositional tokenization approach that represents such phrases as a single lexical base token (table) attached with a small set of reusable surface modifiers, composed in embedding space at input and predicted jointly at output. In controlled from-scratch pretraining at 780M and 1.3B scales, CoBPE shortens sequences by 30% and improves downstream performance by 1.2 points relative to standard BPE under matched training compute. More broadly, our results suggest that part of what is now expressed through token sequences can instead be modeled through structured representations, opening a broad design space for more token-efficient and capable language models.

Frontier LLM companies have repeatedly assured courts and regulators that their models do not store copies of training data. They further rely on safety alignment strategies via RLHF, system prompts, and output filters to block verbatim regurgitation of copyrighted works, and have cited the efficacy of these measures in their legal defenses against copyright infringement claims. We show that finetuning bypasses these protections: by training models to expand plot summaries into full text, a task naturally suited for commercial writing assistants, we cause GPT-4o, Gemini-2.5-Pro, and DeepSeek-V3.1 to reproduce up to 85-90\% of held-out copyrighted books, with single verbatim spans exceeding 460 words, using only semantic descriptions as prompts and no actual book text. This extraction generalizes across authors: finetuning exclusively on Haruki Murakami's novels unlocks verbatim recall of copyrighted books from over 30 unrelated authors. The effect is not specific to any training author or corpus: random author pairs and public-domain finetuning data produce comparable extraction, while finetuning on synthetic text yields near-zero extraction, indicating that finetuning on individual authors' works reactivates latent memorization from pretraining. Three models from different providers memorize the same books in the same regions ($r \ge 0.90$), pointing to an industry-wide vulnerability. Our findings offer compelling evidence that model weights store copies of copyrighted works and that the security failures that manifest after finetuning on individual authors' works undermine a key premise of recent fair use rulings, where courts have conditioned favorable outcomes on the adequacy of measures preventing reproduction of protected expression.

While inference-time scaling has improved the reasoning abilities of large language models (LLMs), the need to generate long chains-of-thought (CoTs) is a computational bottleneck. Thus, in contrast to sequential scaling methods like CoT, recent parallel scaling techniques instead use fork and join (FJ) primitives to divide work across multiple LLM threads. Although conceptually promising, this necessitates a centralized controller, limiting scalability. We introduce Message Passing Language Models (MPLMs), a framework for LLM reasoning in which threads communicate directly via lightweight send and receive primitives. MPLMs enable efficient scaling through two key mechanisms: (1) reduced communication costs, achieved by avoiding redundant context sharing, and (2) preemption, which allows threads to terminate early based on partial information from their peers. In Sudoku puzzles, we show that MPLMs require an asymptotically smaller context than both serial CoT and parallel FJ . Most importantly, we fine-tune a single model to solve 25 $\times$ 25 puzzles that remain challenging for standard CoT and FJ approaches, as well as frontier reasoning models without tools. We also demonstrate that on 3-SAT puzzles, where search dominates, the capability of preemption allows termination of unpromising branches, which results in improved efficiency. Together, these results demonstrate that MPLMs could provide a principled and scalable alternative to existing sequential and centrally coordinated parallel reasoning paradigms.

Recent approaches to discourse coherence assessment increasingly rely on language models, but it remains unclear whether they evaluate underlying discourse structure or merely surface-level lexical patterns. To investigate, we introduce metaphor as a controlled linguistic probe. Using a rigorous paraphrase framework, we generate meaning-preserving metaphorical rewrites for a corpus of human-annotated texts. Across multiple architectures, we uncover a pervasive tendency: models paradoxically reward figurative language, consistently assigning higher coherence scores to texts with novel metaphors, provided the metaphors remain semantically compatible with their context. Furthermore, layer-wise analyses reveal that while early model layers are highly sensitive to surface-level figurative features, higher layers successfully recover the underlying semantics. Ultimately, our findings demonstrate that coherence judgments are influenced by local stylistic variation, indicating that current models often evaluate linguistic realization rather than relying on discourse-level reasoning alone.