CoBit: Language Modeling with Bitstream Diffusion

Diffusion language models (DLMs) promise parallel, order-agnostic generation, but on standard benchmarks they have historically lagged behind autoregressive models in sample quality and diversity. Recent continuous flow and diffusion approaches over token embeddings have narrowed this gap, suggesting continuous state spaces are highly effective for language. In this work, we further close the autoregressive gap by modeling text as a continuous diffusion process over fixed-width binary bitstreams. We refer to the resulting model as CoBit (Continuous Bitstream Diffusion). Our approach represents semantic tokens as analog bit sequences and uses a matched-filter residual parameterization to isolate contextual learning from analytic independent-bit posteriors. Crucially, we adopt a stochastic sampler that applies Langevin-type corrections gated by the entropy-rate profile, automatically concentrating stochasticity in high-information regions while remaining nearly deterministic elsewhere. On the One Billion Word Benchmark (LM1B), our 130M-parameter bitstream model reaches a generative perplexity (GenPPL) of 59.76 at matched real-data entropy (4.31) using 256 neural function evaluations (NFEs), decisively outperforming prior DLM baselines and reaching the autoregressive reference. On OpenWebText (OWT), our stochastic sampler establishes a new continuous-DLM Pareto frontier, achieving GenPPL = 27.06 at an entropy of 5.26 using $4\times$ fewer steps than previous 1024-NFE baselines. Scaling the same recipe to a 462M-parameter model (CoBit-M) substantially improves the OWT GenPPL–entropy Pareto frontier over the 130M model (CoBit-S) and over medium-scale continuous and discrete DLM baselines, reaching GenPPL = 19.5 at an entropy of 5.40, near the real-data entropy (5.44), and approaching pretrained GPT-2 Medium nucleus sampling over the high-quality region of the trade-off. Beyond likelihood-style evaluation, we test CoBit on two verifiable reasoning tasks under the protocol of Deschenaux & Gulcehre (2026) — Sudoku and executable-code math (TinyGSM→GSM8K) — using no guidance and no temperature scaling. CoBit-S is the strongest continuous diffusion/flow model on both: on Sudoku it surpasses every published baseline, including the discrete model Duo, across all difficulties (98.5/91.7/65.9 vs. 96.3/84.7/58.4 exact match); on GSM8K it reaches 25.4% (matched budget) / 29.2% (longer training), surpassing the best continuous model S-FLM (18.0%) and beating the discrete-diffusion models MDLM and Duo at standard temperature (18.0/17.2%). Under their optimized low-temperature decoding these discrete models remain ahead — a sharpening mechanism we flag as a natural next step. Finally, by predicting $\mathcal{O}(\log V)$ bitwise logits via semantic bit-patching rather than an $\mathcal{O}(V)$ vocabulary head, CoBit reduces memory and increases throughput — a scalable paradigm as vocabulary sizes grow.