CARL: Cartridge Adaptation through Reinforcement Learning

Reinforcement learning with verifiable rewards (RLVR) commonly adapts 2 language models by updating model weights. This makes each specializa- 3 tion a new weight-space artifact and causes gradient and optimizer state to 4 scale with model size. We ask whether useful reward-driven adaptation 5 can instead be stored in a compact key–value (KV) prefix attached to an 6 otherwise frozen model. CARL (Cartridge Adaptation through Reinforce- 7 ment Learning) treats the cartridge itself as the policy parameters: rollouts 8 are generated by a frozen base model conditioned on trainable per-layer 9 KV states, task reward is computed from the output, and policy-gradient 10 updates flow only to the cartridge. On Countdown with Qwen3-1.7B, a 11 128-slot cartridge containing 7.34M parameters raises held-out pass@1 from 12 11.0% to 46.5%. Under the same data split, prompts, rollout budget, and 13 evaluation protocol, full-model GRPO reaches 53.5%. Thus the cartridge 14 captures 83.5% of the observed full-model improvement while making only 15 0.43% as many parameters mutable and storing the learned adaptation 16 in a 29.4 MB file rather than a 6.88 GB checkpoint. These single-seed re- 17 sults establish that online RL can write substantial task-adaptive behavior 18 into persistent KV state; they do not establish superiority to weight-space 19 adapters, and leave open matched LoRA, untrained-prefix, capacity, and 20 multi-task controls.