When Is a Probe Monitor Safe to Act On? A Transferability Stress Test for Eval-Awareness Probes under RLVR

A growing safety practice is to train a small probe on a language model's internal activity and use it as a fixed monitor — for example, to flag when the model can tell it is being tested rather than used for real. The probe is checked once and then trusted. But deployed models do not stay still: they keep being trained, and a probe that passed its check can quietly stop working while no one notices. We make this "does it still work later?" question testable for probes that detect whether a model knows it is being evaluated, using Llama 3.1 8B Instruct, where each prompt varies in format — benchmark-style or casual — and context — evaluation or deployment. Two results stand out. First, the usual fix for a probe that goes by wording rather than meaning — balancing the two formats evenly across the labels, called paired-decorrelated training — only half-works: the probe tells evaluation from deployment cleanly on casual-worded prompts but only partly on benchmark-worded ones, so it can pass a one-number check while still partly relying on how the prompt is worded. Second, under reinforcement-learning (RLVR) post-training — training that improves the model on a separate task, with the probe never part of that training — the probe's overall score barely changes, but that average hides the real problem: for borderline prompts, the ones the probe could barely tell apart as evaluation or deployment, a small change inside the model is enough to flip its answer, and these flips land mostly on the benchmark-style prompts the fix only partly corrected. The takeaway is preventive: re-training the stale probe recovers almost nothing, so a monitor like this can't be patched after the model changes. Validate each format on its own, use the monitor only where it comes out clean, and re-check after further training. Reliability has to be designed in up front, not retrained in later.