When Does a Probe Still Measure Its Construct? Construct Validity of Eval-Awareness Probes under Non-Stationarity (RLVR)

Interpretability increasingly treats a probe direction as a measurement instrument: once validated, it is read off a model's activations to measure a hidden property—here, whether a model is treating an input as a test or as ordinary use. Standard validation checks only predictive validity (does the score separate the two classes on a fixed split?), not construct validity (does it measure the intended property rather than a stand-in, and keep doing so when the data shifts?). We make the second testable on two fronts: a format confound at validation time, and drift in the model's internal representations as it trains further. We study Llama 3.1 8B Instruct. Each prompt varies in two ways: its format—how it is worded, either like a benchmark question or casually—and its context—whether it is a real evaluation or ordinary deployment. A naive probe measures format instead of context, flagging any benchmark-format prompt as an evaluation. Balancing format evenly across evaluation and deployment examples (paired-decorrelated training) only half-fixes this: the instrument then separates evaluation from deployment cleanly on casual-format prompts (AUC $0.98$) but only partly on benchmark-format ones ($0.83$), so it clears a single-number check while still partly measuring format. We then freeze the instrument and read it across a seven-checkpoint MBPP-Honeypot RLVR run. Its overall AUC drops only a little ($0.794 \rightarrow 0.772$) as the task's success rate rises $35.6$ points—yet that pooled number hides what it does on different prompts. On cleanly-separated prompts it barely changes, but on a borderline set (casual-format deployment prompts that start right at the decision cutoff) a small shift flips up to a quarter of its decisions, and the drift concentrates on the benchmark format that training never fully cleaned up. Tracking each decision on fixed inputs (operational disagreement) exposes the failure the pooled AUC hides, and re-fitting the instrument on post-RLVR activations recovers only $0.003$ AUC (cosine $0.92$), so periodic retraining is not the fix. We package this as an audit: validate each format separately, track how close decisions sit to flipping rather than only whether they flip, and re-measure decisions as the model changes rather than trusting pooled AUC or retraining.