LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition

As data propagates through a Transformer, the norms of its hidden states grow by orders of magnitude with depth, a phenomenon variously framed as “curse of depth”, and nearly universally treated as a pathology to be suppressed. We take the opposite view. Analyzing pre-trained LLAMA like4 LLMs, we show that this growth is indicative of an emergent depth-positional encoding that the network learns to express through the only learned per- layer gain on the residual stream, the RMSNorm weight γ. We show that γ grows in magnitude and rotates in direction near-monotonically with depth, jointly encoding layer index. We make this depth conditioned encoding explicit with LayerRoPE, which replaces all per-layer γ vectors with a single shared γ and a depth-conditioned magnitude and rotational factor, adding only four scalars per layer at a net reduction in parameters and < 0.2% FLOPs. LayerRoPE consistently outperforms Pre-, Post-, and Peri-Norm, Layer-Norm Scaling, and DeepNet across a 60M–7B model ladder, and is the only approach whose loss improves monotonically as depth scales to 512 layers. Inspecting its learned schedule inverts the prevailing premise of mitigating residual variance growth: LayerRoPE does not shrink the residual stream but widens it, damping each block’s input while amplifying its output. Depth stability, our results suggest, calls not for uniform suppression of the residual stream but for depth-conditioned regulation of the computational blocks it feeds.