Score Centering Stabilizes Off-policy Reinforcement Learning

The 30-second take

• What: The authors blame RL instability under training-inference mismatch on accumulating engine drift, derive an additive score-centering correction, and show it matches or beats importance sampling from 0.6B to 30B, especially under quantization, and composes with IS under staleness.
• Abundance angle: today, stable RL for large models is a scarce, brittle luxury because train and serve stacks never quite match. A cheap additive stabilizer is a step toward post-training as a default rather than a specialist ritual (near-term if it ports; still not a consumer product).
• Who should care: LLM post-training teams, RLHF/RLAIF engineers fighting TIM, and infra groups that quantize rollouts for speed.

What the paper actually did

Reinforcement learning of large language models is sensitive to small differences between training and inference engines — the training-inference mismatch (TIM). Fully eliminating TIM would smash rollout efficiency, so it is impractical. The authors argue that RL instability under TIM is mainly drift: a persistent bias between engines that accumulates every training step.

They derive an additive “score centering” correction that stabilizes RL under TIM by canceling that drift. Training models from 0.6B to 30B parameters, score centering alone matches or outperforms importance-sampling methods under quantization, with a larger gap as mismatch gets worse. Because the term is additive, it also composes with importance sampling; the combination beats pure IS in their staleness experiments.

What makes this disruptive

The scarce capability is stable on-policy-looking RL when rollouts come from a faster, slightly wrong engine (quantized, stale, or different kernels). If drift is the villain and an additive center cancels it, labs can keep efficient rollouts instead of chasing perfect train–infer identity.

Beating or matching IS under quantization from sub-billion to 30B, and composing with IS for staleness, is a strong systems result. It pressures the “IS or bust” toolkit.

This is an optimizer/correction paper, not a new reward model.

Why it matters (outside the lab)

Abundance lens: turning base models into useful assistants via RL is still expensive and fragile. A stabilizer that lets you keep fast mismatched rollouts is a step toward cheaper default post-training — more teams can afford RL, not only those with identical train/serve stacks.

Near-term, try the correction in your PPO/GRPO-like stack. Medium-term, whether it generalizes across algorithms and kernels decides if it is default. No calendar for free frontier models.

Reliability of the math under your mismatch is the gate.

Limitations & open questions

The abstract does not write the formula or name the exact RL algorithm. Scale range is 0.6B–30B in their setup; behavior at larger scales is unstated. “Matches or outperforms IS under quantization” is their comparison, not a universal theorem.

Completely eliminating TIM is called impractical; centering does not claim zero mismatch. Preprint ≠ guaranteed stable run. Abundance is not automatic: a correction term does not make RL cheap by itself.