Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

The 30-second take

• What: The authors measure a reasoning step’s real importance as its advantage — how much including it changes expected reward, estimated with Monte Carlo rollouts — then test whether LLM judges can spot high-advantage steps from the text alone.
• Why it matters (abundance angle): Safer, cheaper cognitive tools need supervision that actually tracks what the model is doing. If critics reward fluent text instead of causal steps, we waste compute and trust on a scarce expert-judgment problem that looks solved.
• Who should care: Alignment researchers, process-reward and critic-model builders, eval teams, and anyone treating CoT traces as interpretability for high-stakes assistance.

What the paper actually did

Reasoning traces from chain-of-thought models look like a readable story of how an answer was reached. A lot of current practice treats that story as evidence: LLM judges diagnose errors, score faithfulness, and give step-level labels for process reward models and generative critics. Those uses assume the wording of a step carries information about its functional role.

This paper asks a sharper question: does the text of a step encode which steps actually matter? The authors define importance as advantage — the change in expected reward (for example, getting the final answer right) from including that step — and estimate it with Monte Carlo rollouts. Using those estimates as ground truth, they test whether LLM judges can identify high-advantage steps. Capable judges beat a prevalence baseline but stay well short of a noise ceiling. Fine-tuning a step-level critic helps a lot on incorrect responses and much less on correct ones, which the authors read as evidence that step importance is only partly recoverable from the trace text.

What makes this disruptive

If this result holds, it undercuts a popular shortcut: treating legible CoT as interpretability and using judges or critics as if they can see causal structure. Process reward modeling and generative critics are already being used to steer training. A systematic gap between judged importance and rollout-measured advantage means those pipelines may be optimizing a readable narrative rather than the steps that change outcomes.

The scarcity under pressure is trustworthy oversight of cognitive labor. Today, knowing which reasoning steps matter still looks like an expert, expensive audit. The paper does not claim that traces are useless; it claims they are an incomplete channel. That is a roadmap signal for anyone building cheap default supervision — not a claim that interpretability is finished or that consumer tools will suddenly become honest.

Why it matters (outside the lab)

Abundance lens (today’s luxuries → tomorrow’s defaults): capable assistance only becomes a default if we can cheaply tell when the model’s reasoning is doing real work versus performing fluency. This paper says the text of a trace is a leaky, partial signal of that work.

Scarcity today is expert judgment about whether a model’s intermediate steps deserve credit. If critics cannot recover advantage on correct traces, we risk paying for supervision that looks scientific while remaining closer to literary criticism. Horizon is near-term for research practice (years, not decades) if labs update training and evals — and still not a consumer-product promise.

Near-term: treat judge scores and process rewards as hypotheses to check against rollout advantage, not as a window into the model. Medium-term: cost and reliability of better supervision — not prettier traces — decide whether abundant AI assistance stays trustworthy.

Limitations & open questions

This is a preprint. Advantage is defined via expected reward and estimated with Monte Carlo rollouts; those estimates are themselves noisy, and the paper’s noise ceiling is a reminder that ground truth is approximate. Results speak to whether judges recover high-advantage steps from text, not to every notion of faithfulness or every model family.

Fine-tuning a critic helped more on incorrect than correct responses; that split may depend on the tasks, models, and reward used. Preprint ≠ product: nothing here ships a safer assistant on a calendar. Abundance is not automatic — better process rewards still need independent replication, and treating traces as interpretability remains a design choice, not a solved science.