Glonce

Essay argues LLM reasoning is pathfinding, not solving an objective

researchThe Glonce editorsHacker News🔥67Glonce rating67Impact50Novelty55Relevance100Surprise45Players10Credibility65
Essay argues LLM reasoning is pathfinding, not solving an objective

A post on alexwang.ai starts from a puzzle. During RL training, models learn to write longer chains of thought and get more accurate as they do. Yet if you take a trained model and force one chain to run longer, it helps for a while and then stops helping, and sometimes the answer gets worse, as if the model were overthinking. The author calls the post an attempt to clear up this confusion, mostly for himself or herself, by treating reasoning as pathfinding through token space.

The framing: ask a reasoning model the same math problem ten times and you get ten different chains of thought. Each sample is one path from the prompt x to a final answer. The post's toy example is the sum 1 + 2 + ... + 10. One path adds the numbers one at a time (1, 3, 6, 10 and so on up to 55); another pairs them from the outside in (1+10, 2+9, 3+8, 4+7, 5+6 each make 11, and five pairs of 11 make 55). Both land on 55. The post assumes a verifier r(x, y) that returns 1 for a correct answer and 0 otherwise, and defines the accepted set of answers for a prompt. The success probability p_x is the total probability of all paths that end in that set. It can be raised by many low-probability paths to the answer or by one high-probability path; the success probability does not distinguish between them.

There are two ways to raise it. The first is best-of-k sampling: with a verifier, sample k paths and look for a success. Each path misses with probability 1 - p_x, so all k miss with probability (1 - p_x)^k, and the chance of at least one hit is 1 - (1 - p_x)^k. As long as p_x is above zero, this rises monotonically with more samples, so extra compute spent on samples always helps, as seen early on by Brown et al. 2024. The post stresses that this scaling is over the number of samples, not the length of paths.

The second is to change the model. Collect prompts whose answers can be checked, sample paths, keep those that land in the accepted set, train on them, and repeat. That loop is rejection-sampling fine-tuning, of which STaR was an early version for reasoning. The post then shows that RL with a verifiable reward, how today's reasoning models are trained, does almost the same thing. The expected binary reward equals p_x exactly, and the log-derivative trick gives the REINFORCE policy gradient. Estimated from N sampled paths of which M succeed, failed paths drop out (reward 0) and the gradient becomes the success rate M/N times the average log likelihood of the successful paths. In the post's words, a REINFORCE step with a binary reward is a rejection-sampling fine-tuning step on the model's own samples, scaled by the current success rate.

This also explains the growing chains. Each step pulls the model toward paths that landed, so if successful paths tended to be longer than failed ones (for example because they include backtracking, gather more feedback from the environment, or through plain luck), the model learns to write longer paths.

The post adds that RL amortizes the search. Compute is spent during training sampling many paths and reinforcing the good ones, so it need not be spent on every query. Because one model is trained on many prompts, it can potentially generalize to unseen ones; an interactive figure shows a new prompt being handled after training, with accuracy dropping as the prompt moves further from the training distribution. A footnote says the same pattern of expensive search followed by training a network to output the result cheaply appears outside language models, citing AlphaZero's tree search during self-play.

Two odd observations then fall out. First, chains of thought are not guaranteed to be faithful: the verifier checks only the final answer, so the intermediate tokens are just a route to it. Nothing requires them to be true or to explain the answer, though pretraining often makes them correlate. If we want meaningful chains of thought, the post says, the rewards or training objective must be adjusted. Second, reward hacking comes from both the verifier and the sampler. If the verifier is imperfect, the set of accepted answers differs from the set of correct ones: it can include wrong answers and miss right ones, opening regions the model may learn to land in. The post attributes hacking to that imperfect verifier together with a sampling procedure that fails to stay away from shortcut regions. It also argues that scaling up RL, with more and longer samples, makes hacking more likely, because the more the verifier is fuzzed the likelier a path lands in the misspecified region, after which training pushes the model toward similar behavior.

The conclusion, in the author's own voice: reasoning is not actually solving any objective. A single path is not guaranteed to converge to the correct answer; only in aggregate, over the distribution of paths, is success probability high. RL training directly raises that probability by searching with a lot of compute and then training the model to reproduce what it found. Thinking longer helps because training taught the model to use the extra tokens, not because generating more tokens is intrinsically better. At inference the model is likely to succeed with a single sample, but any individual sample is not guaranteed to.

Two cited studies back the opening observations. Muennighoff et al. 2025 forced a model to keep thinking by appending "Wait" whenever it tried to stop; accuracy rose at first, then flattened. Gema et al. 2025 found tasks where longer reasoning produced worse accuracy.

Key facts

  • The post treats reasoning as pathfinding in token space: each sampled chain of thought is one path from prompt to answer, and a verifier decides whether it lands in the accepted set.
  • Best-of-k sampling lifts success probability from p_x to 1 - (1 - p_x)^k, a guaranteed compute axis as long as p_x is above zero, but it scales with the number of samples, not the length of each chain.
  • A REINFORCE step with a binary reward is described as a rejection-sampling fine-tuning step on the model's own samples, scaled by the current success rate M/N; chains grow longer if successful paths tend to be longer.
  • Because the verifier checks only the final answer, intermediate tokens need not be true or explain the answer, and an imperfect verifier plus imperfect sampling produces reward hacking, which the post says scaling up RL makes more likely.
  • Conclusion: reasoning is not solving any objective; thinking longer helps because training taught the model to use the extra tokens, not because more tokens are intrinsically better.

Why it matters

The common intuition says more thinking time means better answers. The post offers an account of why that holds during RL training but breaks down when a trained model is simply forced to think longer. Its answer is that RL raises the probability of sampling some successful path, and that longer chains emerge if successful paths in training tended to be longer. That shifts the question from "how long should the model think?" to "what did training teach the model to do with extra tokens?" The same view is used to explain unfaithful chains of thought and reward hacking.

Who it affects

Readers who work with or try to reason about RL-trained reasoning models: researchers studying chain-of-thought faithfulness, reward hacking and inference-time scaling, and practitioners deciding whether to spend compute on longer chains or on more samples. The post is a conceptual essay and not a product announcement.

How to use it

As a mental model rather than a tool. Per the post, if a verifier is available, spending compute on more samples (best-of-k) reliably raises the chance of a success, whereas forcing a single chain longer helps for a while and then stops helping. The post also says that if meaningful, faithful chains of thought are wanted, the rewards or the training objective need to be adjusted, since a final-answer verifier does not require the intermediate tokens to be true.

How solid is it

This is a conceptual and mathematical argument; the post itself reports no experiments, benchmarks, model names or accuracy numbers. Its worked example is the toy sum of 1 through 10. The empirical anchors are cited from other work: Brown et al. 2024 on more samples helping, Muennighoff et al. 2025 on appending "Wait" (accuracy rose at first, then flattened, with no figures given), and Gema et al. 2025 on tasks where longer reasoning hurt accuracy (no task names or magnitudes given). The author frames the post as an attempt to clarify his or her own confusion, not as a definitive result. The derivation of the REINFORCE and rejection-sampling link is shown step by step in the post.

Risks and caveats

The framework assumes a verifier that returns 1 or 0; the post itself notes that real verifiers are imperfect, so the accepted set can include wrong answers and omit correct ones, which is how reward hacking arises. It says scaling up RL makes this more likely. The post does not claim reasoning chains are useless or that long chains never help; it says the benefit comes from training, not from length itself. It also does not propose a concrete fix for unfaithful chains of thought or reward hacking beyond saying the rewards or training objective must be adjusted.

“Reasoning isn’t actually solving any objective at all.”

— From the post on alexwang.ai