Can you buy better reasoning?

Does "think step by step" still do anything? In 2022 that phrase was the closest thing this field had to a magic spell, and this whole episode hangs on the question behind it: can you buy better reasoning? Three videos, three levers you can pull. This one covers the cheapest lever there is, the prompt itself.

Start where it started. A Google team, and flag that out loud, Google sells the models being tested, prompted a 540 billion parameter model with eight worked examples that showed their intermediate steps. That paper hit then state of the art on GSM8K, a grade school math benchmark, and it beat a fine-tuned model that had a trained verifier helping it. The gains showed up on arithmetic, commonsense, and symbolic tasks, and they appeared only once the model was large enough; below that scale, the gains were absent. That was chain-of-thought prompting. And unlike plenty of results from that era, it reproduced widely across labs, so the vendor flag matters less here than it usually does. The trick was real.

The 2026 question is narrower: where does the trick work, and did it survive the arrival of models that think on their own?

For the first half, we have the widest lens the literature offers. Sprague and colleagues published "To CoT or Not to CoT?" at ICLR 2025. Peer reviewed, independent, no vendor stake, so it carries full weight on our scale. It is a meta-analysis covering more than 100 papers, plus fresh evaluations across 20 datasets. The finding is blunt: chain-of-thought helps mainly on math and symbolic tasks. Across the broad run of other language tasks, the gains mostly evaporate. The magic words were a math and logic tool from the beginning; they just wore a general purpose costume for a few years.

For the second half, the baseline moved underneath the technique. Reasoning models generate their own thinking before they answer, so what does typing the incantation add on top? Meincke and colleagues at Wharton measured exactly that in 2025, independent academic work, eight models across the GPQA Diamond benchmark. On the older non-reasoning models the magic words still bought a lot, about 13 points for Gemini 2.0 Flash and 12 for Claude Sonnet 3.5. But on the reasoning models the gain collapsed: roughly 3 points for o3-mini and o4-mini, and Gemini 2.5 Flash actually dropped about 3 points when told to think step by step. And it charged for the privilege, 20 to 80 percent more latency. The model already thinks. Telling it to think buys you a rounding error and a longer wait.

Put those two findings side by side and the picture is clean. On plain models, the effect is real and it lives in a specific neighborhood: math, logic, symbols, problems with checkable intermediate steps. On reasoning models, the vendors trained the step-by-step behavior into the product itself, so the prompt that used to summon the behavior now mostly duplicates it.

Here's the grade, said out loud. The evidence on this question is strong. An independent, peer reviewed meta-analysis of over 100 papers agrees with an independent 2025 report, and both agree with the original vendor paper about where the effect lives. Strong evidence, narrow effect.

So the answer: yes, think-step-by-step still does something, on math-shaped and symbolic problems, on models that don't already reason on their own. On reasoning models it adds almost nothing you can measure, and it charges you latency for the privilege.

What should you do with that? If you run a plain model on math, logic, or anything with exact intermediate steps, keep the magic words; that is where the effect actually lives. If you pay for a reasoning model, drop the incantation and let the model do the thing you bought it for. Michael takes the next question, because the obvious follow-up costs real money: if the model now sells you its thinking by the token, is more of it worth paying for?

Sources

• Sprague et al., "To CoT or Not to CoT?", ICLR 2025 [PR][Independent]. Meta-analysis of 100+ papers plus 20 datasets: CoT gains concentrate on math and symbolic tasks.
• Meincke, Mollick, Mollick, Shapiro, "The Decreasing Value of Chain of Thought in Prompting", Wharton GAIL 2025 [Independent], arXiv:2506.07142. 8 models on GPQA Diamond: non-reasoning models gain double digits (Gemini 2.0 Flash +13.5%, Sonnet 3.5 +11.7%), reasoning models near-zero or negative (o3-mini +2.9, o4-mini +3.1, Gemini 2.5 Flash -3.3), 20 to 80 percent latency cost.
• Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", NeurIPS 2022 [Vendor: Google], https://arxiv.org/abs/2201.11903

Is paying for more thinking time worth it? Whitney just showed you the cheapest reasoning lever, the prompt, and ended on the question that costs real money: reasoning models now sell you their deliberation by the token, and the longer they think, the bigger your bill. So this video's lever is a meter. Let's find out what the meter actually buys.

Start with the case for paying. Snell and colleagues ran a systematic study of test-time compute in 2024. Flag before the findings: this work is vendor-adjacent, the authors sit close to the labs that sell inference, so it carries less weight than independent replication would. What they tested is how to spend a fixed budget of inference-time compute, and the finding is that strategy beats volume. Allocating compute by problem difficulty was more than four times as efficient as brute force, the naive approach of generating a pile of answers and picking one. Better still, with difficulty-aware allocation, a smaller model outperformed a model fourteen times its size on easy and medium problems. Read that from the buyer's side: on a big slice of everyday problems, a small model spending its thinking budget wisely beat a giant. Thinking time is real purchasing power when it lands on the right problems.

Now the other side, and it comes from the seller, which is exactly why it matters. In 2025, Anthropic published inverse-scaling work on reasoning. Say the flag out loud: this is a vendor whose revenue includes thinking tokens, and the finding runs against the vendor's interest, which on our scale makes it more credible, not less. They found tasks where longer reasoning made the model worse. On some tasks, accuracy fell from about 70 percent to about 30 percent as the reasoning ran longer. Cut by more than half, by the very feature on the price sheet. And the failure modes are recognizable: given more time to think, the model wandered into irrelevant details in the problem and got distracted by them. A company that bills for thinking documented cases where the thinking it bills for hurts you.

One vendor result would be a curiosity. This one replicated. Independent overthinking studies across 2025 and 2026 found the same shape again and again: accuracy rises as thinking increases, peaks, then declines. Rise, peak, decline. And the peak moves with difficulty. Easy problems cross the overthinking threshold early; the model has the answer almost immediately, and everything it generates past that point is where the new errors come from.

Put the two halves together and they stop looking like a contradiction. The pro-thinking result and the anti-thinking result agree on the mechanism: difficulty decides. Snell's efficiency wins came from matching compute to how hard the problem is. The overthinking curve says the same thing from the other direction; the peak sits early on easy problems and later on hard ones. Nobody in this literature found a dial you can just turn up.

Here's the grade, said out loud. The evidence is strong, and it is genuinely two-sided. A vendor-adjacent study shows adaptive thinking beats brute force and lets small models punch far above their size. The seller's own against-interest data, replicated independently, shows a peak past which more thinking costs you accuracy while it charges you money. Both findings hold. They answer different halves of the question.

So what should you do with the meter? Buy thinking for hard problems; that is the region where the curve is still climbing. Cap it on easy ones, explicitly, in your settings or your budget, because past the peak you are paying extra for worse answers. And stop treating the thinking dial as a quality dial; it is a difficulty-matching dial, and it can be set wrong in both directions. Whitney closes the episode with the third lever, the one with the biggest single number in the whole pool: taking the arithmetic away from the model entirely and letting it write code instead.

Sources

• Snell et al., "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters", arXiv 2024 [Vendor-adjacent], https://arxiv.org/abs/2408.03314. Difficulty-adaptive allocation more than 4x as efficient as naive best-of-N; smaller model with compute-optimal allocation beats a 14x larger model on easy and medium problems.
• Anthropic inverse-scaling work on reasoning models, 2025 [Vendor, against interest]: tasks where longer reasoning dropped accuracy from about 70% to about 30%; failure modes include distraction by irrelevant detail.
• Independent overthinking studies, 2025 to 2026 [Independent]: replicate the rise-peak-decline curve; easy problems cross the overthinking threshold early.

Why does letting the model write code beat letting it do the math? That is the last lever of this episode, and it comes with the biggest single number in the pool. Michael just showed you that thinking time has a peak. This lever skips the deliberation question entirely: it takes the part models are bad at and hands it to a machine that never gets it wrong.

The study is PAL, program-aided language models, from Gao and colleagues, published at ICML 2023. Peer reviewed, independent, no vendor stake, so it carries full weight on our scale. The setup: instead of asking the model to reason in words and compute in its head, PAL has the model write its reasoning as a Python program, and a plain interpreter runs that program to produce the answer. The model still does the thinking. The interpreter does the arithmetic. That split is the whole experiment, because it isolates which part was failing. And the answer is execution. On arithmetic with large numbers, chain-of-thought prompting scored 23 percent. Same task, PAL scored 61 percent. The models were laying out correct plans and then fumbling the actual computation, the classic right plan, wrong arithmetic failure. Take the computation away and accuracy nearly triples. Across thirteen reasoning tasks, PAL set new state of the art, and a smaller model with an interpreter beat much larger models computing in their heads.

One study is a result. Here is the replication. A second team published Program of Thoughts in TMLR in 2023, peer reviewed, academic work. Same principle, separate implementation: the model writes the computation as code and delegates the execution. Across math word problem benchmarks, it averaged about a 12 percent gain over chain-of-thought. Two teams, two codebases, one conclusion: when the task contains real computation, the interpreter wins.

Now the boundary, because every lever in this episode has one. A peer reviewed paper at AAAI 2024 mapped where the program trick works and where it stops. The gains are specific to exact computation: arithmetic, anything with a single checkable answer the code can produce. Outside that zone, on tasks that need judgment or commonsense rather than calculation, routing through code degrades performance. The calculator is a specialist. Send it specialist work and nothing else.

So the grade for this lever: the evidence is strong, peer reviewed and independent twice over, with a clearly measured boundary. That makes it the cleanest result of the three we covered.

Which brings us to the episode verdict, all three levers graded. Lever one, the magic words: strong evidence, narrow effect. Chain-of-thought helps math-shaped and symbolic work on plain models, and on reasoning models it adds a rounding error at real latency cost. Lever two, thinking time: strong and two-sided. Adaptive thinking beats brute force on hard problems, and the seller's own data shows a peak past which more thinking makes answers worse. Lever three, code: strong within its boundary. For exact computation, handing execution to an interpreter nearly tripled accuracy in the head-to-head, and it degrades things outside that boundary.

Notice what all three verdicts share. None of these levers is a general upgrade. Each one works in a specific neighborhood, and each one has a measured place where it stops working or turns harmful. The science in this episode is unusually good by AI standards: peer reviewed meta-analysis, against-interest vendor data, independent replications. What it supports is matching, a lever to a failure, and nothing broader.

The playbook, then, for the whole episode. Running a plain model on math, logic, or symbols? Keep the step-by-step prompt; that is its home turf. Paying for a reasoning model? Drop the incantation and budget the thinking instead: spend on hard problems, cap it on easy ones, because past the peak you are buying errors. And whenever the task contains actual arithmetic or anything with an exact answer, stop asking the model to be a calculator. Give it one.

Sources

• Gao et al., "PAL: Program-aided Language Models", ICML 2023 [PR][Independent], https://arxiv.org/abs/2211.10435. 61% vs 23% against chain-of-thought on large-number arithmetic; new state of the art across 13 reasoning tasks; smaller model plus interpreter beats much larger pure-LLM models.
• "Program of Thoughts", TMLR 2023 [PR]: about 12% average gain over chain-of-thought on math word problem benchmarks.
• AAAI 2024 boundary study [PR]: gains specific to exact computation; performance degrades on tasks outside that scope.