The AI Inevitable
S1 · E13moderate

Spec-driven development: where is the science?

With Michael Forrester and Whitney Lee

Spec-driven development: where is the science? Answered in three solo segments with every source funding-flagged.

In this episode · 3 segments

  1. 1Does writing a spec first make the model produce better work?
  2. 2Branded spec-driven toolchains: where is the benchmark?
  3. 3When a plan helps and when it is ceremony.
Segment 1moderateMichael Forrester

Does writing a spec first make the model produce better work?

Evidence · MODERATE: plan/decompose studies show single-digit to 25% relative gains (spec-driven brief).

Read the transcript

Everyone in AI engineering right now tells you to write the spec before you let the model touch the problem. So here is the season finale question: does making a model plan first actually produce better work, or does it just feel more disciplined? There are controlled studies on this. Let's walk them one at a time.

Start with a Google Research team: Zhou and colleagues, published at ICLR 2023, peer reviewed. The technique is called least-to-most prompting. You have the model break the problem into sub-problems, then solve them in order. On SCAN, a compositional generalization benchmark, least-to-most reached about 99 percent accuracy while standard chain-of-thought managed 16. And it kept working on problems harder than the ones shown in the prompt. This one comes from Google, so flag that. It is a prompting-technique paper rather than a product pitch, but you should know who ran it.

Second study. Wang and colleagues, ACL 2023, peer reviewed, and there is no vendor funding flag on file for this one. They call it Plan-and-Solve prompting: ask the model to devise a plan, then execute the plan step by step. They tested ten datasets across three categories, math, commonsense, and symbolic reasoning, all on GPT-3. The plan-first variant beat zero-shot chain-of-thought, in the authors' words, by a large margin, and on math it approached the performance of prompts carrying eight worked examples.

Third, the one closest to your day job. Jiang and colleagues, self-planning code generation, peer reviewed in ACM Transactions on Software Engineering and Methodology in 2024, again no vendor flag on file. The model outlines the steps of the solution first, then implements against its own outline. The result: up to 25.4 percent relative improvement in pass at one over direct generation, and up to 11.9 percent over chain-of-thought code generation. Human raters also scored the planned code as more readable.

Three peer-reviewed wins. Now the counterweight, and it is a serious one. Stechly, Valmeekam, and Kambhampati at Arizona State, published at NeurIPS 2024. This one is independent, no vendor money in the result. The paper is titled Chain of Thoughtlessness, which tells you where it lands. They ran controlled experiments on Blocksworld, a classic planning domain, and found that decomposition-style prompts helped only when the examples were exceedingly specific to the exact problem class. The moment the queried problem grew larger than the demonstrated ones, the improvement collapsed. Their conclusion: the gains come from carefully engineered, problem-specific prompts. The model does not walk away with a transferable planning procedure.

One caveat before the grade. The foundational results were measured on GPT-3-class models. Current models already do a lot of planning internally, so the marginal benefit of forcing an explicit plan may be smaller today than the 2023 numbers suggest. Treat the exact percentages as era-bound.

Here is the verdict. The evidence that plan-first prompting improves multi-step work is moderate. The wins are peer reviewed and they repeat across math, reasoning, and code, with relative gains running from single digits up to about 25 percent. The claim stops short of strong on our standard for two reasons: no independent, differently funded group has replicated the effect on current models, and the Arizona State result shows the gains end at the edge of what the prompt demonstrated. So say it the way our rubric says: the evidence suggests planning first helps on problems that decompose. It does not promise your plan will hold one step past that.

What should you do with this? For multi-step, decomposable work, code features, multi-hop reasoning, data pipelines, have the model plan first. That is the measured case, and the written plan gives you an artifact you can review even when the accuracy delta is modest. Whether the branded spec-driven toolchains earn any of this credit is a separate question, and Whitney has that one next.

Sources

  • Zhou, Schärli, Hou, Wei et al. (2023), Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR 2023 [PR][Vendor-funded: Google Research] : https://arxiv.org/abs/2205.10625
  • Wang, Xu, Lan et al. (2023), Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models, ACL 2023 [PR][no vendor flag on file] : https://aclanthology.org/2023.acl-long.147/
  • Jiang et al. (2024), Self-planning Code Generation with Large Language Models, ACM TOSEM 2024; arXiv:2303.06689 [PR][no vendor flag on file] : https://arxiv.org/abs/2303.06689
  • Stechly, Valmeekam, Kambhampati (2024), Chain of Thoughtlessness? An Analysis of CoT in Planning, NeurIPS 2024 [PR][Independent: Arizona State] : https://arxiv.org/abs/2405.04776
  • Evidence brief: docs/research/spec-driven-engineering-reasoning.md (verified June 2026)
Segment 2Whitney Lee

Branded spec-driven toolchains: where is the benchmark?

Evidence · NONE: zero controlled evidence any branded SDD methodology beats plain prompting as of 2026-07.

Read the transcript

Michael just walked you through the planning science. Here is my question: the branded spec-driven toolchains, the ones with names and logos and getting-started guides, where is the benchmark?

Let me tell you what I went looking for. A controlled study. Any controlled study. Take one set of tasks, run them through a branded spec-driven methodology, run a matched set through plain prompting, and compare the output. That is the whole experimental design. It fits on an index card.

As of July 2026, we could not find one. No controlled study shows any branded spec-driven methodology beating plain prompting. That is the evidence state for this segment: none.

Now let me walk you through what does exist, because something exists, and it matters what kind of thing it is.

The most visible brand is GitHub's Spec Kit, open sourced in September 2025. The workflow is Spec, then Plan, then Tasks, then Implement. The launch post argues that an explicit spec removes ambiguity, because models, in GitHub's words, "are exceptional at pattern completion, but not at mind reading." That is a reasonable argument. I want to be precise about what it is, though. It is a vendor blog post. It contains zero benchmarks and zero comparative metrics. Say the funding flag out loud, the way we always do: Spec Kit is GitHub's product play, built to move users from vibe coding toward a repeatable Copilot and agent workflow. The company selling the workflow wrote the argument for the workflow.

So what are the brands? Methodology documentation. A written-down process, justified by reasoning about how models behave. And that reasoning can be good reasoning. Michael just showed you peer-reviewed studies where plan-first prompting produced real gains on multi-step problems. The branded toolchains borrow that credit. But those studies measured prompting techniques on benchmark tasks. Nobody has run those measurements on the branded workflows themselves. Related hypotheses. One of them has evidence.

Here is where I want to slow down, because there are two claims in the room, and they keep getting merged into one.

Claim one: this toolchain is useful. Plenty of engineers will tell you it is. It gives a team a shared process, a reviewable artifact, a place to put decisions before the code exists. I believe those engineers are describing their real experience.

Claim two: this toolchain produces measurably better output than plain prompting. That is a different claim. It has a test. The test has not been run.

Useful is a different claim from proven, and our job on this show is to say both claims out loud and keep them separate. You can hold the first one and stay honest that the second one is open.

This segment is our no-science format doing exactly the job we built it for. Our evidence standard has a policy for topics like this one: when we cannot find good science, we say so on air, and we explain why the question cannot be answered yet. Here is the why for spec-driven development. No controlled comparisons exist. There are no released artifacts anyone could re-run. There is no replication by a group without a stake in the outcome. The numbers that circulate are vendor-adjacent or anecdotal.

Which brings me to the part I most want you to take home. Since nobody has run the test, every strong opinion you have heard about spec-driven development, for it or against it, is testimony. The conference talk. The viral thread. The engineer who swears it changed their team. The skeptic who calls the whole thing ceremony. All of it is testimony. Some of it is probably even correct. Nobody currently knows which part, including the people who sound most certain.

The verdict, as a grade: on the branded spec-driven toolchains, there is no science on this, and that is the answer.

What should you do? If a spec-driven toolchain feels useful to your team, use it, and call it what it is: a process choice, adopted on judgment. If anyone tells you it is proven to beat plain prompting, ask for the study. As of July 2026, there is not one. Michael is up next with when a plan earns its keep and when it is ceremony.

Sources

Segment 3Michael Forrester

When a plan helps and when it is ceremony.

Evidence · Synthesis on boundary conditions.

Read the transcript

Whitney just showed you the missing benchmark. Here is the last question of the season: when does writing a plan actually help, and when is it ceremony?

The studies from this episode give us real boundary conditions. Let me lay them out.

First boundary: the gains live on multi-step problems that decompose. The three peer-reviewed wins I walked in the opener, least-to-most from Google Research, Plan-and-Solve from Wang and colleagues, self-planning code generation from Jiang and colleagues, all measured work that splits into ordered sub-steps. Math word problems. Multi-hop reasoning. Code you can outline before you write it. Inside that shape, planning first paid, with relative gains from single digits up to about 25 percent.

Second boundary: the gains stop at the edge of what the prompt demonstrated. The Arizona State study, independent, no vendor money in the result, ran controlled experiments on Blocksworld and watched the improvement collapse the moment the queried problem grew larger than the examples in the prompt. The engineering paid off on the exact problem class it was tuned for, and the model did not carry a planning procedure to the next size up. So a plan that tested well on your small tasks carries no promise about your big ones.

Third boundary: on the wrong task, deliberation has a price. A 2024 preprint, so weight it below the peer-reviewed work, found chain-of-thought reduced performance on tasks where deliberation also makes humans worse, pattern and implicit-learning tasks among them. And every forced plan adds latency, cost, and one more surface where an early error can snowball. If the job is a single lookup or a short transform, the ceremony is pure overhead.

Fourth boundary: long horizons are their own problem. A 2026 preprint, again weight accordingly, analyzed long-horizon decision-making and found that step-by-step reasoning picks locally plausible actions and cannot revise early commitments for long-term payoff. Writing a longer plan up front does not repair that. Long-horizon planning stays an open weakness even for strong reasoning models.

Put the four together and you get the working rule. A plan helps when the task genuinely splits into ordered, checkable sub-steps, and when your real tasks stay close to the ones you validated on. A plan is ceremony when the task is atomic, when it runs on pattern recognition rather than sequence, or when you trust the plan instead of testing the output. The discipline that survives every study we read this episode: treat the spec as a contract you verify. Keep the pieces small and testable, and check the result against the spec, because the plan does not certify the answer.

What evidence would change these grades? For the branded toolchains, one honest benchmark: a branded spec-driven methodology against plain prompting, on registered tasks, with the harness disclosed, run or replicated by a group with no stake in the result. Whoever publishes that moves the grade, in either direction, and we will read it on air. For plan-first prompting itself, replications on current models would tell us how much of the 2023-era gain survives on systems that already plan internally.

So the episode verdict. Plan-first prompting for multi-step, decomposable work: the evidence is moderate, real gains that are conditional on task shape and that do not generalize past the complexity you demonstrated. Branded spec-driven toolchains: no controlled evidence as of July 2026, methodology documentation until someone runs the test. The line between help and ceremony is the shape of the task, and you can check that shape before you spend the tokens.

The practical close for the whole episode. Match the ceremony to the complexity. Have the model plan first on work that decomposes, keep the sub-steps small enough to test, and verify the output against the spec. Skip the elaborate plan on atomic tasks. Before you adopt anyone's branded workflow, run a before-and-after eval on your own tasks, because right now your eval would be the first controlled evidence in existence.

Sources

Topics

Spec-driven development