The AI Inevitable
S1 · E11strong

Why does the same AI feel different in every app?

With Michael Forrester and Whitney Lee

Why does the same AI feel different in every app? Answered in three solo segments with every source funding-flagged.

In this episode · 3 segments

  1. 1Why does the same model feel different in Cursor, Copilot, and the API?
  2. 2You are not picking a model. You are picking a harness.
  3. 3How to audit the harness you already use.
Segment 1strongMichael Forrester

Why does the same model feel different in Cursor, Copilot, and the API?

Evidence · STRONG: harness-variance study (2026): harness effects 7.8x model effects.

Read the transcript

Why does the same model feel different in Cursor, in Copilot, and on the raw API? You paid for the same brain in all three places, and your gut says the answers differ anyway. Your gut is onto something real. The reason is that you have never talked to a bare model. You have always talked to a model wrapped in software, and the wrapping is where the differences live.

Start with the weights themselves. Anthropic's documentation for Claude on Amazon Bedrock says Bedrock requests are served by the same infrastructure as the first-party API. Same weights file, different door. Funding flag, said out loud: that is vendor documentation, the company describing its own product, current as of June 2026. A comparison article from Respan says Claude called through Anthropic direct and through Bedrock produces the same outputs given the same input. Flag that one too: it is a vendor-comparison piece of the kind written to rank in search results, so treat its parity claim as a floor and nothing more.

Here is the twist. The differences start on that same vendor page. The legacy Bedrock integration does not get the Files API. No server-side tools, so no code execution or web search run on the server. No Agent Skills, no MCP connector, no Message Batches. It adds a 20 megabyte cap per request and a 10 percent pricing premium on regional endpoints, and the model even wears a different ID string on each surface. Identical model, documented feature mask.

Now climb a layer, to the coding tools. A public, community-maintained archive collects extracted system prompts, and it shows Cursor, several variants of GitHub Copilot, and Claude Code each shipping their own substantial system prompt plus their own tool definitions, things like Glob and Grep. Flag: independent, community-run, and you can read every prompt yourself. Those prompts are the literal standing instructions wrapped around the same weights, and they differ on every single surface.

Then the field report. DX Heroes, 2026, a team running Claude Code, Cursor, and Copilot in production at the same time. Flag: an independent practitioner report, which means experience, no controlled experiment. By late May 2026, GitHub had shipped Claude Opus 4.8 inside Copilot, the same backbone behind Anthropic's own CLI, so this team was watching one model wear three coats. What they found: memory is a wrapper-level feature. Copilot remembers out of the box; the same model inside Claude Code does not. Cursor wins when structured design context flows into the IDE, then degrades when the work leaves the IDE. Claude Code reaches shell, filesystem, and MCP tooling while Copilot binds tightly to the repo. Their own summary: the divergence is task-wrapper fit, not raw model quality.

Two quieter variables finish the picture. First, sampling defaults. Per Anthropic's API documentation, vendor docs again, adaptive thinking on the current Opus tier is off unless you enable it, and the effort setting defaults to high. Claude Code switches adaptive thinking on and runs effort at xhigh. Two wrappers on identical weights are buying different amounts of reasoning per request. Second, there is a floor of plain noise underneath everything. Horace He and Thinking Machines Lab published a technical investigation in September 2025, with reproducible code, independent of the app vendors. Sending one prompt 1,000 times at temperature zero produced roughly 80 distinct outputs, and the dominant cause was a lack of batch invariance: the numerics shift with server load, which you cannot control. Some of what you feel between apps is the same endpoint disagreeing with itself.

Time to grade. That the weights are identical while the harnesses differ: firmly documented, straight from vendor paperwork and inspectable prompt archives. That the harness changes output quality: everything I just gave you is field reports, so on its own the honest grade is moderate. Credible, mechanistically consistent, and still anecdote. A controlled measurement exists, and it moves the grade for the episode. Whitney has it next.

Until then, two instructions. When a tool feels smarter, suspect the wrapper before the weights. And never conclude anything from a single run, because at temperature zero the same model already disagrees with itself.

Sources

Segment 2strongWhitney Lee

You are not picking a model. You are picking a harness.

Evidence · STRONG: 10-35 point agentic swings by scaffold alone.

Read the transcript

Michael just showed you the anatomy: identical weights, different wrappers, and a stack of field reports saying the wrapper changes the outcome. Field reports are experience. On their own they grade moderate. This is the measurement, because in 2026 somebody finally measured it.

A study posted to arXiv in 2026 asked a clean question. When benchmark results vary, how much of the variation comes from the model, and how much comes from the harness, meaning the software wrapper that feeds the model its prompts, its tools, and its context? Funding flag, said out loud: an arXiv preprint, and our show re-checked its headline figure against the primary source during the July 2026 three-pass verification of our Evidence Standard. The figure held. Here it is: harness effects came out 7.8 times larger than model effects. Swapping the wrapper moved results almost eight times more than swapping the model did.

Sit with that ratio for a second. The harness is everything Michael cataloged in the last segment: the system prompt, the tool definitions, the way context gets assembled, the sampling defaults. Those pieces, together, moved the results almost eight times more than the choice of model did. Every argument you have heard about which model to pick is an argument about the smaller variable.

A second line of evidence points the same direction. On agentic benchmarks, the kind where the model plans, calls tools, and executes multi-step work, identical weights swing 10 to 35 points depending on the scaffold alone. Same brain, and the score moves by double digits because the scaffolding around the brain changed.

And there is convergence, which our Evidence Standard names as the gold standard. Multiple independent CAIS 2026 groups, different teams working from different angles, landed on the same conclusion: harnesses matter more than deeper reasoning. When separate groups that were never coordinating reach one answer, seed luck and cherry-picking stop being good explanations.

So our published Evidence Standard states the rule flat: a benchmark score is a property of the model plus the scaffold plus the test-time compute plus the harness plus the data subset. Never of the weights alone. The score is a portrait of a whole system, and the weights are one sitter in a group photo.

Now draw the consequence, because this is where your own experience gets explained. If a score belongs to the whole system, then changing the app changes the test subject. Model A beats model B inside Cursor and loses to it inside Copilot, and nothing is broken. Nobody cheated. You measured two different systems in the morning and two other systems in the afternoon, and they happened to share weights. Rank flips between apps are expected behavior. Expected. The surprise would be if the rankings held. And the same arithmetic explains why a model that tops a leaderboard can disappoint inside your editor: the leaderboard graded one harness, and you are living in another.

One caveat before I grade this, because we hold our own evidence to the same rules. The 7.8 figure comes from one study, so carry the precise number with care. What earns the grade is the stack: a controlled measurement, double-digit scaffold swings, and independent groups converging.

The verdict: the evidence here is strong. Michael opened this episode asking why the same model feels different in Cursor, in Copilot, and on the raw API, and he graded the field reports moderate on their own. The measurement settles it. The harness is the bigger variable, by nearly an order of magnitude on the one direct measurement we have.

What should you do with that? When you choose an AI app, spend your evaluation time on the wrapper: what it puts in the context window and what tools it wires up. When two apps disagree on the same model, believe both results, because each one is reporting truthfully on its own harness. And stop treating a rank flip as a scandal; the score followed the system, exactly as the measurement predicts. Michael closes the episode with how to read the harness you already use.

Sources

  • Harness-variance study (2026) [arXiv]: harness effects 7.8x model effects; figure re-verified against the primary source in the Evidence Standard's July 2026 three-pass check. Internal: docs/research/evidence-standard.md, docs/research/question-pool.md (cluster T8)
  • The AI Inevitable Evidence Standard, third-party verification section: 10-35 point agentic swings by scaffold alone; a benchmark score is a property of model + scaffold + test-time compute + harness + subset. Internal: docs/research/evidence-standard.md
  • CAIS 2026 convergence ("harnesses matter more than deeper reasoning"), multiple independent groups, AI Engineer World's Fair corpus, as recorded in the Evidence Standard [Independent]. Internal: docs/research/evidence-standard.md
Segment 3Michael Forrester

How to audit the harness you already use.

Evidence · Practical segment: system prompts, context policies, sampling defaults (claude-across-platforms brief).

Read the transcript

How do you audit the harness you already use? Whitney just gave you the ratio: the wrapper moves results 7.8 times more than the model swap, and scaffold alone swings agentic scores 10 to 35 points. That sounds like bad news until you notice the difference between the two variables. The weights are a sealed box. The harness is software with documentation, and you can read it. Four things to find, then one test to run.

First, find the system prompt. A community-maintained archive on GitHub, asgeirtj's system_prompts_leaks, collects extracted system prompts, and it holds Cursor, several GitHub Copilot variants, and Claude Code side by side. Flag: independent, community-run, and you can verify every line yourself. Those prompts are the literal standing instructions wrapped around your model. Read the one for your tool. If your tool is missing from the archive, check the vendor's docs for a published prompt or behavior guide. If the vendor publishes nothing, write that down, because an unreadable prompt is itself an audit finding.

Second, learn the context policy: what gets stuffed into the window, what gets truncated, and what gets summarized when the window fills. The DX Heroes field report from 2026, flag: an independent practitioner report, so experience rather than experiment, maps how differently this plays out on one model. Memory is a wrapper-level feature. Copilot remembers across sessions out of the box, and the same model inside Claude Code does not. Cursor pipes structured design context into the window, then degrades when the work leaves the IDE. Copilot binds tightly to the repo. Claude Code reaches your shell, your filesystem, and MCP tooling. Same weights everywhere. The context policy is doing all of that.

Third, check the sampling defaults. Per Anthropic's API documentation, vendor docs, flag it, adaptive thinking on the current Opus tier is off unless you enable it, and the effort setting defaults to high. Claude Code switches adaptive thinking on and runs effort at xhigh. Two harnesses on identical weights are buying different amounts of reasoning per request, and neither one asked you.

Fourth, list the tool wiring. The prompt archive shows each coding tool shipping its own tool definitions, things like Glob and Grep. And the wiring question runs past coding tools to the API surface itself. Anthropic's Bedrock page, vendor documentation current as of June 2026, says the legacy Bedrock integration gets no Files API, no server-side code execution or web search, no Agent Skills, no MCP connector, no Message Batches, plus a 20 megabyte request cap and a 10 percent premium on regional endpoints. Identical model, documented feature mask. Your harness has a mask too, whether or not anyone wrote it down.

Now the test, because reading documentation only gets you hypotheses. Pick five tasks from your own backlog, real ones. Run the same model through two harnesses on all five. Log two numbers per task: time to an output you would accept, and the count of corrections it took to get there. Then repeat each pairing, because Thinking Machines Lab showed in September 2025, independent work with reproducible code, that one prompt sent 1,000 times at temperature zero produced roughly 80 distinct outputs. A single trial cannot tell a harness effect from serving noise. Five of your own tasks, two harnesses, repeated runs: that is an afternoon, and it answers the question for your work in a way no leaderboard can.

The episode verdict, graded per our standard. That the weights are identical across surfaces while the harnesses differ: documented and inspectable, strong. That the harness is the bigger variable: strong, resting on a controlled measurement of 7.8 times, double-digit scaffold swings, and independent groups converging. And the practical point the whole episode builds to: the bigger variable is the auditable one. You cannot open the weights, and the evidence says you do not need to.

So this week: pull your tool's system prompt from the archive or the docs, write down its context policy, its sampling defaults, and its feature mask, and run the five-task comparison before you renew a seat or pick a default.

Sources

Topics

Deployment and product surface