Does context make the model smarter, or dumber?

Does giving the model more context make it smarter, or dumber? You would think smarter. Every model release advertises a bigger context window than the last, and the pitch practically writes itself: paste in everything you have and let the model find what it needs. Today I want to show you what happened when researchers actually tested that pitch, because the answer decides how you build every AI feature you ship.

Start with the study that named the problem. Liu and colleagues, a Stanford-led team, published "Lost in the Middle" in TACL in 2024. That one is peer-reviewed and independent: no vendor had a stake in the result. They gave models multi-document question answering and simple key-value retrieval tasks, then moved the relevant information to different positions inside the input. The result is a U-shaped curve. Models scored best when the answer sat at the very beginning or the very end of the context, and accuracy fell off a cliff in the middle: a drop of more than thirty percent for mid-context information. Same facts, same question, different position. Thirty points gone. And it happened even in models explicitly marketed as long-context.

One study is a finding. Replication is what makes it science, so here come the replications.

RULER was peer-reviewed at COLM in 2024, and it comes from NVIDIA, so flag that: this one is vendor work. The RULER team went past the simple find-the-planted-fact tests and measured each model's effective context length, meaning the input size at which the model still actually performs. The effective lengths came in far under the advertised windows. The number printed on the model card and the number measured in the lab are two different numbers.

Then NoLiMa, peer-reviewed at ICML in 2025, mixed funding, so a partial vendor flag on the author list. NoLiMa stripped out the literal word-matching cues that make needle-in-a-haystack tests easy, so the model has to understand the content instead of pattern-matching it. Eleven of thirteen models claiming windows of 128,000 tokens fell below half of their own short-context accuracy by 32,000 tokens. Sit with that. Half their accuracy, gone, at a quarter of the advertised length.

And for breadth, Chroma's "Context Rot" report from July 2025 tested eighteen models and found reliability degrading as input grows, across every family it tested, well below the window limits. Big flag here: Chroma sells a retrieval database, so "long context rots" is a commercially convenient finding for them. They open-sourced the benchmark code, which is the only reason it earns a mention, and you should still weight the peer-reviewed work above it.

So the phenomenon is settled. An independent academic team found it. A vendor replicated it. A harder test confirmed it at a quarter of the advertised length. An interested company measured it across eighteen models with open code. More context degrades quality, and the degradation starts long before the window is full.

What is still argued is why. Anthropic's engineering guide blames attention being stretched across n-squared pairwise token relationships. Maybe so. That comes from a vendor blog telling a plausible story, and nobody has run the controlled study that settles the mechanism. So hold both halves in your head: the degradation is replicated fact, and the explanation is an open fight.

The evidence grade here is strong. Multiple groups, different funders, peer-reviewed venues, converging numbers. On this question we get to say it plainly: the science says more context makes the model dumber.

So here is what you do. Put the information that matters at the very start or the very end of your prompt, never buried in the middle. That is the most replicated, cheapest win in this whole literature. Trim before you add, because a short prompt full of signal beats a long prompt padded with noise. And treat the context length on the model card as unverified marketing until you have measured where accuracy falls off on your own task. Next video, Michael takes the harder question: when the output comes out wrong, can you find the piece of context that broke it?

Sources

• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL 2024) [PR][Independent]. 30%+ accuracy drop for mid-context information. https://arxiv.org/abs/2307.03172
• RULER (COLM 2024) [PR][Vendor: NVIDIA]. Effective context length far under advertised windows.
• NoLiMa (ICML 2025) [PR][Mixed]. 11 of 13 models claiming 128K fell below half their short-context accuracy by 32K.
• Hong, Troynikov, Huber, "Context Rot: How Increasing Input Tokens Impacts LLM Performance" (Chroma, July 2025) [Vendor report, open code]. https://www.trychroma.com/research/context-rot
• Anthropic, "Effective Context Engineering for AI Agents" (Sept 2025) [Vendor blog]. Mechanism claim (n-squared attention stretch), not a controlled study. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Whitney just showed you the replicated science: more context measurably degrades a model's accuracy. Which sets up my question. When the answer comes out wrong, can you find the part of the context that broke it? Think about what you would want here. A debugger. A stack trace for prompts. Point at the bad output, get back the exact span of input that caused it, fix that span, move on. Every other layer of your stack has tooling like that. So does it exist for context? Sort of. And the "sort of" is the whole story.

Here is the closest thing science has produced. TracLLM, peer-reviewed at USENIX Security in 2025, independent, no vendor stake. TracLLM traces a model's output back to the spans of context responsible for it. That is exactly the shape of tool you want. Now notice where it was published: USENIX Security. This is a security paper, built and evaluated as a security tool, because the people who most urgently need to know which span of context caused an output are security researchers studying attacks on these systems. Context attribution is being born in the security field, on security problems, judged by security criteria.

What does not exist, as of July 2026, is a validated debugging methodology for practitioners. Nobody has taken attribution methods like these, put them in the hands of working engineers, and measured whether they find the broken span faster or more reliably than guessing. No study establishes a workflow. No benchmark scores debugging tools on everyday context failures. The algorithms exist in one corner of the literature; the practice has no science at all. And that gap has a real cost: without attribution, you cannot tell whether your fix worked for the reason you think it did. You changed the prompt, the output improved, and you wrote yourself a story about why. That story is untested.

What you do have is the failure map from the degradation studies, and it tells you where to point the flashlight. Position first. The Lost in the Middle result, peer-reviewed and independent, found accuracy dropping more than thirty percent for information stranded mid-context, so the middle of your prompt is your first suspect. Distractors second. Chroma's Context Rot report, and flag it, Chroma sells a retrieval database, found that a single distractor sentence lowers accuracy, that four distractors compound the damage, and that near-miss content, the stuff that looks relevant without being the answer, does the most harm. So your most dangerous context is the plausible-but-wrong paragraph, and it is probably sitting in the middle of your prompt.

Which brings us to the honest state of context debugging today: ablation by hand. Take the failing prompt. Remove one piece of context. Rerun. Compare. Put it back, remove a different piece, rerun again, and keep going until the failure moves. It is slow, it costs tokens, and it is the same brute-force loop experimenters used in every field before real instruments arrived. As of July 2026, that loop is your best validated tool, because it is the only one you can trust end to end.

So grade this one messy. The attribution algorithms are real, peer-reviewed, and independent. The practitioner methodology is missing entirely. You are watching a debugging science get born, and it is being born sideways, in the security literature, one field over from where most of us need it.

Here is what to do while it grows up. Assemble your context from identifiable, separable pieces, because you can only ablate what you can remove cleanly. Log what the model actually saw on every call, because you cannot ablate what you cannot reconstruct. And when an output goes wrong, work the failure map: check the middle of the prompt and the plausible-looking distractors before anything else. Whitney closes the episode next with the question that turns out to have no science behind it at all: who should control what the model sees, your app or your provider?

Sources

• TracLLM (USENIX Security 2025) [PR][Independent]. Attribution method tracing model outputs to responsible context spans; security literature, no validated practitioner methodology.
• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL 2024) [PR][Independent]. 30%+ mid-context accuracy drop. https://arxiv.org/abs/2307.03172
• Hong, Troynikov, Huber, "Context Rot" (Chroma, July 2025) [Vendor report, open code]. Distractor findings: one distractor lowers accuracy, four compound; near-miss distractors most damaging. https://www.trychroma.com/research/context-rot

Should your app control what the model sees, or should the provider handle that for you? This might be the loudest architecture argument in AI engineering right now, and I want to tell you what the science says about it. Here is the problem with that plan. I went looking for the science, and there is none.

First, the two camps, so you know exactly which argument I mean. Camp one says your application should own context assembly: your code decides what gets retrieved, what gets summarized, which pieces of history survive, and what the model sees on every single call. Camp two says push that job to the provider: let the platform's managed memory and retrieval features decide, because the vendor knows its own model best and you have better things to build. Teams are picking sides right now, and some are rearchitecting real systems around whichever camp their most confident engineer belongs to.

Now comes the part of the show where I walk you through the studies comparing the two approaches. Except I cannot, because as of July 2026 there are zero controlled comparisons of client-side versus server-side context assembly. Zero. No benchmark. No head-to-head. No peer-reviewed evaluation of the question in any venue we could find. We checked for this episode and came up empty, and under our published evidence standard, saying so out loud is the content.

The closest adjacent work is MemGPT, from 2023, an arXiv preprint, independent. MemGPT explored managing a model's memory outside the context window automatically, and it matters here as an existence proof: context assembly is a real, designable system, interesting enough to study. What MemGPT never did is compare who should own that system, your app or your provider. It is adjacent, it is three years old, and it is not an answer.

What you will find in quantity is vendor guidance. Anthropic's context-engineering guide from September 2025, a vendor blog, so flag it, says to find the smallest possible set of high-signal tokens, and names techniques: compaction, just-in-time retrieval, structured note-taking, sub-agents with clean windows. Useful reading. It is engineering advice, consistent with the degradation science I walked you through in video one. It is not a controlled comparison of client against server, and it does not pretend to be.

So when someone tells you with total certainty that owning your context pipeline is the only serious way to build, or that provider-managed context is obviously the future, ask for the citation. There is not one. Every strong opinion you have heard on this question, from conference keynotes to your own team's architecture reviews, is architecture taste. Taste is fine. Taste is what fills the room before evidence shows up. It should just be labeled as taste.

The grade on this question: there is no science on this, and that is the answer. Not messy, like Michael's attribution question. None.

Which lets me land the whole episode. Does context make the model smarter, or dumber? Here is where the evidence stands. The degradation is strong, replicated science: Lost in the Middle, RULER, NoLiMa, independent and vendor groups converging on the same result, with accuracy falling well before the advertised window fills. Finding the context that broke a bad answer is messy: real attribution algorithms exist in the security literature, no validated practitioner methodology exists anywhere, and ablation by hand is the working tool. And the ownership question, app versus provider, has no science at all.

So here is the episode's practical close, sized to that evidence. Curate small: assemble the smallest set of high-signal tokens you can manage, and put the critical information at the start or the end, never the middle. Measure degradation on your own tasks at your own context lengths, because the advertised window is a claim and your own evaluation is the verification. And when somebody tells you they are certain about server versus client, treat them as uncited.

Sources

• MemGPT (2023) [arXiv][Independent]. Closest adjacent work: automated memory management outside the context window; no client-vs-server comparison.
• Anthropic, "Effective Context Engineering for AI Agents" (Sept 2025) [Vendor blog]. Curation techniques (compaction, just-in-time retrieval, structured note-taking, sub-agents); engineering guidance, not a controlled study. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
• Absence-of-evidence check: docs/research/question-pool.md (cluster T1, verified 2026-07-04). Zero controlled comparisons of client-side vs server-side context assembly found.