Why does running AI in a loop work so well?

Why does running an AI model in a loop work so much better than asking it once? If you have used a coding agent this year, you have watched it happen. The agent writes code, runs the tests, reads the failures, patches, and runs again. By the third pass the thing works. Ask once and you get whatever the first draft happens to be. Today I want to walk you through the science of the loop, because the science comes with a boundary, and the boundary turns out to be the real finding.

Start with the study everybody cites. Reflexion, peer reviewed at NeurIPS in 2023. This one is independent academic work, no vendor stake in the result. The setup is simple. The agent writes code against the HumanEval benchmark, runs the unit tests, and when the tests fail, it writes itself a short note about what went wrong and tries again with that note in hand. That loop lifted the pass rate from 80 percent to 91 percent. Eleven points, from nothing except retries fed by test results.

That is where most of the coverage stops. Here is where it gets interesting. What was doing the work in that loop? The retrying, or the test results? Because if it is the retrying, you can loop on anything. If it is the test results, the loop is only as good as the grader you attach to it.

In 2024, a team published a paper at ICLR with a blunt title: large language models cannot self-correct reasoning yet. Flag the funding on this one out loud: Huang and colleagues work at Google DeepMind, a vendor lab. They removed the external feedback and asked models to fix their own answers using nothing except their own judgment. Performance did not improve. It often got worse, because the models flipped right answers to wrong ones. And the earlier papers that had reported self-correction gains turned out to depend on an oracle, something outside the model quietly telling it when to stop.

One study could be an outlier. So here is the meta-survey. Kamoi and colleagues, peer reviewed in TACL in 2024, independent. They went through the self-correction literature looking for cases where a model successfully corrected its own reasoning with no outside information. They found zero. The count of successful intrinsic self-correction studies in the published record was zero.

Put the three together and you get a law you can carry around: loops work when something outside the model says pass or fail. Unit tests. A compiler. A checker that knows the ground truth. Reflexion worked because the unit tests were the judge. The pattern shows up across the reasoning literature too: rank candidate answers with a real external checker and quality climbs. Remove the judge and you have a model grading its own homework, and the DeepMind result says those grades flip right answers to wrong.

Notice what this means for all the agent products being sold to you right now. The pitch is usually about the model. The evidence says the judge in the loop deserves at least as much of your attention as the brain in the loop.

So the verdict. The evidence here is strong, with a boundary. Strong because independent groups converge: an independent team demonstrated the gain, a vendor lab documented the failure mode, and an independent meta-survey confirmed the pattern across the whole literature. The boundary is that every documented win involves external feedback. Nobody has published a working loop without it.

What should you do with that? Before you build a loop, ask one question: what in my setup can say pass or fail without asking the model? If you have tests, loop on the tests. If you have a schema validator or a retrieval check, loop on that. And if nothing in your pipeline can grade the output, know that an extra self-review pass buys you a warmer feeling and roughly the same answer. Build the judge first. The loop comes second.

Whitney is up next, because the most viral coding trick of the year is exactly this kind of loop, running in the wild, with this exact question hanging over it.

Sources

• Reflexion (NeurIPS 2023) [PR][Independent]: HumanEval pass rate 80% to 91% by looping on unit-test results. Grade: Strong (with boundary).
• Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet" (ICLR 2024) [PR][Vendor: Google DeepMind]: intrinsic self-correction without external feedback does not improve performance and flips right answers to wrong; earlier gains depended on an oracle. https://arxiv.org/abs/2310.01798
• Kamoi et al., meta-survey of self-correction (TACL 2024) [PR][Independent]: zero successful intrinsic self-correction studies without outside information.

The most viral coding trick of the year is a while-loop. Is there any science behind it? People call it the Ralph Wiggum loop, and if you have spent time anywhere near AI coding circles since July 2025, someone has probably told you it changed their life. So I went looking for the evidence, and what I found is the story.

First, the trick itself. It traces to a blog post by Geoffrey Huntley in July 2025. The whole idea fits in one sentence: put a coding agent inside a while-loop and rerun the same prompt, again and again, until the work is done. Same instruction, fired at the model on repeat, named after a Simpsons character. That is the entire technique.

Now the evidence question, the one this show exists to ask. What supports the claim that this works? The primary source is the blog post, and the evidence it offers is one testimonial. Let me say the sourcing flag for this segment out loud: a personal blog, a single first-person account, no benchmark, no comparison against asking once, no comparison against any other loop.

So I went hunting for anything academic. Across everything we could find as of July 2026, the count of controlled studies on this trick is zero. Zero on coding. Zero anywhere else. There is exactly one academic touchpoint: a preprint out of CMU in 2026, on arXiv, independent, no vendor money flagged. And here is the fun part. It is about battery-pack design. The while-loop trick shows up in that paper as the baseline, the thing the authors are trying to beat with their own method. The entire academic footprint of the most shared coding technique of the year is one preprint, in a different field, where the trick plays the control group.

Let me place this next to what Michael just showed you. The loop science that exists says loops work when something external grades each pass. A codebase with a real test suite can provide that grader, so there is a plausible reason the Ralph Wiggum loop might help. Plausible is doing all the work in that sentence. Nobody has measured it. Nobody has compared it against a single careful prompt, or against any other way of spending the same tokens.

Our evidence standard has a rule built for this moment. No evidence and evidence of no effect are two different claims, and we have to say which one we mean. Here we mean the first one. Nobody has shown this loop fails. Nobody has shown it works. The comparison has simply never been run. Our standard also sets an inclusion bar, and single anecdotes sit below it. One testimonial, however sincere, is where science has yet to start.

So the verdict, said as a grade: there is no science on this, and that is the answer. This is the segment format we built the show around, because in AI the loudest techniques and the best-measured techniques are often two different lists, and you deserve to know which list you are looking at. When the whole industry is quoting one blog post, saying "nobody has tested this" out loud is the most useful sentence we can offer.

What should you do? If the Ralph Wiggum loop is saving you hours, keep using it, and be honest about what you know: you are running an uncontrolled experiment with a sample size of one, which is the same evidence Huntley had. If you want the odds on your side, attach the one thing the actual loop science demands: an external grader. Run it against a project with real tests, cap how long the loop can run, and write down what it cost and what it fixed. Your log will not be a controlled study either. It will still be more data on this trick than anyone has published.

Sources

• Geoffrey Huntley, blog post (July 2025) [Blog][Testimonial]: origin of the Ralph Wiggum loop; a while-loop rerunning the same prompt; evidence offered is one first-person account. Grade: None (below inclusion bar).
• CMU preprint (2026) [arXiv][Independent]: battery-pack design study; uses the same-prompt loop as the baseline to beat. Only academic touchpoint found as of 2026-07.
• Question pool, cluster T2, item 5 (docs/research/question-pool.md): evidence-state verification, 2026-07-04.

How many times should you let the model retry before you are burning money? You heard the first two segments. Loops with an external grader are the best-supported trick in the agent playbook, and every pass through the loop costs real tokens. So somewhere between retry one and retry infinity there is a line where you stop buying accuracy and start buying a bill. Where is it?

Straight answer first: the science here is messy, and no replicated canonical number exists. Nobody has earned the right to hand you an exact integer. What exists is directional evidence, and the directions agree more than you might expect.

Direction one: the gains are front-loaded. Refinement studies cluster their improvements in iterations one through three. A 2025 arXiv preprint called FAIR-RAG observed degradation by cycle four. More looping, worse output. Two flags on that one, spoken out loud: it is a preprint, and our notes carry no funding flag for it, so we could not tell you who paid for it. Hold it loosely. But the shape it reports, early gains and a fast fade, matches the wider refinement literature.

Direction two: past a few tries, your problem changes shape. CodeMonkeys, peer reviewed at ICML in 2025, independent, studied what happens when you throw serious sampling compute at coding problems. The finding: as compute grows, generating candidates stops being the constraint and selecting among them becomes the bottleneck. The model can produce plenty of plausible solutions. Picking the right one is what gets hard. So retry number twelve is usually money spent on a problem you already have too many answers to.

Direction three: budgets should flex with difficulty. Snell and colleagues, 2024, and flag this one as vendor-adjacent, found that allocating test-time compute by problem difficulty was more than four times as efficient as naive best-of-N sampling, and that the right method depends on how hard the problem is. There is no single dial. The same study found that on problems where a small model already had some traction, extra inference-time compute could beat a model fourteen times larger. The retry budget is real capability money. Which makes wasting it hurt twice: easy problems burn the extra passes for nothing, and very hard problems eat them without improving.

Grade this segment honestly: messy. The evidence suggests, and I am choosing the word suggests deliberately, that iterations one through three carry the value, that degradation can start around cycle four, and that a fixed retry count is cruder than a difficulty-aware budget. None of that has been replicated into a number I can state as fact. When someone quotes you a precise optimal retry count, ask for the study, because we could not find it.

Now the episode verdict, since this segment closes the episode. Three questions, three evidence states. Why does the loop beat asking once? Strong, with a boundary: proven when something external says pass or fail, and failing without that signal, with zero exceptions in the published record. The viral while-loop everyone shared this year? No science at all: one testimonial and a preprint from another field. The retry cap? Messy: gains front-loaded, one preprint reporting decay by cycle four, no canonical number.

That gives you the practical close for the whole episode in three moves, and three is the honest count. Loop with a verifier: tests, a compiler, a checker, something that grades the output without asking the model for its opinion of itself. Cap the iterations: start the cap around three, since that is where the measured gains cluster, and tune it from your own results rather than from anyone's blog post. And log the cost: tokens and dollars per task, recorded next to whether the task passed. The literature cannot yet tell you the retry number for your workload. Your log can.

Sources

• FAIR-RAG (2025) [arXiv][funding not flagged in our notes]: refinement gains cluster in iterations 1 through 3; degradation observed by cycle 4. Grade: Messy/directional.
• CodeMonkeys (ICML 2025) [PR][Independent]: as sampling compute grows, selection becomes the bottleneck rather than generation. Grade: Moderate within scope.
• Snell et al., "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters" (2024) [arXiv][Vendor-adjacent]: compute-optimal allocation by difficulty more than 4x as efficient as naive best-of-N; the right method depends on difficulty. https://arxiv.org/abs/2408.03314
• Question pool, cluster T2, item 6 (docs/research/question-pool.md): evidence-state verification, 2026-07-04.