Does prompting actually matter?

This whole episode asks one question: does prompting actually matter? Three segments, three pieces of the answer, and every source gets its funding flagged out loud. We start with the most famous prompting claim of all. Does telling an AI "you are an expert" actually do anything?

You have seen the advice everywhere. Start your prompt with "you are a world-class mathematician" or "act as a senior engineer" and the answers supposedly get better. It may be the single most repeated prompting tip on the internet. So let's ask it the way a scientist would. Does the persona improve accuracy, yes or no?

Somebody ran that test properly. Zheng and colleagues, published in the Findings of EMNLP 2024. Peer reviewed, and this one is independent, no vendor money on the result. They took 162 personas, 50 interpersonal and 112 occupational, and ran them against nine models from four different families, on 2,410 factual questions from the MMLU benchmark. Not an afternoon of vibes with one chatbot. A controlled sweep.

The result: adding a persona to the system prompt did not improve performance over a plain, no-persona control. Even when they matched the persona to the subject, an expert in the exact field the question came from, the gain was about 0.4 percent. And here is my favorite detail. They asked whether you could at least learn to pick the best persona for each question. No selection strategy they tried beat picking a persona at random. The upside is tiny, and you cannot even aim at it.

Maybe that was just those nine models, though. None of them were frontier systems. So here is the replication. Wharton's Prompting Science Report 4, December 2025, independent university research, re-ran the persona question on current frontier models. Same null result. The expert costume did not buy accuracy on the newest models either.

Then 2026 brought the study with the twist. The PRISM paper, from a University of Southern California team, an arXiv preprint, independent. On MMLU they measured a baseline of 71.6 percent accuracy. Add a minimal persona and it drops to 68.0. Add a detailed expert persona and it drops again, to 66.3. Sit with that for a second. The more expert you told the model it was, the worse it did on factual questions. The authors' explanation: expert framing flips the model into act-the-part mode, and playing the part competes with recalling what it actually knows.

PRISM also found what personas are good for. A few task categories saw small lifts, and tone, style, and audience fit genuinely moved. Personas shape how the model talks. The tax landed on math, coding, and memorized knowledge, which are the places people most want the expert boost.

So where did the belief come from? There was real science under it once. Kong and colleagues, peer reviewed at NAACL 2024, independent, reported big reasoning gains from role-play prompts. On one arithmetic benchmark, accuracy jumped from 53.5 percent to 63.8. But look at the test subject: GPT-3.5, a 2023-era model. On that generation, a persona may have worked as a crude trigger for step-by-step reasoning. On newer models the effect shrinks toward zero or turns negative. The advice fossilized while the models moved on.

Verdict. The evidence here is strong, and it is negative. Three independent groups, across three years and three model generations, converge on the same answer. Personas do not improve factual accuracy. Detailed expert personas can measurably lower it. What they reliably change is tone.

So keep the role when you want a voice. "Explain this to a CFO." "Write like a patient teacher." That is what personas are for, and the science backs it. Drop the role when you want correct answers, and spend that effort on techniques with actual evidence behind them. Which techniques are those? That is exactly where Michael picks it up.

Sources

• Zheng, Pei, Logeswaran, Lee & Jurgens, "When 'A Helpful Assistant' Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models," Findings of EMNLP 2024 (arXiv:2311.10054) [PR][Independent], https://arxiv.org/html/2311.10054v3
• Wharton, Prompting Science Report 4, December 2025 [Independent]
• Hu, Rostami & Thomason (USC), "Expert Personas Improve LLM Alignment but Damage Accuracy" (PRISM), arXiv:2603.18507, March 2026 [arXiv][Independent], https://arxiv.org/abs/2603.18507
• Kong et al., "Better Zero-Shot Reasoning with Role-Play Prompting," NAACL 2024 (arXiv:2308.07702) [PR][Independent], https://arxiv.org/abs/2308.07702

Whitney just showed you that the expert persona does nothing for accuracy. So which prompt tricks do have evidence? Some do, with serious numbers behind them. But nearly every one comes with an expiration date, and the date is the part almost nobody quotes.

Start with the oldest and most canonical: few-shot examples. Show the model a handful of worked examples before your real question. That result comes from Brown and colleagues at NeurIPS 2020, the GPT-3 paper. Peer reviewed, but it comes from OpenAI, the vendor whose model it describes, so flag that. Flag or no flag, few-shot became the founding trick of prompting, and on the chat models of that era it reliably helped. For five years, "always give examples" was the closest thing prompting had to a law.

Now the reversal. In 2024, Microsoft reported that few-shot examples actively hurt o1, one of the new reasoning models. And DeepSeek's R1 technical report says the same thing about their own reasoning model: few-shot prompting degrades performance. Both of those are vendor reports, so flag them. But notice which direction they point. These are companies telling you that a beloved technique makes their own products worse. When a vendor reports against its own interest, that is when a vendor claim is most believable. The moment models started doing their own internal reasoning, the canonical trick flipped sign.

Next trick: "think step by step," chain-of-thought. Here we finally get a meta-analysis, which is the best evidence type in our hierarchy. Sprague and colleagues, ICLR 2025, peer reviewed and independent. They reviewed more than 100 chain-of-thought papers, then ran their own evaluation across 20 datasets and 14 models. The finding: chain-of-thought delivers large gains on math and symbolic reasoning, and close to nothing everywhere else. On the MMLU benchmark, step-by-step prompting performed about the same as answering directly, unless the question contained an equals sign. That is how narrow the win is. It shows up when there is something to compute.

And the era split hits this trick too. On reasoning models, which already think before they answer, Wharton's 2025 report, independent, measured explicit chain-of-thought prompting adding roughly 3 points or less, at real latency cost. You are paying wait time to request a behavior the model already runs internally.

Now the flashy one: multi-agent debate. Spin up several model instances, have them argue, and let the truth emerge from the committee. It demos beautifully. Here is what happened when three independent groups tested it, at ICML 2024, at ACL 2024, and again in 2025. Each compared debate against compute-matched self-consistency, which is the boring baseline: ask one model the same question several times and take the majority vote. Debate lost all three times. Give a single model the same compute budget and the committee earns nothing. A Stanford preprint from April 2026, independent, tightened the screws further. Hold the thinking-token budget constant, from 100 tokens up to 10,000, and single-agent setups matched or beat every multi-agent variant tested.

So here is the verdict, and the evidence is strong: prompt tricks are real, and they are perishable. Few-shot examples, canonical since 2020, reversed on reasoning models, per two vendors testifying against their own playbooks. Chain-of-thought helps math and symbolic work and little else, per an independent meta-analysis of over 100 papers. Multi-agent debate lost to a simple majority vote three times, per three independent replications. The pattern across all of it: what worked on 2020-era chat models is a coin flip on 2025-era reasoning models.

What should you do? Ask two questions before adopting any prompting technique. First, what is the actual study, and is it independent of the company selling the model? Second, what year, and what generation of model, did the evidence come from? A 2020 result tested on GPT-3 is a historical fact, and possibly a live mistake on the model you are running today. Whitney is next with the strangest case of that: the era when punctuation mattered more than words.

Sources

• Brown et al., "Language Models are Few-Shot Learners," NeurIPS 2020 [PR][Vendor: OpenAI]
• Microsoft finding that few-shot prompting hurt o1 (2024) [Vendor, against interest]
• DeepSeek-R1 technical report [Vendor, against interest]
• Sprague et al., "To CoT or Not to CoT?" ICLR 2025 (arXiv:2409.12183) [PR][Independent], https://arxiv.org/abs/2409.12183
• Wharton, Prompting Science report on chain-of-thought with reasoning models (2025) [Independent]
• Multi-agent debate vs compute-matched self-consistency: three independent replications (ICML 2024; ACL 2024; 2025) [Independent]
• Tran & Kiela (Stanford), "Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets," arXiv, April 2026 [arXiv][Independent], https://arxiv.org/html/2604.02460v1

Michael just promised you the strangest case in prompting: the era when punctuation mattered more than words. That era was real, and it is our last question. Does how you format a prompt matter more than what it says?

By formatting I mean the plumbing. The separators between fields. Spaces versus newlines. Whether the label says "Answer" in caps or lowercase. Changes a human reader would never notice carrying meaning.

Somebody measured it properly. Sclar and colleagues, peer reviewed at ICLR 2024, and independent, no vendor money on this one. They took prompts that meant exactly the same thing, varied only separators, spacing, and casing, and ran them across tasks on open models from the 2023 generation. The worst case: accuracy swung up to 76 points between two semantically identical prompts. Not seven point six. Seventy-six. The typical spread was around ten points, and remember from Michael's segment that chain-of-thought on a reasoning model buys you roughly three. The formatting noise ran bigger than the celebrated technique.

Here is my favorite detail, and it is the cruel one. The best format on one model did not transfer to the next model. You could tune the punctuation until one system scored beautifully, carry that exact format to a different model, and the gain vanished. There was no winning format to learn. There was only noise, and you had to measure it separately for every model you used.

So the formatting folklore had a real basis. On 2023-era open models, the plumbing genuinely could matter more than the wording.

Second study, and flag it before the numbers: He and colleagues, 2024, from Microsoft, a vendor with its own models in the test. They asked a cleaner version of the question. Take the same content and serve it as plain text, as Markdown, as JSON, as YAML. On GPT-3.5, the choice of wrapper shifted performance by up to 40 percent on some tasks. On GPT-4, the swings mostly flattened out. The bigger, newer model largely stopped caring which costume the prompt wore.

Then 2025 brought the correction. Peer-reviewed work at EMNLP 2025 found that part of the measured sensitivity was never the model at all. It was an evaluation artifact. Change the format and the automatic grader misreads answers that were correct, so some of the famous fragility lived in the ruler rather than in the model.

Watch the shape of that evidence, because it is the shape of this entire episode. A real, replicated effect on one model generation. A shrink on the next generation. Then a measurement critique trimming the number again. Which is why I will say the rule plainly: every prompting number needs a model-era stamp. Seventy-six points is a 2023 number about 2023 open models. The GPT-4 calm is a 2024 number from a vendor. Quote a prompting statistic without its year and its model generation and you have quoted nothing.

Now the episode verdict, all three questions.

Personas: strong evidence, negative. Three independent groups found no accuracy gain. Keep the role for tone, drop it for correctness.

Prompt tricks: real, with expiration dates. Few-shot examples helped the 2020 chat models and hurt the new reasoning models, per two vendors testifying against their own products. Chain-of-thought pays mainly on math. Multi-agent debate lost to a simple majority vote three times.

Formatting: real and shrinking. Enormous on 2023 open models, much smaller on frontier models, and partly a measurement artifact even then.

So what do you do with all of this? Three things, sized to the evidence. Test your prompt changes on your own tasks, because effects vary by model and by task, and a ten-point formatting swing can silently eat a careful A/B result. Date your prompt lore: write down which model generation every trick was proven on, and re-test when you switch models. And distrust undated advice. A prompting tip with no year and no model attached may have been true once, on a system nobody runs anymore. If the advice has no date, test it before you trust it.

Sources

• Sclar, Choi, Tsvetkov & Suhr, "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design," ICLR 2024 [PR][Independent], https://openreview.net/forum?id=RIu5lyNXjT
• He et al. (Microsoft), prompt-format study: plain text vs Markdown vs JSON vs YAML on GPT-3.5 and GPT-4, 2024 [Vendor: Microsoft]
• EMNLP 2025 work showing part of measured prompt-format sensitivity is evaluation artifact [PR]