Fine-tune, RAG, or just prompt?

Can you teach a large language model new facts by fine-tuning it? Vendors sell fine-tuning as the way to make a model yours, and pouring your company's documents into its weights sounds like exactly that. So the question for you today: what does the science say happens when you try?

Start with the team that asked most directly. Gekhman and colleagues, from the Technion and Google Research, peer reviewed at EMNLP 2024. This one comes from Google, so flag it. Notice, though, that the result cuts against the sales pitch, which makes it more credible: the funder gains nothing from this finding. They ran a controlled closed-book question-answering study and varied how many of the fine-tuning examples carried knowledge the model genuinely did not have. Two findings. First, the new-knowledge examples were learned much more slowly than facts the model already knew. Second, as those examples were eventually learned, the model's tendency to hallucinate increased linearly. Sit with that for a second. The more new facts you push into the weights, the more the model makes things up. Their conclusion: models acquire their facts mostly in pretraining, and fine-tuning teaches them to use what they already know more efficiently.

Second study. Ovadia and colleagues, also peer reviewed at EMNLP 2024, and this one comes from Microsoft, so flag that too. They compared fine-tuning against retrieval-augmented generation, RAG, where you leave the weights alone and hand the model the relevant documents at question time. Across knowledge-intensive tasks, MMLU subsets plus a current-events set built to be genuinely new, RAG consistently won. That held for knowledge the model had seen in pretraining and for knowledge it had never seen. On new facts, RAG roughly doubled fine-tuning's accuracy. Their wording: large language models struggle to learn new factual information through unsupervised fine-tuning.

Third, the replication. A peer-reviewed study at SIGIR-AP 2024, no vendor flag on this one in our notes, tested twelve models of varying sizes and found the gap is widest exactly where you would most want fine-tuning to help: the rarest, least popular facts. RAG surpassed fine-tuning by a large margin there.

Then somebody measured the products you can actually buy. FineTuneBench, from a Stanford team. Independent, no vendor money, and pointed in the right direction: outsiders grading the vendors' homework instead of vendors grading their own. It is an arXiv preprint from November 2024, so no peer review yet. They tested five commercial fine-tuning APIs, GPT-4o, GPT-4o mini, GPT-3.5 Turbo, Gemini 1.5 Pro, and Gemini 1.5 Flash, on learning new information and on updating information the models already had. Average accuracy on new knowledge: 37 percent. On updating existing knowledge: 19 percent. The Gemini APIs managed neither at all.

One precision point before I grade this, because it changes what you take home. Most of these results test unsupervised fine-tuning, meaning continued pretraining on raw documents. Supervised fine-tuning on question-and-answer pairs, plus data tricks like paraphrasing each fact many ways, can narrow the gap, and Ovadia's team says so explicitly. So "fine-tuning cannot learn facts" is too strong, and I will say the version the data supports: fine-tuning is a slow, hallucination-prone way to teach a model facts, and retrieval beats it.

The grade, then. For the knowledge question, the evidence is strong. Two peer-reviewed studies funded by different vendors, a peer-reviewed replication across twelve models, and an independent Stanford benchmark of the commercial products all land in the same place. Can you teach a model new facts by fine-tuning it? Barely, and the model lies more as it learns.

So here is what to do. When your model needs to know something new, put it in the prompt or a retrieval layer, and keep fine-tuning for a different job entirely. What job? Michael has that answer next.

Sources

• Gekhman et al., "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?", Technion and Google Research, EMNLP 2024 [PR][Vendor-funded: Google]. https://arxiv.org/abs/2405.05904
• Ovadia, Brief, Mishaeli, Elisha, "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", Microsoft, EMNLP 2024 [PR][Vendor-funded: Microsoft]. https://arxiv.org/abs/2312.05934
• Soudani, Kanoulas, Hasibi, "Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge", SIGIR-AP 2024 [PR]. https://arxiv.org/abs/2403.01432
• Wu, Wu, Zou (Stanford), "FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?", arXiv preprint, November 2024 [Independent]. https://arxiv.org/abs/2411.05059

Whitney just showed you that fine-tuning is a bad way to teach a model facts. So what is fine-tuning actually for? Companies keep paying for it, and the same literature that buries the knowledge use case points straight at the real one.

Start with the mechanism. Go back to Gekhman and colleagues, the Technion and Google Research team, peer reviewed at EMNLP 2024, and from Google, so flag it. Inside their hallucination result sits the positive finding: fine-tuning teaches a model to use knowledge it already has more efficiently. The weights already hold the facts. Fine-tuning changes how the model deploys them. And that mechanism predicts exactly what fine-tuning should be good at: form. Output format, style, tone, and following instructions.

Who says that lane actually works? My first witness is a compromised one, and I want you to hear the flag before the content. OpenAI's own supervised fine-tuning documentation. This is vendor documentation from a company that sells both fine-tuning and retrieval tooling, so it is a vendor telling you when to buy which of its products. Flag it hard. Their listed use cases: classification, nuanced translation, generating content in a specific format, and correcting instruction-following failures. Every one of those is about shaping how the model responds. Zero of them are about teaching it new facts. Vendor docs can never carry a claim alone. Here they do not have to, because they line up with the mechanism from the peer-reviewed study.

Second witness: the economics at scale. A peer-reviewed NeurIPS 2022 study by Liu and colleagues, the T-Few work, no vendor flag on this one in our notes. Parameter-efficient fine-tuning beat in-context learning by about 6 percent accuracy on few-shot tasks while using roughly a thousand times fewer inference FLOPs. The reason is structural. Few-shot examples ride along in the prompt, so you pay to process them on every single call. A fine-tuned behavior is baked into the weights and paid for once. When you have the labeled data and real call volume, the tuned model is both more accurate and enormously cheaper per call.

Third witness, and this is the cost that never makes the pitch deck: forgetting. Biderman and colleagues, "LoRA Learns Less and Forgets Less," peer reviewed in Transactions on Machine Learning Research, no vendor flag in our notes. Full fine-tuning learned the target domain best and forgot the most general capability. LoRA, the parameter-efficient method, learned somewhat less and preserved far more, with the trade-off controlled by rank. One number to hold onto: after continued pretraining on 20 billion tokens, the fully fine-tuned model scored 0.545 on a held-out general benchmark, against 0.617 for LoRA at rank 256. Tuning for your task can quietly break the model everywhere else, and you only find out if you test for it.

Now the honest counter-case, because the precise claim matters here. A UC Irvine team, arXiv preprint from January 2026, single group, no peer review yet, no vendor flag. They tested multi-hop question answering across three 7-billion-parameter models, on QASC plus more than ten thousand questions built from 2024 Wikipedia events, meaning knowledge from after pretraining. RAG beat unsupervised fine-tuning on the novel questions, consistent with everything Whitney covered. But supervised fine-tuning, training on task-format question-and-answer pairs, achieved the highest overall accuracy across models and datasets. So here is the precise claim: on some 2026 multi-hop setups, supervised task-format fine-tuning has beaten RAG overall. One team, unreplicated, so treat it as corroborating rather than foundational. The narrow takeaway: teaching the task format can matter as much as where the facts live.

The grade. The mechanism, that fine-tuning shapes how a model uses what it already knows, is strong, and it comes from the same peer-reviewed work that killed the knowledge use case. The specific claim that behavior, format, and instruction-following are fine-tuning's lane grades moderate, because part of that support is vendor documentation.

So use fine-tuning to shape behavior: a strict format, a house style, an instruction-following habit you need thousands of times a day. Prefer LoRA over full fine-tuning to protect the base model, and keep an off-domain eval running so you catch what you broke. Whitney puts the whole decision tree together next.

Sources

• Gekhman et al., "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?", Technion and Google Research, EMNLP 2024 [PR][Vendor-funded: Google]. https://arxiv.org/abs/2405.05904
• Supervised fine-tuning, OpenAI API documentation, accessed June 2026 [Vendor docs]. https://developers.openai.com/api/docs/guides/supervised-fine-tuning
• Liu et al., "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning", NeurIPS 2022 [PR]. https://dl.acm.org/doi/10.5555/3600270.3600412
• Biderman et al., "LoRA Learns Less and Forgets Less", Transactions on Machine Learning Research [PR]. https://arxiv.org/abs/2405.09673
• Yang, Song, Ahmed, Harris (UC Irvine), "Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge", arXiv preprint, January 2026 [Independent, not yet peer reviewed]. https://arxiv.org/abs/2601.07054

Prompt, retrieve, or tune? You have heard the evidence for two videos now, and the question you actually face on Monday is which one to reach for. The literature supports a decision tree with three forks: prompt first, retrieve for knowledge, tune for behavior. I will walk you through one real system at each fork.

Fork one: prompt first. Say you need a one-off analysis. The model reads a pile of customer interviews and pulls out the themes, once, this week. Put the documents in the prompt, give it a few examples of the output you want, and stop there. The economics for this fork come from Liu and colleagues, the parameter-efficient fine-tuning study peer reviewed at NeurIPS 2022, no vendor flag in our notes. Michael cited it for fine-tuning's win, and the same numbers mark fine-tuning's boundary: the tuned model only pays off once you have labeled training data and enough call volume to amortize the training cost, because what you save is the price of reprocessing prompt examples on every call. A job you run once has no volume to amortize against. Prompting costs nothing up front and iterates in minutes. And Biderman's forgetting result, peer reviewed in Transactions on Machine Learning Research, adds the risk side: full fine-tuning can quietly degrade the base model's general ability. You would be accepting that risk for a task that ends on Friday.

Fork two: retrieve for knowledge. Now a support bot that must answer from your current product catalog. Prices, version numbers, the feature that shipped last Tuesday. That is new and changing factual knowledge, and this fork holds the strongest science in the episode. Gekhman and colleagues, Technion and Google Research, peer reviewed at EMNLP 2024, Google money so flag it: new facts pushed into the weights were learned slowly, and as they were learned, hallucination rose linearly. Ovadia and colleagues at Microsoft, also peer reviewed at EMNLP 2024, flag that one too: on genuinely new facts, RAG roughly doubled fine-tuning's accuracy. And the independent Stanford benchmark, FineTuneBench, measured the commercial fine-tuning APIs at 37 percent average accuracy on new knowledge. Your catalog changes weekly. A retrieval index updates when your documents update; weights sit frozen until you retrain. The support bot gets retrieval.

Fork three: tune for behavior. A report generator that must produce your house format at scale. Same structure, same section order, same tone, thousands of times a month. No new facts involved, pure form at volume. Michael gave you the mechanism from Gekhman's own data: fine-tuning teaches a model to use what it already knows more efficiently, which is exactly a format job. The Liu economics now point the other way, because at thousands of calls a month the behavior baked into the weights is paid for once, while format instructions and examples in the prompt are paid for on every single call. Use LoRA rather than full fine-tuning, per Biderman, and keep an off-domain eval running so you catch any general ability you broke.

One precision point before the verdict, because the tree has a live counter-case. The UC Irvine preprint from January 2026, single team, independent, not yet peer reviewed: on multi-hop question answering, supervised fine-tuning on task-format question-and-answer pairs achieved the highest overall accuracy, beating RAG, while RAG still beat unsupervised fine-tuning on the novel facts. Read it carefully and it fits the tree: what the supervised training taught was the shape of the task, which is behavior, our third fork. It is one unreplicated result, so it bends the tree rather than breaking it.

The episode verdict. The knowledge fork is strong: two peer-reviewed studies from rival vendors, a peer-reviewed replication, and an independent benchmark all say facts belong in retrieval, and weights lie more as you stuff facts into them. The behavior fork grades strong on mechanism and moderate on the lane claim, since part of that support is vendor documentation. The ordering, prompt before either, rests on peer-reviewed economics.

So here is Monday. Start every new problem in the prompt. When the model does not know something, wire in retrieval before you consider training. When it knows enough and will not behave, fine-tune with LoRA and hold out an off-domain eval. Revisit the tree when the supervised task-format result gets a replication.

Sources

• Gekhman et al., "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?", Technion and Google Research, EMNLP 2024 [PR][Vendor-funded: Google]. https://arxiv.org/abs/2405.05904
• Ovadia, Brief, Mishaeli, Elisha, "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", Microsoft, EMNLP 2024 [PR][Vendor-funded: Microsoft]. https://arxiv.org/abs/2312.05934
• Soudani, Kanoulas, Hasibi, "Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge", SIGIR-AP 2024 [PR]. https://arxiv.org/abs/2403.01432
• Wu, Wu, Zou (Stanford), "FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?", arXiv preprint, November 2024 [Independent]. https://arxiv.org/abs/2411.05059
• Liu et al., "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning", NeurIPS 2022 [PR]. https://dl.acm.org/doi/10.5555/3600270.3600412
• Biderman et al., "LoRA Learns Less and Forgets Less", Transactions on Machine Learning Research [PR]. https://arxiv.org/abs/2405.09673
• Yang, Song, Ahmed, Harris (UC Irvine), "Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge", arXiv preprint, January 2026 [Independent, not yet peer reviewed]. https://arxiv.org/abs/2601.07054