RAG vs fine-tuning: when to use which

Short answer: If the model needs to answer from your documents, use . If it needs to answer in a particular format, tone or pattern, consider . And know that as of September 2026, an ordinary customer cannot fine-tune the newest models from OpenAI, Anthropic or Google’s Pro line.

Knowledge or behaviour

OpenAI’s own guidance splits the problem in two. If the model lacks knowledge, because it is new, out of date or internal, the fix is to give it the right . That makes answers more accurate. If the model answers inconsistently, in the wrong format or tone, the fix is to change the model. That makes behaviour more consistent.

Need	Use
Answers from documents that change	RAG
Small, stable knowledge base	Everything in context, with caching
Fixed format, tone or classification	Examples in the prompt first, fine-tuning if that fails
A narrow task on a cheaper model	Fine-tuning
Up-to-date facts	Never fine-tuning alone

Why fine-tuning is poor at teaching facts

The most cited comparison, Ovadia and colleagues in 2023, tested both methods on questions about news the models could not have known. RAG won every time. With Mistral-7B, accuracy went from 0.481 to 0.875 with RAG, but only to 0.504 with fine-tuning.

A study at EMNLP 2024 found something worse: when a model is fine-tuned on new knowledge, it learns it more slowly, and once it has learned it, its tendency to make things up increases. That is the opposite of what most people hope for.

Newer work gets fine-tuning to handle facts better with carefully built training data. Several of those papers are not yet peer reviewed.

The newest models cannot be fine-tuned

Status on 14 September 2026, from the vendors’ own pages:

Vendor	Fine-tuning
OpenAI	Being shut down. New customers locked out since May 2026, no new jobs from 6 January 2027. No GPT-5 or GPT-6 models.
Anthropic	Not in the Claude API. The only route was Claude 3 Haiku via AWS, which reached end of life on 10 September 2026.
Google	Not in the Gemini API. Available in Google Cloud for Flash and older Pro models, at 1.5 times the price for the tuned model.
Microsoft Azure	Available for GPT-4.1 and GPT-4o. Every fine-tuned deployment carries a fixed hourly hosting fee.

In practice, fine-tuning has become a tool for small and older models. If you build on the best models today, RAG and good context are what you have.

Long context has changed the maths

The newest models take 1 million tokens, and Anthropic prices the whole window the same. Meanwhile makes fixed content cheap to resend: at Anthropic a cached token for Claude Sonnet 5 costs $0.20 per million, against $2 normally.

If the knowledge base is small and rarely changes, putting everything in context can be simpler than building search. But research shows models get worse at finding information in the middle of a long context, and that long context gives better answers than RAG on average but costs more. There is no universal winner. The task decides, and it has to be measured.

Where RAG goes wrong

RAG is not free quality. A study of RAG systems in production found 7 typical failure points, from the answer not existing in the documents to the right document being found but not ranked high enough. Two deserve particular care:

Permissions. A search index that ignores who may see what can leak documents between users. OWASP lists this as a risk of its own.
Stale permissions. When rights change in the source system, search often does not reflect it until the index is synced again.

Both only surface if you test the system systematically. How is covered in the article on evaluating AI systems.

Sources

All sources checked on 14 September 2026.