RAG chatbot accuracy: how data preparation beat model size

We built a retrieval-augmented generation (RAG) chatbot for the International Team for Implantology (ITI), a network of 25,000+ implant dentistry specialists. The client's brief was blunt: safe answers come first, ahead of everything. ITI spent decades earning the trust its members bring to it, and one wrong answer can burn that trust in seconds.

Off-the-shelf tools had already failed that test in their own pre-project evaluation, which is why they wanted a custom build. We offered something better: an approach that pairs the speed of SaaS with the flexibility of custom.

The answer was not a bigger model. We reached that accuracy with a small one, by spending the engineering effort on data preparation instead of model size. Here is how we did it.

Bigger model or better data? What a RAG chatbot actually rewards

In a RAG chatbot, the answers do not live in the model. They live in retrieval. The model reads whatever the pipeline hands it and rewrites that into a reply, which means a bigger model buys you more fluent rewriting, not better sourcing. And when retrieval surfaces the wrong passage, or nothing useful, a larger model does not save you. It delivers the same wrong answer, only more convincingly.

That reframes the problem. In a system grounded in retrieval, most wrong answers are a data and retrieval problem, not a model-capability problem. The lever is the pipeline that feeds the model, not the size of the model itself.

ITI had already tested the model-first path. In a 2025 proof of concept, the team fine-tuned Llama 3.2 (LoRA, 4-bit) on ITI content and wired it to a simple in-memory FAISS retrieval pipeline. It beat the untrained baseline, which proved the idea was sound. The weak point it exposed was not the model. The team logged it plainly at the time: an unoptimized RAG pipeline underneath a capable model.

The question was where to spend the effort instead.

What the medical benchmarks say about RAG

A study in Nature Medicine, published June 2026, tested two specialized clinical AI tools, OpenEvidence and UpToDate Expert AI, against three general-purpose models: GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6. The specialized tools were the ones built for medicine. They did not come out ahead.

On a 500-question medical exam, the two specialized tools scored 89.6% and 88.4%, behind two of the three general models, with the third just above. The same gap held on the benchmark the authors trusted most: a blinded review of real physician queries, where all three general models ranked in a higher tier. The tools built for medicine lost to the general ones.

The likely reason is retrieval. The architectures are not public, but both specialized tools probably use RAG, and RAG can drag performance down when it pulls the wrong material or the model integrates it poorly. Adding a knowledge base to a model does not guarantee better answers. Done badly, it makes them worse.

That is the trap we had to avoid. Bolting retrieval onto a model is not the same as engineering it. The ITI chatbot result came from treating retrieval and data preparation as the actual product, which is the next section.

What we built: a healthcare chatbot grounded in evidence

ITI AI Concierge is an educational chatbot for ITI's 25,000+ members: implant dentistry specialists, academics, and students. They ask clinical and scientific questions in plain language, and it answers from ITI's own evidence base, with sources attached. The goal was narrow: make a vast, scattered body of content reachable through one channel, without ever inventing an answer.

For the first version we scoped tightly. English only, text only, grounded in two content areas, clinical tools and publications, across roughly 140 documents unified into a single retrievable library. The approach we brought was Chatguru, Netguru's open-source RAG platform which sits between SaaS and full custom. It gives you ready building blocks to start from, which shortens delivery the way SaaS does, but because it is open source you can adapt it as freely as a custom build. It also self-hosts, so member queries and ITI content never leave controlled infrastructure. From there the setup is standard RAG: documents are chunked, embedded, and stored in a vector database, currently SQLite, and retrieval pulls the relevant passages for the model to write from.

The model is deliberately small: GPT-5 mini, running with low reasoning effort. That choice only works because of what happens before the model ever sees the question.

Why a small LLM held up in production

ITI AI Concierge runs on GPT-5 mini with low reasoning effort. When retrieval hands over the right passages, the model is not reasoning toward an answer, it is rewriting prepared content into clear language. A small model does that well, and a bigger one would add cost and latency for fluency the task does not need.

Scope discipline backed the same choice. The team built a second, graph-based retrieval path (LightRAG) for harder, multi-step questions, then kept it in reserve. In practice the questions users asked did not need it, and the simpler setup stayed faster and cheaper.

The economics follow from the architecture. Smaller model, lower cost per answer, and latency around two seconds to first token and ten seconds to a full answer. Clean the input first, and a small model is often enough.

Where the chatbot accuracy actually comes from - data preparation

Four things carried the accuracy, and none of them is the model: how we cleaned the sources, how we chunked them, what retrieval had to return, and what the prompt forbade.

Cleaning the sources

The library held about 140 documents of two kinds: treatment guides and consensus papers. They did not look the same to a retrieval system. Treatment guides carried more measurements, symbols, and non-English characters. Consensus papers were closer to clean prose. One cleaning pass would not fit both.

In a clinical corpus, extraction noise lands on the tokens that carry meaning. A hyphenated line break splits peri-implantitis into two fragments. A dropped operator turns "5mm or more" into a different clinical claim. Abbreviations carry the same weight. RC means Regular CrossFit, PES the Pink Esthetic Score, WES the White Esthetic Score. Strip them or garble them, and a specialist loses the thread. So normalization was accuracy work, not cleanup.

The pipeline repaired hyphenated breaks, standardized measurement formats, aligned clinical-term variants, and kept the comparison operators a generic cleaner would strip.

The goal was narrow: make sure the tokens a specialist relies on reach the vector store intact.

Chunking, decided by measurement

Chunk size is a tradeoff. Cut too small and a complete answer splits across two chunks, so retrieval can never return it whole. Cut too large and you dilute retrieval and pay for it in cost and latency.

We took 177 passages, each a full answer to a specific clinical question, and tested how different chunk sizes treated them. At 256 tokens with no overlap, about 37% of those answers split across a boundary. The longest ones survived whole only 19% of the time. Keeping the long answers intact around 80% of the time took roughly 1,000 to 1,250 tokens. So we sized chunks to fit whole answers, not to hit a round number.

What reaches the model, and what it may do with it

Retrieval decides what the model sees. The prompt decides what it may do with it. Three constraints came from the client and were non-negotiable. Answer only from the supplied documents, and say so when the answer is not there. Stay brand-neutral, never naming a manufacturer. Surface what the documents say without turning it into instructions for treating a patient. We enforced these behaviors in the prompt and tested each one on its own.

How we measured near-zero AI hallucinations

Accuracy here is a composite. An answer counts only if it is correct, relevant, and complete. On that basis, ITI AI Concierge reached 98%.

Getting to a number we could trust meant building the measurement first. Before production code, the team defined a golden dataset: questions paired with an expected answer and the exact source document the system should retrieve. That last field lets us test retrieval, not just the wording of the answer.

Evaluation runs on Promptfoo, with an LLM-as-judge scoring each response on three metrics. Context Faithfulness checks whether the answer is fully supported by the retrieved sources, our closest automated proxy for "not hallucinating." Answer Relevance checks whether it addressed the question. Hit-based retrieval metrics check whether the system pulled the authoritative document. Every run feeds into Langfuse, so we can trace any regression to the change that caused it.

The 98% is the automated figure. Whether specialists trusted the answers is a separate question, and we tested it two ways. First, 24 specialists worked through real clinical queries in 1:1 sessions across two rounds. Then a survey of 31 members rated the tool on accuracy, trust, and usefulness, and returned a Net Promoter Score of 81 with no detractors.

Both pointed the same way, and the reason was trust. Answers were grounded in official ITI material with genuine citations, not the fabricated references specialists expect from general chatbots. Of the 24, only one did not see a clear difference from ChatGPT or Claude, an outlier with a heavily AI-integrated workflow.

Testers were not uncritical. They flagged answers that ran long or stayed too generic for a specific patient, and one caught a citation to a superseded guide, since removed. These were retrieval and phrasing issues, not fabricated facts, and each was traceable to a real source we could fix.

This is how the answers read to specialists, not a measurement across the full pilot. The 98% is the measured figure, and its underlying values sit with our QA team.

What building a RAG chatbot this way proves

The lesson from ITI AI Concierge is narrow on purpose. When answers have to be grounded in a fixed body of evidence, accuracy is decided before the model runs. Clean the sources, chunk them well, retrieve the right passage, and constrain the prompt, and the generation step becomes easy enough for a small model to handle.

That order matters because it is where the effort went. Not into a larger model, but into unifying scattered documents, testing retrieval against a golden dataset, and refusing anything outside the domain. The model size question came last, and the answer was "small is enough."

It also depended on scope. A first version limited to two content areas and one language is what made the evidence base small enough to prepare properly. A broader launch would have meant more sources, more retrieval edge cases, and a harder accuracy problem from day one. Starting narrow was not a compromise. It was how the quality got built.

None of this closes the door on bigger models. On open-ended reasoning they earn their cost, and with the data groundwork already in place, they become an opportunity rather than a crutch: room to scale into more scope, more languages, and harder questions. The same groundwork makes new features possible: richer interaction, more sources, and finer control over how answers are shaped. For a chatbot answering from a trusted library, the ITI build shows the order that works: the data is the product, and the model is the part you can scale up later.

FAQ

Does RAG stop a chatbot from hallucinating?

It does not eliminate the risk, but done well it reduces it significantly. RAG grounds answers in retrieved sources, yet if retrieval pulls the wrong passage or the model integrates it poorly, a RAG chatbot can still hallucinate. What drives the risk down is the data preparation underneath: clean sources, sound chunking, and retrieval that returns the right passage.

Do you need a large model to build an accurate RAG chatbot?
No. When retrieval hands the model the right content, generation is close to rewriting, and a small model does that well. ITI AI Concierge runs on GPT-5 mini with low reasoning effort and reached 98% accuracy. Model size matters most when the input is messy. Prepare the data first, and a small model is often enough.

Why build a custom RAG chatbot instead of using ChatGPT?

Because a general chatbot cannot guarantee its sources. In a field like implant dentistry, a confident but fabricated reference is a safety risk. ITI needed answers traceable to its own vetted evidence, brand-neutral, and refused when outside scope. In testing, 23 of 24 specialists saw a clear difference from general tools, and the reason was the citations they could trust.