RAG Chatbot Architecture: Retrieval Quality, Evals, and Cost Control

A RAG chatbot answers questions from your own documents and systems instead of from the model's general memory: retrieval finds the relevant passages or records, the model writes an answer grounded in them, and everything it says traces to a source. That architecture — retrieval augmented generation — separates a support bot that deflects tickets from one that invents your refund policy, and it is the pattern behind most of what our AI integration services ship.

What is a RAG chatbot, and why a plain LLM goes stale

A plain LLM chatbot answers from whatever the model absorbed during training, which does not include your docs or yesterday's policy change; without retrieval its knowledge goes stale, and retraining is the most expensive way to refresh it. RAG splits the job in two: a retrieval step searches your knowledge base — documents, help center, policies and, where needed, live records from your own systems — and the generation step writes an answer from that material only. The bot can therefore answer about things the model never saw, its knowledge updates when a document does, and a grounded answer can be audited — or refused with "I don't know" when retrieval finds nothing, the cheapest defence against hallucinations.

RAG stands for retrieval-augmented generation. The term comes from a 2020 research paper by Patrick Lewis and colleagues, who paired a language model with a searchable index of Wikipedia so it could look facts up rather than rely on memory alone. In an AI model, then, RAG means relevant material is handed to the model with each question; the model itself does not change.

How a RAG chatbot works: one question, start to finish

Follow one question through the system. A customer on an annual plan asks: "Can I get a refund if I cancel now?" The example is invented; the sequence is what a production build runs:

1. Identify. The request arrives from a logged-in session, so the bot knows who is asking and on which plan.
2. Search. The retriever searches the documents this customer may see and returns a shortlist: the refund policy, the annual-plan terms, an old post about a past promotion.
3. Rerank. A second model reorders the shortlist for this exact question, and the old promotion drops out.
4. Generate. The top passages, the question and the rules go to the model in one prompt.
5. Answer or refuse. The customer gets a short answer citing the annual-plan terms; if the terms do not cover the case, the bot says so and offers a person.
6. Log. The question, passages and answer are stored, and that record supplies next month's eval cases.

The model only enters at step four; everything before it is retrieval.

Retrieval augmented generation, explained for buyers

Retrieval augmented generation stands or falls on a blunt rule: the model cannot answer from a document it never saw. A wrong or empty answer is usually retrieval serving the wrong passages, or the right passage shredded during indexing — so the conversation belongs left of the model:

• Ingestion — which sources are in (docs, help center, policies, the operational data behind them) and which are out (stale wikis, contradictory drafts).
• Chunking — how documents are split into retrievable pieces. Too small loses context; too large gets vague; splitting by headings usually beats fixed-size chopping.
• Search — embeddings for meaning plus keyword matching for exact terms and error codes.

RAG chatbot architecture: the five pieces that matter

A production RAG chatbot architecture is five components, and a quality problem traces to exactly one of them at a time:

1. The index — your chunked, embedded knowledge base, rebuilt when source documents change.
2. The retriever — the search layer returning a shortlist of candidate passages.
3. The generator — the model plus a prompt contract: retrieved passages, the question and the rules (answer from the sources, cite them, refuse otherwise) in one prompt. That is RAG prompt engineering, and it is a versioned artifact, not a text box.
4. Guardrails — topic boundaries, sensitive-question handling, and the handoff to a human when confidence drops: retrieval found nothing, the user signals frustration, the topic touches money or account changes, or the same question loops twice. The handoff carries the conversation and the retrieved context, so nobody repeats themselves.
5. The logging loop — every question, retrieved set and answer recorded; unanswered questions are your content roadmap.

The pattern extends beyond support: we built CAIDR, an AI symptom checker, on the same grounded-assessment idea — user input matched against structured medical content with careful boundaries — because grounding plus guardrails is what makes an AI feature shippable where wrong answers hurt.

Retrieval quality: where the bot wins or loses

Retrieval quality is the share of questions for which the right passage reaches the model. Three techniques to ask for by name:

• Chunks that keep their context. "The fee is waived after the first year" is useless if the passage does not say which plan. A short line of context added to each chunk before indexing restores what splitting removed.
• Hybrid search. Embeddings find meaning and miss exact identifiers; keyword matching does the reverse, so run both.
• Reranking. Retrieve a wide shortlist, then let a second model pick the few passages worth sending.

Anthropic published numbers for this stack. In its contextual retrieval tests, contextual embeddings plus contextual keyword search cut the top-20 retrieval failure rate by 49 percent (from 5.7 to 2.9 percent), and adding a reranker took the cut to 67 percent (1.9 percent). That is a vendor testing its own method, so read it as direction; your eval set gives you the real number.

When an answer is wrong, your team should be able to say which of these four broke:

• The document is missing or out of date. Fix the content; no engineering change helps.
• The document exists and was not retrieved. Look at chunking and search.
• It was retrieved and then ranked out. Look at the reranker and the context cap.
• It reached the model and the answer was still wrong. Look at the prompt contract. Only here is changing the model a fair conversation.

RAG chatbot examples: documents, live data, and your own team

The best RAG chatbot examples are unglamorous, because the bot sits next to data that already existed:

• Support answers from your documentation — release notes, help center and policies, with a citation and a handoff when retrieval finds nothing.
• Answers from live account data. A conventional bot is limited to a fixed intent list and recites router-restart steps to a customer whose internet is down; a RAG bot that may also retrieve the account record and the current outage status tells them about the known fault in their area and when service returns.
• Knowledge for your own people — agent assist for the support desk with cross-sell context, policy lookups for HR, runbook search for engineers.

Data readiness: the RAG architecture work nobody budgets

Most RAG architecture budgets go to the model and the chat widget; most RAG failures trace to data readiness. Three questions decide it. Where does the data live — static documents, or operational records that need an API, change data capture or a streaming feed to stay fresh? Who may see what — retrieval must enforce the same per-user permissions your application does, so one customer never retrieves another's record and personal data stays out of the prompt; privacy and data protection are architecture here, not a compliance chapter. How fresh is fresh — rebuild the index when documents change; fetch live records at question time. Tabular data (accounts, orders, plans) and unstructured documents need different retrieval paths; combining both is where the engineering time goes.

Chatbot evals and deflection: how an AI chatbot for customer support proves itself

Chatbot evals are the difference between "the demo looked good" and "we measured it." An eval set is a list of real questions from actual tickets, each with a known answer and source; every change (chunking, model, prompt) runs against it and yields three numbers: retrieval hit rate, answer quality (grounded, not just fluent), and refusal correctness.

For an AI chatbot for customer support, the business case is deflection: conversations resolved without a human, multiplied by what a human conversation costs. Two choices keep it honest — count a conversation as deflected only when the user stops asking, and keep escalation one click away; a bot that traps users converts support cost into churn.

How much does a RAG chatbot cost? Designing the meter

How much does a RAG chatbot cost? Two bills, both designed rather than discovered. The build covers ingestion and the index, the retriever and prompt contract, guardrails, the eval set, and the integration into your product's permission model — data readiness is usually the largest line, which is why the data questions come before any quote. The run cost has two meters: embedding on ingestion, which recurs only when content changes, and retrieval plus generation on every question.

Four levers hold the second meter down: cache, because support questions repeat; route easy questions to a small model; cap retrieved context, since fifteen passages where three suffice wastes tokens; and set per-tenant limits. The deliverable is a cost per 1,000 questions figure measured on realistic traffic before go-live; the reliability half — timeouts, retries, provider fallback — is ordinary hygiene from our API integration checklist. Neither half is theoretical: on Sourcd.ai, the OpenAI-powered outreach assistant we built, cost grew with usage, and provider outages and rate limits hit it.

When RAG is the wrong answer

When clients ask us to add AI, they usually ask for a chatbot, and what they usually need is search or answers over their own data. Frequently the real need is automation (extract, classify, route, draft) with the model as one step, and often the request started with a pitch deck or a competitor rather than a user problem.

We tell clients not to add an AI feature in two cases: when a rule, a filter or ordinary search would do, and when wrong answers are expensive, as with legal, medical or financial outputs. The second is a warning rather than a ban: a product in that territory, like CAIDR above, needs its boundaries before it needs a model.

Can I build my own RAG chatbot?

Yes, and if the goal is to learn, you should. Open source kits and hosted templates get a prototype running over a folder of documents, and a free kit stays free only until the model calls and hosting start. Tutorials stop short of the gap between a demo and a product: contradictory documents, per-user permissions, an eval set, re-indexing, a refusal path. A short way to decide:

• Build it yourself when the bot is internal, the documents are clean, no customer account data is involved, and someone on your team will own it after launch.
• Buy a platform when the questions are generic and your help center answers them; our support chatbot build vs buy guide covers that decision.
• Bring in a partner when answers depend on customer data, permissions matter, or a wrong answer costs money.

RAG development services: how IvorySoft does it

Our RAG development services start with a fixed $2,700 discovery week: we pick the one capability, map where the data lives and who may see it, write the first eval set from real tickets, and price the build from that plan — credited toward the build if we proceed. The build ships behind a feature flag to an internal cohort first, with the eval set in CI and the cost meter wired before any customer sees it — the rollout discipline from adding AI features to an existing SaaS product. Open source RAG chatbot kits prototype well; the product work — permissions, evals, the meter — is what they leave to you, and when the engagement ends you own all of it.

Are RAG chatbots legal? Liability, disclosure and prompt injection

They are, and three duties come with them. This is general information rather than legal advice; take the specifics to your counsel.

You answer for what the bot says. In Moffatt v. Air Canada, a British Columbia tribunal found in 2024 that the airline had not taken reasonable care to keep its website chatbot accurate after it misinformed a customer about bereavement fares, and ordered it to pay C$812.02 in total. It is one small-claims decision, but the tribunal called the idea that a chatbot answers for itself "a remarkable submission".

In the EU, people must know it is a bot. Article 50 of the AI Act requires providers to design systems that interact directly with people so those people are told they are dealing with an AI, unless that is obvious. In practice, label the chat widget from the first message.

Your documents are an attack surface. OWASP's prompt injection entry describes indirect injection, where instructions arrive inside external content such as websites or files, and notes that research shows RAG does not fully mitigate it. If the bot retrieves anything an outsider can write, such as tickets or uploaded files, treat that text as data and never as instructions, and keep every action behind your application's permission checks.

RAG vs fine-tuning, agentic RAG and MCP: what comes next

RAG vs fine-tuning is the question under every "should we train our own model" conversation, and for knowledge that changes retrieval wins: fine-tuning bakes facts into weights that go stale the day a policy changes, while RAG reads the current document. Fine-tuning earns its place for tone, format or a specialised vocabulary — on top of retrieval, rarely instead of it; the decision rule is the one from AI integration vs AI development: start with retrieval, measure, escalate only when the numbers say so.

The next step is agentic: the bot that answers "what am I eligible for" today will file the application tomorrow, calling tools with the retrieved context in hand. The Model Context Protocol (MCP) is the emerging standard for exposing tools and data to a model in one consistent way. Every action needs what every answer needs: permissions, logs, an eval case, and human approval wherever money or account state moves.

The RAG readiness checklist

• Knowledge sources chosen deliberately; stale or contradictory content excluded
• Chunking follows document structure; search combines semantic and keyword matching
• The index rebuilds when documents change; live records are fetched, not copied
• Retrieval enforces the same per-user permissions as the application
• An eval set of real questions exists before launch and grows from logs
• Caching, routing and context caps designed; cost per 1,000 questions measured before go-live
• Retrieved text treated as data, never as instructions; every action behind the application's permission checks
• The bot labelled as AI from the first message; wrong answers traced to one of the four failure points

FAQ

1. What is a RAG chatbot?

A RAG chatbot is a chatbot built on retrieval augmented generation: instead of answering from the model's training memory, it first retrieves relevant passages or records from your own knowledge base, then generates an answer grounded in them. That grounding lets it answer about your product and policies, cite sources, stay current by re-indexing rather than retraining, and refuse questions its sources don't cover. Quality depends mostly on retrieval, not on which model writes the final sentence.

2. What is RAG vs LLM?

An LLM is the language model itself: it writes fluent text from what it learned in training and knows nothing about your business. RAG is an architecture around that model — your documents are indexed, relevant passages are fetched per question, and the model answers from them or declines. RAG is the LLM plus retrieval, the way consumer assistants bolt web search onto a model; a RAG answer can be audited, updated by editing a document, and constrained to "no source, no answer".

3. How much does a RAG chatbot cost?

A RAG chatbot costs a build and a meter. The build covers ingestion and indexing, the retriever and prompt contract, guardrails, an eval set and the integration into your product's permissions — at IvorySoft it starts with a fixed $2,700 discovery week that maps the data, writes the first eval set and prices the rest. The running cost is embedding on ingestion plus retrieval and generation per question, held down by the four levers above; demand a cost per 1,000 questions figure before launch.

4. Is a RAG chatbot agentic AI?

A RAG chatbot is not agentic by itself: it retrieves and answers, nothing more. It becomes agentic when it is given tools — your APIs, a ticketing system — and permission to act on the retrieved context, such as opening a ticket or filing a form. That multiplies value and risk together, so every action needs per-user permissions, logging, an eval case per tool, and human approval wherever money or account state changes.

5. Can I build my own RAG chatbot?

Yes. Open source kits and templates make a prototype over your own documents a manageable first project, and for an internal tool with clean documents and no customer data that may be enough. The gap shows up in production: per-user permissions, an eval set, re-indexing, a refusal path and a cost meter. If answers depend on customer data, or a wrong answer costs money, plan for that product work rather than treating the prototype as the finished bot.

Thinking about a support bot over your own docs? Book a 30-minute RAG scoping call — bring your help-center link, and we'll sketch the index, the eval set and the cost picture for your actual content. If your docs aren't ready for a bot yet, we'll say exactly that.