RAG Chatbot Architecture: Retrieval Quality, Evals, and Cost Control

A RAG chatbot answers questions from your own documents and systems instead of from the model's general memory: retrieval finds the relevant passages or records, the model writes an answer grounded in them, and everything it says traces to a source. That architecture — retrieval augmented generation — separates a support bot that deflects tickets from one that invents your refund policy, and it is the pattern behind most of what our AI integration services ship.

What is a RAG chatbot, and why a plain LLM goes stale

A plain LLM chatbot answers from whatever the model absorbed during training, which does not include your docs or yesterday's policy change; without retrieval its knowledge goes stale, and retraining is the most expensive way to refresh it. RAG splits the job in two: a retrieval step searches your knowledge base — documents, help center, policies and, where needed, live records from your own systems — and the generation step writes an answer from that material only. The bot can therefore answer about things the model never saw, its knowledge updates when a document does, and a grounded answer can be audited — or refused with "I don't know" when retrieval finds nothing, the cheapest defence against hallucinations.

Retrieval augmented generation, explained for buyers

Retrieval augmented generation stands or falls on a blunt rule: the model cannot answer from a document it never saw. A wrong or empty answer is usually retrieval serving the wrong passages, or the right passage shredded during indexing — so the conversation belongs left of the model:

• Ingestion — which sources are in (docs, help center, policies, the operational data behind them) and which are out (stale wikis, contradictory drafts).
• Chunking — how documents are split into retrievable pieces. Too small loses context; too large gets vague; splitting by headings usually beats fixed-size chopping.
• Search — embeddings for meaning plus keyword matching for exact terms and error codes.

RAG chatbot architecture: the five pieces that matter

A production RAG chatbot architecture is five components, and a quality problem traces to exactly one of them at a time:

1. The index — your chunked, embedded knowledge base, rebuilt when source documents change.
2. The retriever — the search layer returning a shortlist of candidate passages.
3. The generator — the model plus a prompt contract: retrieved passages, the question and the rules (answer from the sources, cite them, refuse otherwise) in one prompt. That is RAG prompt engineering, and it is a versioned artifact, not a text box.
4. Guardrails — topic boundaries, sensitive-question handling, and the handoff to a human when confidence drops.
5. The logging loop — every question, retrieved set and answer recorded; unanswered questions are your content roadmap.

The pattern extends beyond support: we built CAIDR, an AI symptom checker, on the same grounded-assessment idea — user input matched against structured medical content with careful boundaries — because grounding plus guardrails is what makes an AI feature shippable where wrong answers hurt.

RAG chatbot examples: documents, live data, and your own team

The best RAG chatbot examples are unglamorous, because the bot sits next to data that already existed:

• Support answers from your documentation — release notes, help center and policies, with a citation and a handoff when retrieval finds nothing.
• Answers from live account data. A conventional bot is limited to a fixed intent list and recites router-restart steps to a customer whose internet is down; a RAG bot that may also retrieve the account record and the current outage status tells them about the known fault in their area and when service returns.
• Knowledge for your own people — agent assist for the support desk with cross-sell context, policy lookups for HR, runbook search for engineers.

Data readiness: the RAG architecture work nobody budgets

Most RAG architecture budgets go to the model and the chat widget; most RAG failures trace to data readiness. Three questions decide it. Where does the data live — static documents, or operational records that need an API, change data capture or a streaming feed to stay fresh? Who may see what — retrieval must enforce the same per-user permissions your application does, so one customer never retrieves another's record and personal data stays out of the prompt; privacy and data protection are architecture here, not a compliance chapter. How fresh is fresh — rebuild the index when documents change; fetch live records at question time. Tabular data (accounts, orders, plans) and unstructured documents need different retrieval paths; combining both is where the engineering time goes.

Chatbot evals and deflection: how an AI chatbot for customer support proves itself

Chatbot evals are the difference between "the demo looked good" and "we measured it." An eval set is a list of real questions from actual tickets, each with a known answer and source; every change (chunking, model, prompt) runs against it and yields three numbers: retrieval hit rate, answer quality (grounded, not just fluent), and refusal correctness.

For an AI chatbot for customer support, the business case is deflection: conversations resolved without a human, multiplied by what a human conversation costs. Two choices keep it honest — count a conversation as deflected only when the user stops asking, and keep escalation one click away; a bot that traps users converts support cost into churn.

How much does a RAG chatbot cost? Designing the meter

How much does a RAG chatbot cost? Two bills, both designed rather than discovered. The build covers ingestion and the index, the retriever and prompt contract, guardrails, the eval set, and the integration into your product's permission model — data readiness is usually the largest line, which is why the data questions come before any quote. The run cost has two meters: embedding on ingestion, which recurs only when content changes, and retrieval plus generation on every question.

Four levers hold the second meter down: cache, because support questions repeat; route easy questions to a small model; cap retrieved context, since fifteen passages where three suffice wastes tokens; and set per-tenant limits. The deliverable is a cost per 1,000 questions figure measured on realistic traffic before go-live; the reliability half — timeouts, retries, provider fallback — is ordinary hygiene from our API integration checklist.

RAG development services: how IvorySoft does it

Our RAG development services start with a fixed $4,900 discovery week: we pick the one capability, map where the data lives and who may see it, write the first eval set from real tickets, and price the build from that plan — credited toward the build if we proceed. The build ships behind a feature flag to an internal cohort first, with the eval set in CI and the cost meter wired before any customer sees it — the rollout discipline from adding AI features to an existing SaaS product. Open source RAG chatbot kits prototype well; the product work — permissions, evals, the meter — is what they leave to you, and when the engagement ends you own all of it.

RAG vs fine-tuning, agentic RAG and MCP: what comes next

RAG vs fine-tuning is the question under every "should we train our own model" conversation, and for knowledge that changes retrieval wins: fine-tuning bakes facts into weights that go stale the day a policy changes, while RAG reads the current document. Fine-tuning earns its place for tone, format or a specialised vocabulary — on top of retrieval, rarely instead of it; the decision rule is the one from AI integration vs AI development: start with retrieval, measure, escalate only when the numbers say so.

The next step is agentic: the bot that answers "what am I eligible for" today will file the application tomorrow, calling tools with the retrieved context in hand. The Model Context Protocol (MCP) is the emerging standard for exposing tools and data to a model in one consistent way. Every action needs what every answer needs: permissions, logs, an eval case, and human approval wherever money or account state moves.

The RAG readiness checklist

• Knowledge sources chosen deliberately; stale or contradictory content excluded
• Chunking follows document structure; search combines semantic and keyword matching
• The index rebuilds when documents change; live records are fetched, not copied
• Retrieval enforces the same per-user permissions as the application
• An eval set of real questions exists before launch and grows from logs
• Caching, routing and context caps designed; cost per 1,000 questions measured before go-live

FAQ

1. What is a RAG chatbot?

A RAG chatbot is a chatbot built on retrieval augmented generation: instead of answering from the model's training memory, it first retrieves relevant passages or records from your own knowledge base, then generates an answer grounded in them. That grounding lets it answer about your product and policies, cite sources, stay current by re-indexing rather than retraining, and refuse questions its sources don't cover. Quality depends mostly on retrieval, not on which model writes the final sentence.

2. What is RAG vs LLM?

An LLM is the language model itself: it writes fluent text from what it learned in training and knows nothing about your business. RAG is an architecture around that model — your documents are indexed, relevant passages are fetched per question, and the model answers from them or declines. RAG is the LLM plus retrieval, the way consumer assistants bolt web search onto a model; a RAG answer can be audited, updated by editing a document, and constrained to "no source, no answer".

3. How much does a RAG chatbot cost?

A RAG chatbot costs a build and a meter. The build covers ingestion and indexing, the retriever and prompt contract, guardrails, an eval set and the integration into your product's permissions — at IvorySoft it starts with a fixed $4,900 discovery week that maps the data, writes the first eval set and prices the rest. The running cost is embedding on ingestion plus retrieval and generation per question, held down by the four levers above; demand a cost per 1,000 questions figure before launch.

4. Is a RAG chatbot agentic AI?

A RAG chatbot is not agentic by itself: it retrieves and answers, nothing more. It becomes agentic when it is given tools — your APIs, a ticketing system — and permission to act on the retrieved context, such as opening a ticket or filing a form. That multiplies value and risk together, so every action needs per-user permissions, logging, an eval case per tool, and human approval wherever money or account state changes.

5. When should a chatbot hand off to a human?

A chatbot should hand off to a human the moment confidence drops or stakes rise: retrieval found nothing relevant, the user signals frustration, the topic touches money, legal exposure or account changes, or the same question loops twice. The handoff must carry the full conversation and the retrieved context so the user never repeats themselves — that detail decides whether the bot reads as helpful triage or as an obstacle.

Thinking about a support bot over your own docs? Book a 30-minute RAG scoping call — bring your help-center link, and we'll sketch the index, the eval set and the cost picture for your actual content. If your docs aren't ready for a bot yet, we'll say exactly that.