Qatreh AI Logoقطره
Back to blog
Conversational AI

Chatbot Architecture: Language Model or Retrieval?

Retrieval costs nothing per answer and invents nothing; a model adds flexibility and the risk of a confident wrong answer. How to choose, and where hybrids fit.

تیم قطرهSeptember 5, 20266 min read
Chatbot Architecture: Language Model or Retrieval?

The question organisations ask is "should our chatbot use a large language model?" The answer does not depend on company size. It depends on how varied the incoming questions are and how expensive a wrong answer is for you.

The short answer

There are two architectures. In retrieval, the system maps a question to one of the organisation's own pre-written answers; it never composes a new sentence. In a generative design, the answer is produced on the spot. Retrieval has no per-answer cost, sends no data outside, and never invents anything — at the price of knowing only what you have written. The choice is a trade between control and flexibility.

Two architectures, two kinds of risk

Retrieval over your own knowledge. The question is folded into a comparable form, matched against the organisation's topics and documents, and the closest pre-written answer is returned. Its characteristic risk is not arriving: a question outside the written domain gets the default reply.

Generation with a language model. The question goes to a model and an answer is composed. Its characteristic risk is arriving at a confident wrong answer. A model says "I don't know" less often than it produces something that reads as correct.

These two risks are not equivalent. The first leaves a user without an answer; the second gives them incorrect information they then act on. For questions about payment terms, delivery commitments or technical specifications, the second is far more expensive.

Cost: what actually consumes budget

With retrieval, the marginal cost per answer is effectively zero; the cost is paid once, in writing the knowledge. With a model, every answer carries a compute cost that scales with traffic.

The less obvious point is that model cost also scales with abuse. If the answering endpoint has no daily ceiling, a simple script — with no malicious intent at all — can consume a month's budget overnight. If you choose this architecture, three things are required before launch: a per-address rate limit, a global daily cap, and cost monitoring.

Data that leaves the organisation

If the model runs on an external service, the user's question text — which may contain an order number, a name or a phone number — leaves your organisation.

For many businesses that is acceptable. For organisations handling financial, health or identity data it is a legal decision rather than a technical one, and it belongs before the architecture is chosen, not after.

Retrieval does not raise the question, because processing stays entirely on your own infrastructure.

Dependency on an external service

An architecture built on an external model means that when that service is unavailable, your answering system is too. We looked at this separately in AI service outages and business continuity; the short version is that every external dependency needs a fallback path.

Our own answering system is retrieval-based and connects to no external model. The measured result: 10 to 14 milliseconds of server-side processing, no per-answer cost, and no surface for prompt injection — because there is no model to inject into.

Which architecture suits which business

Retrieval fits when:

  • Incoming questions repeat: opening hours, pricing, address, delivery terms, product specifications
  • The answer must be exact and identical every time
  • Data must not leave the organisation
  • Monthly operating cost must be predictable

A language model fits when:

  • Questions are open-ended and unpredictable
  • Users expect multi-turn conversation and summarisation
  • The reference material is too large for manual mapping to be practical
  • The organisation has the capacity to monitor answer quality

A practical test: count your own real messages from the past six months. If more than seventy percent fall into ten repeating categories, retrieval is sufficient, and it will be simpler and cheaper.

The hybrid

In practice the best result is usually a combination: retrieval for known questions, and a model consulted only when no topic matched.

That structure has three benefits. Most traffic is handled at zero marginal cost with an exact answer; model cost is paid only for genuinely open questions; and if the external service is unavailable the system falls back to retrieval instead of failing.

Whichever path you take, evaluating answer quality is not optional — the method is in how to actually test a Persian chatbot, and the language-level issues common to both architectures are in why your Persian chatbot answers the wrong question.

Deployment in Karaj and Alborz

For businesses in Karaj and Alborz province one practical factor joins the decision: paying for external services and relying on uninterrupted access to them is not always straightforward. It is not the only criterion, but it is a real one, and it belongs on the table before the design is fixed rather than after.

Our team runs this session in person, at Alborz Science and Technology Park or on the organisation's own site, because the right decision depends on understanding the character of your customers' real questions — and that understanding sits with your support team.

Frequently asked questions

Which architecture is cheaper?

Retrieval, in operation. It has no per-answer cost and negligible compute load. In exchange it carries the up-front cost of writing and structuring the knowledge. A model-based system starts faster and its cost grows with traffic.

Does retrieval mean a simple, weak chatbot?

No. Retrieval quality depends on text folding, knowledge structure and scoring logic, not on apparent simplicity. A system that correctly understands Persian questions across spelling variants and routes to the right page is more useful than a large model without precise retrieval behind it.

How do we prevent a model from answering incorrectly?

By constraining scope: the model may compose an answer only from retrieved documents, and must say plainly that it does not know when none were found. That is the hybrid pattern, and implementing it costs less than one wrong answer to a customer.

Where does our customers' data go?

With retrieval on your own infrastructure, nowhere. With an external model, the question text is sent to the provider. Read that provider's data-retention terms before choosing.

How long does each take to deploy?

It depends on scope and no figure is meaningful before the problem is defined. Our approach is to start with a prototype in a narrow domain, so real effect is measured before any larger commitment.

Can the architecture be changed later?

Yes, if the organisation's knowledge is stored separately from the answering logic from the start. Organisations that write answers inside their code effectively rebuild everything when they switch.

Related services from Qatreh