Qatreh AI Logoقطره
Back to blog
Conversational AI

How to Actually Test a Persian Chatbot

A plausible answer proves nothing. A fixed question set, an assertion on the source of each answer, and three silent failures only real testing ever reveals.

تیم قطرهSeptember 5, 20267 min read
How to Actually Test a Persian Chatbot

Most teams accept a chatbot the same way: they ask it a handful of questions, the answers look reasonable, and the service is signed off. That procedure proves almost nothing.

The short answer

Evaluating a Persian-language assistant properly needs three things: a fixed set of questions replayed identically every time, an explicit assertion for each one about which source the answer must come from, and response time measured separately from network time. Without all three, any change to the system can break something nobody notices.

"The answer looked fine" is not a criterion

A retrieval or generation system almost always returns something. If it does not recognise the topic it returns a default; if it recognises the wrong topic it returns an answer that is correct in itself and irrelevant to the question.

The second failure is the more deceptive one, because it reads well. In our own evaluation, the question "do you offer enterprise training?" was answered with the project-timeline text — entirely accurate, entirely unrelated. Anyone reading only the answer sees nothing wrong.

The fix is for each test case to assert not "the answer was good" but that the answer contains a specific marker: the path of the page it should point to, a distinctive phrase from the reference text, or the contact number. A fluent irrelevant answer then fails.

Build a question set, not a few scattered checks

What worked in practice was a list where every entry carries four fields: the question text, the language, a string that must appear in the answer, and a label saying what the case is testing.

Ours grew to 47 cases and covers:

  • Ordinary questions for each service, in Persian, English and Arabic
  • The same question in two spellings — with a space and with a zero-width non-joiner — which must both reach the same destination. The mechanics are in why your Persian chatbot answers the wrong question
  • Every question that previously produced a wrong answer. A fixed defect without a test case comes back
  • One nonsense input, to confirm the default reply actually works
  • A negative assertion: something the organisation does not offer must never be promised

Do not skip that last one. If you do not provide a service, the system must say so plainly. A customer who calls on the strength of an incorrect promise costs you both time and credibility.

Three silent failures only testing reveals

1. Rate limiting that returns empty responses. Our middleware allows 20 requests per minute per address. Sending 42 questions back to back meant everything from the twenty-first onward came back empty — which looked exactly like a broken service, while the service was healthy and the test was wrong. A question set must be paced and must report the status code, not only the answer body.

2. An interface that never opens. It is possible for the interface to fail to open for every user while the API answers perfectly. No API test can see this; only loading the site in a real browser and clicking the button. Server logs answer it too: if no request from a user's browser has reached the conversation endpoint in days, the interface is not working.

3. Stale knowledge. If the knowledge base is read only at service start-up, anything added today stays invisible until the next restart. The test case is simple: publish something new and immediately ask about it.

Separate server time from network time

Measured from your own machine, response time is the sum of network and server time. Knowing which to improve takes two measurements:

  • A request for a static file on the same domain, giving the network floor
  • The same answering request issued from inside the server against its local address

In our measurements, total time from outside was around 400 milliseconds and the same request on the server itself took 10 to 14. Almost all of it was network, and further code optimisation would have changed nothing for the user. Without the second measurement, days could have gone into optimising the wrong thing.

Keep a log of unanswered questions

Every question the system could not answer should be appended to a file. It is the only reliable list of your knowledge gaps, because a user who gets no answer rarely complains — they simply leave.

Practical advice: store only the question text, a timestamp and the language; cap the file size; review it monthly. Questions that recur are next month's work list.

Running this for an organisation in Karaj

For an organisation in Karaj or elsewhere in Alborz province deploying Persian-language support, we recommend building the question set from the organisation's own real messages rather than from what the technical team imagines users ask. Six months of WhatsApp messages, direct messages and repeat phone enquiries are the most accurate source available.

This works better in person, because whoever answers the phone usually knows what people actually ask better than any document does. Our team at Alborz Science and Technology Park runs that session with the customer support team, and its output is the initial question set.

Frequently asked questions

How many test cases are enough?

There is no magic number, but three groups must be covered: ordinary questions for each service, every defect previously found, and nonsense input. Ours began at 42 cases and grew with each new defect.

How often should it run?

Before every release, and after every change to the knowledge base. A full run takes a few minutes, and those minutes move defect discovery out of production and into testing.

Can this be automated?

Yes, and it should be. A script that reads the question list, issues the requests and checks the assertions. The one technical caveat is to send Persian text from inside the program rather than from a command line — some shells corrupt the encoding, and the result looks like a system fault.

How is this different from testing an ordinary website?

On an ordinary website the correct output is unique. In an answering system several answers can be correct, so the assertion belongs on the source of the answer rather than its exact wording — "the answer must reference the chatbot page", not "the answer must be this sentence".

Does the same method work for a model-based system?

Yes, with one addition: include a case that checks the system says plainly that it does not know when no reference document exists. The two architectures and their different risks are compared in language model or retrieval from your own knowledge.

What if we have no internal technical team?

Build the question set yourselves — that takes business knowledge, not technical knowledge — and have a contractor automate the run. Ownership of the question list should stay with you, because it is what defines quality.

Related services from Qatreh