Qatreh AI Logoقطره
Back to blog
Conversational AI

Why Your Persian Chatbot Answers the Wrong Question

A zero-width joiner, an Arabic yeh and three digit sets turn one word into several strings. The mechanism behind the failure, and the fix that resolves it.

تیم قطرهSeptember 5, 20266 min read
Why Your Persian Chatbot Answers the Wrong Question

When a Persian-language assistant answers half its questions with "I don't have information about that," the first assumption is always that its knowledge base is too small. Usually it isn't. The program is seeing one word as several unrelated strings.

The short answer

In Persian — and in Arabic script generally — a single word can be written several ways that look identical to a reader and are completely different to a computer. "چت‌بات" written with a zero-width non-joiner and "چت بات" written with an ordinary space share no byte sequence. Until text is folded into one comparable form before any comparison, a large share of real questions will miss, and nothing in the logs will indicate a fault.

One word, several spellings

None of these variants is a spelling mistake. All of them appear in formal published Persian, and all are encoded distinctly in Unicode:

  • The zero-width non-joiner (U+200C) is a real character with no width. It separates parts of a compound without a space, so "می‌رود", "می رود" and "میرود" are three distinct strings.
  • Persian yeh (U+06CC) versus Arabic yeh (U+064A) render nearly identically in most fonts, and different keyboard layouts produce different ones.
  • Persian keheh (U+06A9) versus Arabic kaf (U+0643) — the same situation.
  • Heh versus teh marbuta — a writer using an Arabic layout types "شرکة" rather than "شرکه".
  • Three digit sets. Persian digits (U+06F0–U+06F9), Arabic-Indic digits (U+0660–U+0669) and ASCII digits are three separate encodings of the same numbers.
  • Diacritics and tatweel. Optional vowel marks and the decorative elongation character survive in pasted text and break matching silently.

For a term as ordinary as "chatbot", that yields at least four common written forms, all correct. A system that recognises one of them fails three users in four.

Why nothing appears in the logs

This is the part that makes the problem persist. When matching fails the program does not crash — it falls through to a default reply. The web server records a 200, uptime monitoring stays green, and no dashboard turns red.

From outside, the only symptom is that a user asks once, receives a generic answer, and closes the window. That pattern was present in our own system and stayed invisible until a fixed set of real questions was run against it. The method is in how to actually test a Persian chatbot.

The second trap: substring matching

Suppose the system detects intent by searching for keywords inside the incoming question. It is the simplest approach and it carries a serious flaw, because Persian attaches prefixes and suffixes freely.

A real example from our own system: the keyword "زمان" (time) was registered to detect questions about project schedules. A user asked "آموزش سازمانی برای شرکت ما دارید؟" — do you offer enterprise training? The four-letter keyword was found as a substring inside "سازمانی" (organisational), and the system answered with a project timeline.

The fix is to compare at whole-token level rather than by substring, and to allow a prefix match only for terms long enough that the shared opening cannot be coincidental — so "آموزش" still matches "آموزشی", while "زمان" no longer matches "سازمانی".

The order the folding has to happen in

Before any comparison, the incoming text and the reference text must pass through the same pipeline:

  1. Strip diacritics and tatweel
  2. Convert the zero-width non-joiner and directional marks to a plain space
  3. Convert Persian and Arabic-Indic digits to ASCII
  4. Unify letters: yeh, alef maqsura and yeh hamza to Persian yeh; Arabic kaf to keheh; teh marbuta to heh; the alef forms to plain alef
  5. Lowercase any Latin characters
  6. Replace every non-letter, non-digit character with a space and collapse runs of whitespace

The subtlety is in step two. The zero-width non-joiner must become a space, not be deleted. Delete it and "چت‌بات" becomes "چتبات", which still does not match "چت بات". Convert it to a space and both spellings arrive at the same two tokens and find each other.

Why this matters more for a local deployment

Customers of a shop, a training centre or a factory in Karaj and across Alborz province write from a range of devices and keyboard layouts. Some produce standard Persian characters; others produce the Arabic forms. Both families are therefore present in genuine inbound messages.

A system validated only against clean, uniform text performs flawlessly in a demonstration and stumbles on contact with real users. Our team at Alborz Science and Technology Park runs this evaluation against real message traffic rather than synthetic text, because the gap between those two is the gap between a successful demo and a service that holds up.

Frequently asked questions

What exactly is the zero-width non-joiner and why does it break things?

It is Unicode character U+200C. It has no visual width but is stored in the string. "می‌رود" and "می رود" are therefore identical to the eye and different to any string comparison.

Does this only affect chatbots?

No. Anywhere Persian text is compared is affected: product search, customer-name lookup, address matching, deduplication. In a database, two spellings of one company name remain two separate records.

What share of questions fail because of this?

There is no fixed figure; it depends on the vocabulary involved. In our own evaluation, before folding was introduced, three out of ten perfectly ordinary Persian questions fell through to the default reply. Afterwards, all ten reached the correct topic.

Do large language models avoid the problem?

They are more tolerant of spelling variation, but they do not remove it. Wherever string matching happens before or after the model — document retrieval, filtering, routing to a department — the same trap returns. We compare the two architectures in language model or retrieval from your own knowledge.

How expensive is the folding step?

Negligible. It is six replacements over a short string. In our measurements the entire server-side processing time, including retrieving the answer, was 10 to 14 milliseconds. The cost of not doing it is far higher.

Related services from Qatreh