24 July 2026
What makes Bangla hard for systems built on English assumptions
Script, tokenisation, code-switching and OCR, and where each one breaks a pipeline that was not designed for it.
It is not a translation problem
The usual instinct with a non-English language is to treat it as English with different words: swap the tokeniser, find a multilingual model, move on. That works until it does not, and where it fails is rarely where people expect.
Script and shaping
Bangla is an abugida. Consonants carry an inherent vowel, vowel signs attach around the consonant rather than following it, and consonant clusters form conjuncts that are visually a single glyph. A naive character-level operation that is safe in Latin script, truncating a string to a length, reversing it, splitting on what looks like a boundary, can produce something that is not a word and sometimes not renderable.
Length limits are the common casualty. A field capped at a character count behaves differently in Bangla than the developer assumed, because the relationship between characters, code points and rendered glyphs is not the one they have in their head.
Tokenisation
Subword tokenisers trained predominantly on English spend far more tokens per unit of meaning in Bangla. The practical effects are a smaller usable context window, a higher cost per request, and, in a retrieval system, chunks that hold less actual content than the token count suggests. If you size your chunks in tokens and validate on English, your Bangla chunks are quietly smaller.
Code-switching
Real Bangla text, especially anything typed by a person under time pressure, mixes English freely. Not paragraph by paragraph, but inside a single sentence, including English technical terms in Bangla grammatical structure and Bangla words typed in Latin script.
A pipeline that detects the language of a document and routes it down one of two paths handles this badly, because there is no correct answer to route on. Handling code-switched input directly, rather than deciding what language it is first, removes a whole class of failure.
OCR
Scanned Bangla has a worse error profile than scanned English, and the errors are different in kind: conjuncts and vowel signs are where it goes wrong, and a misread conjunct can produce a valid but different word rather than obvious garbage. This is why layout context matters more here. Knowing that a region is a date field constrains what a misread can plausibly have been.
What to do about it
Test on the real thing. Not translated English, not clean corpus Bangla, but the actual documents and the actual questions, including the ones with three English words in the middle. Almost every failure above is invisible until you do.