What Thai word segmentation is, and why it decides translation, speech, and search

The step is invisible in English because the space already did the work. In Thai the space exists and means something else. This post explains what the space marks and shows the strings that can be cut two ways. It then follows one wrong cut into a translation, a spoken sentence, a subtitle file, and a search index.

Thai writing runs its words together inside a phrase. ผมกินข้าวที่ร้านอาหารไทย reads as six words to a Thai speaker and as one token to any function that splits on whitespace. The space character is reserved for a larger unit. It marks the end of a phrase or a clause, the job a comma or a period does in English, and it appears far less often than an English reader expects.

That convention is old and consistent. Numbers, names, and foreign words sometimes take a space on either side, and a sentence-final particle sometimes stands apart, but the words inside a Thai phrase never do. A reader learns to find the boundaries from vocabulary and context, the same way an English reader resolves which sense of lead a sentence uses.

Words are short. On the BEST2010 corpus the average Thai word is 3.855 characters long (COLING (2014)), and this site rounds that to 4 when it converts a word count into billable characters. A hundred-word paragraph is therefore a string of about four hundred characters with perhaps three spaces in it, and every one of the hundred boundaries has to be inferred.

Most Thai text segments one way once the vocabulary is known. The hard cases are strings where two different cuts both produce real words. A dictionary cannot settle them, because the dictionary contains every word on both sides.

The first two rows are the ones a reader meets in practice. ตากลม is a real ambiguity, and only the sentence around it decides which reading is meant. ไปหามเหสี looks ambiguous to a machine and never to a person, because the second cut leaves syllables that do not form words. The last two rows show the same shape: one cut is a phrase people say, the other is a sequence of dictionary entries that nobody would write.

The figure is the whole problem in one line. Nothing inside ตากลม chooses a reading. The choice comes from เด็กคนนี้ (this child) or เอาผ้าไป (take the cloth), which sit outside the string a per-word matcher is looking at.

A translation system that segments first and translates second inherits every wrong cut. เอาผ้าไปตากลม cut as ตา กลม produces a sentence about a cloth and round eyes. The output is fluent, grammatical, and wrong, and a reviewer who does not read Thai has no signal that anything happened.

Names are the common case. A brand written in Thai script, ไลน์ for a chat app or แกร็บ for a ride service, is a string of ordinary syllables. Cut the wrong way, ไลน์ becomes a word for a line and the brand disappears from the translation. The translation API takes a glossary for exactly this reason. A term listed there in its published form is matched as a unit and rendered as the caller wrote it.

Markup is the other place a cut goes wrong. An inline tag reinserted by character offset lands inside a word whenever the offset drifts by one, and in Thai that means between a consonant and the vowel that belongs to it. The format field exists so that the model receives the markup and keeps it on word boundaries it can see.

A speech model needs the word to know two things: where to breathe and how to read the letters. Phrasing follows words and clauses. A pause dropped inside a word sounds like a stutter, and a sentence read without any pause sounds like a machine. Neither the space nor the punctuation tells the model where the words are.

Syllable reading depends on the word as well. The same letters cut into different syllables in different words, and only the word decides. ปลา reads as one syllable, plaa, with a consonant cluster. ปลัด opens on the same two letters and reads as two syllables, pa-lat. จักรยาน reads jak-kra-yaan and จักรี reads jak-kri, with a different cut after the shared จักร. The doubled รร reads un at the end of a word in สรร (san) and um before a final consonant in กรรม (kam).

A model that has found the word reads each of these correctly. A model that reads letters in sequence gets each one wrong on the first syllable. The text-to-speech page lists more of these reading rules with the reading a Thai speaker produces.

A word timestamp is a span of audio attached to a word. In English the words are given and the alignment is the whole task. In Thai the model has to produce the words first, and the timestamps inherit whatever it produced. A wrong cut yields a span that covers one and a half words, and a subtitle built from those spans highlights the wrong syllables.

The speech API returns timestamps at word, sentence, or utterance granularity, and the word spans follow the model's own segmentation of the text. The subtitles page shows what that looks like as a highlighted read-along. Sentence granularity sidesteps the question entirely when a caption only needs to change once per line.

A search index stores terms, and a term is a word. An index built by splitting Thai on spaces stores whole phrases as single terms, and a query for one word inside a phrase finds nothing. An index built with a segmenter finds the word, and finds it under whichever cut the segmenter chose. The query has to be cut the same way, by the same segmenter, or the two sides disagree.

The practical consequence is that Thai search quality is capped by segmentation quality, and that changing the segmenter means rebuilding the index. Product catalogs feel this first, because product names are exactly the strings a segmenter has never seen.

Two families of method exist. Dictionary methods walk the string and take the longest word the dictionary contains at each position, with a backtrack when the remainder cannot be segmented. They are fast, transparent, and correct on most text. They fail on the ambiguous strings above, because both cuts are in the dictionary and the method has no way to look at the surrounding sentence. They also fail on every word that is not in the dictionary, which includes most new names.

Context models read the characters around each position and predict whether a boundary falls there. They learn that เด็กคนนี้ is followed by a description and เอาผ้าไป by an action, and they cut ตากลม accordingly. They handle unknown words by shape, since a Thai word has a recognizable structure of consonant, vowel, and tone mark. Their cost is that they are models, with the training data and the failure cases that implies.

• Send whole sentences or segments. A fragment cut at an arbitrary character count can start or end inside a word, and the model then has to guess at both edges.
• Put brand names, product names, and place names in the glossary in the exact form you publish. A listed term is matched as a unit and never split.
• Use do_not_translate for strings that must survive untouched, and send format when the text carries markup. The model then preserves the tags on boundaries it can see.
• Request timestamps at the granularity the product needs. Sentence spans are stable; word spans are the model's own cut of the text.
• Segment queries with the same method that built the index. A mismatch between the two sides is the usual cause of a Thai search that misses obvious matches.
• Test the ambiguous strings from the table above in the playground with the sentence around them, and read the output as a Thai speaker would.

Segmentation is the first decision any Thai pipeline makes, and the one every later output carries. The translation and text-to-speech models on this site make that decision inside the model, with the whole sentence in view. The fields above are how a caller keeps it from being made alone.