Why word error rate misleads on Thai transcripts

This post is about scoring a Thai transcript in a way two models can be compared on. It covers what the tokenizer does to the number, which metric to use instead, what to normalize before counting, and how to build a test set from the audio you already have. It gives no figure for any model, including ours. A figure without the method beside it is the problem the post is about.

Take one reference line and one hypothesis that differs by a single syllable. Score it under two common Thai tokenizers. One treats งบประมาณ as a word, the other splits it into งบ and ประมาณ. The same mistake produces a different word error rate under each, and neither is wrong.

A published word error rate for Thai therefore carries a hidden variable. Two vendors scored with two tokenizers cannot be compared, and a single vendor's number cannot be reproduced without the tokenizer named. A tokenizer that splits more finely lowers the number for everyone.

Character error rate counts the same three operations over characters. Thai characters have boundaries. The count is the same on every machine and needs no tokenizer. It is the metric most Thai speech research reports, and it is the one to ask a vendor for when a word error rate is quoted.

A character error rate is smaller than the word figure for the same transcript, because an error touches one character in a word of several. The scale is different and the comparisons still hold, as long as both models are scored the same way on the same references.

Character error rate has one blind spot. A tone mark or an upper vowel written in the wrong order is a character error even when the rendered word looks right. Normalize the Unicode order of both texts before counting, and the blind spot closes.

A transcript and a reference disagree in ways that are conventions, and a score that counts them measures the convention. Decide each one before scoring and apply the same rule to both sides.

• Numbers. A reference written with digits and a hypothesis written with number words differ by every character of the number. Choose one form. The transcription API writes either on request, and the spoken form matches a reference typed from the audio.
• Spaces. Thai spacing between phrases is inconsistent between two typists. Remove every space before counting, or count spaces separately as their own figure.
• The repetition mark. ๆ and the word written twice are the same speech. Expand the mark on both sides.
• Punctuation and case in English words. A Thai sentence that switches into English carries English conventions with it. Lowercase and strip punctuation on both sides.
• Filler words. A verbatim reference keeps เอ่อ and a clean one drops it. Score verbatim output against a verbatim reference. A clean style is a readability choice and scores worse against a verbatim reference by construction.

A vendor's test set sounds like the vendor's data. Yours sounds like your calls, your meetings, or your lectures, with your product names and your speakers. Fifty recordings of two to five minutes each, chosen to cover the speakers, rooms, and topics you have, is a test set a model can be judged on.

Have the reference typed by a person listening to the audio, under a written rule for each item in the list above. Have a second person check a sample. A reference with a five percent error rate of its own cannot resolve a difference of two points between two models.

Keep the references and the audio together and score every candidate model on the whole set with one script. Report the character error rate, the normalization rules, and the size of the set. A reader with all three can reproduce the number, and a number a reader can reproduce is one they can trust.

Paxa STT Lite publishes no accuracy figure of its own. The evaluation method comes first, and a figure follows only beside the set and the rules that produced it. Until then, the recording on the product page and the transcribe tool are the way to judge the model on audio you choose.

• Read the transcription guide for the conventions and the style option that the normalization rules above depend on.
• Read the numbers post for how the spoken and written conventions differ, line by line.
• Open the transcribe tool and run a few of your own recordings before writing a single reference.