UAX #29 and WB4
Daniel Bünzli via Unicode
unicode at unicode.org
Wed Mar 4 11:01:25 CST 2020
Hello,
My implementation of word break chokes only on the following test case from the file [1]:
÷ 0020 × 0308 ÷ 0020 ÷ # ÷ [0.2] SPACE (WSegSpace) × [4.0] COMBINING DIAERESIS (Extend_FE) ÷ [999.0] SPACE (WSegSpace) ÷ [0.3]
I find:
÷ 0020 × 0308 × 0020 ÷
Basically my implementation uses WB4 to rewrite the first two characters to WSegSpace and then applies WB3ad resulting in the non-break between 0308 and 0020.
Re-reading the text I suspect I should not restart the rules from the first one when a WB4 rewrite occurs but only apply the subsequent rules. Is that correct ?
Best,
Daniel
[1]: https://unicode.org/Public/13.0.0/ucd/auxiliary/WordBreakTest.txt
More information about the Unicode
mailing list