<div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The root rules for word boundaries ends with:<br> <rule id="16"> [^$RI] ($RI $RI)* $RI × $RI </rule><br> <!-- Otherwise, break everywhere (including around ideographs). —><br>Which makes the final rule "Any ÷ Any” implicit as I read the spec?</blockquote><div><br></div><div>This looks like an omission in the CLDR word rules. <a href="https://unicode.org/reports/tr29/#WB999">UAX-29 word break</a>, from which the cldr rules derive, includes the default.</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">Do not break within emoji flag sequences. That is, do not break between regional indicator (RI) symbols if there is an odd number of RI characters before the break point.<br></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px">WB15 sot (RI RI)* RI × RI</blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px">WB16 [^RI] (RI RI)* RI × RI</blockquote></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px">Otherwise, break everywhere (including around ideographs).<br></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><blockquote style="margin:0 0 0 40px;border:none;padding:0px">WB999 Any ÷ Any<br><br></blockquote></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> At any point in a given subject string, the whole String is considered during the match process. In my implementation I have<br>been discarding text as I “move” the pointer forward.</blockquote><div><br></div><div>Yes. Although it turns out you never need to look further backwards than the previous break position, assuming you are finding them all in order.</div><div><br></div><div> -- Andy</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 10, 2020 at 3:36 PM Kip Cole <<a href="mailto:kipcole9@gmail.com">kipcole9@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Andy, thanks for the assist, much appreciated.<div><br></div><div>> If some set of rules failed to include a default, that would be a bug in the rules.</div><div><br></div><div>The root rules for word boundaries ends with:</div><div><br></div><div><div> <rule id="16"> [^$RI] ($RI $RI)* $RI × $RI </rule></div><div> <!-- Otherwise, break everywhere (including around ideographs). —></div><div><br></div><div>Which makes the final rule "Any ÷ Any” implicit as I read the spec?</div><div><br></div><div>> Between the space and the "<b>T</b>", rule SB11 </div><div><br></div><div>I think I see the light now. At any point in a given subject string, the whole</div><div>String is considered during the match process. In my implementation I have</div><div>been discarding text as I “move” the pointer forward. Therefore when the pointer</div><div>Is at the “ Two.” point I no longer have the prior “.” In scope and hence SB11</div><div>fails. </div><div><br></div><div>Back to that part of the drawing board ….. </div><div><br></div><div>Many thanks, —Kip</div><div><br></div><div><br></div><div><br><blockquote type="cite"><div>On 11 Jul 2020, at 4:41 am, Andy Heninger <<a href="mailto:andy.heninger@gmail.com" target="_blank">andy.heninger@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div>Kip Cole writes: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I have been assuming that the general algorithm for evaluating segmentation rules is this:</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">... </blockquote><div><br></div><div>I think you are quite close, with just a couple of comments...</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">3. If the rule matches, break or don’t break depending on the operator of the rule (one of “×÷”) <b>and then move the string pointer forward</b></blockquote><div>... </div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">6. Repeat until the end of the string</blockquote><div><br></div></div><div>The algorithm tests any single arbitrary position in the string for being or not being a boundary. If you want to apply it to every position of a string, in sequence, that's fine, but it's not required.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">5. If no rule matches, apply the default rule of "Any ÷ Any" which will always match and break and then advance the string pointer</blockquote><div><br></div><div>For some types of boundaries, the default is "Any × Any"; for others it is "Any ÷ Any". In any event, the default is always included explicitly in the rules, so the algorithm itself doesn't need to mention it. If some set of rules failed to include a default, that would be a bug in the rules.<br></div><div><br></div><div>For the specific question on the sentence break of “<b>One. Two.</b>” </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The string pointer is here: “. Two.”</blockquote><div> </div><div>Between the first "." and the space there is no boundary, with the rules applied as you described.</div><div><br></div><div>Between the space and the "<b>T</b>", rule SB11 </div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><b>SATerm Close* Sp* ParaSep? ÷</b></div></blockquote><div>applies, causing a boundary. The space character binds to the preceding sentence, not the following one.</div><div><br></div><div>It's often easier to look at the rules in Unicode <a href="https://unicode.org/reports/tr29/#Sentence_Boundary_Rules" target="_blank">UAX 29</a> than in CLDR. The UAX rules usually match the root CLDR rules.</div><div><br></div><div> -- Andy</div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 10, 2020 at 12:31 AM Kip Cole via CLDR-Users <<a href="mailto:cldr-users@unicode.org" target="_blank">cldr-users@unicode.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>I have been assuming that the general algorithm for evaluating segmentation rules is this:<div><br></div><div>1. At the current pointer in the subject string</div><div>2. Evaluate rules in order until a rule “passes” (ie matches)</div><div>3. If the rule matches, break or don’t break depending on the operator of the rule (one of “×÷”) and then move the string pointer forward</div><div>4. If the rule does not match, try the next rule</div><div>5. If no rule matches, apply the default rule of "Any ÷ Any" which will always match and break and then advance the string pointer</div><div>6. Repeat until the end of the string</div><div><br></div><div>However when applying this approach to the sentence break rules in the root locale for the string “One. Two.” the following is resolved:</div><div><br></div><div>The string pointer is here: “. Two.” Apply the following sentence break rules (partial)</div><div><br></div><div><div><!-- Break after sentence terminators, but include closing punctuation, trailing spaces, and any paragraph separator. [See note below.] Include closing punctuation, trailing spaces, and (optionally) a paragraph separator. --></div><div><rule id="9"> $SATerm $Close* × ( $Close | $Sp | $ParaSep ) </rule></div><div><!-- Note the fix to $Sp*, $Sep? --></div><div><rule id="10"> $SATerm $Close* $Sp* × ( $Sp | $ParaSep ) </rule></div><div><rule id="11"> $SATerm $Close* $Sp* $ParaSep? ÷ </rule></div></div><div><br></div><div>Rule 9 will match:</div><div> "<font><span>$SATerm $Close*" matches the “.”</span></font></div><div><font><span> </span></font><span> "</span><font><span>( $Close | $Sp | $ParaSep )" matches the “ Two.”</span></font></div><div><font><span><br></span></font></div><div><font>Since it matches, and is a `no break` match then rule processing finishes and the string pointer is advanced. Therefore there is never a sentence break. Removing rule 9 results in rule processing to get to Rule 11 which matches and then breaks as expected.</font></div><div><font><br></font></div><div><font>Am I incorrectly understanding the flow of rule evaluation?</font></div><div><font><br></font></div><div><font>Thanks for the help as always,</font></div><div><font><br></font></div><div><font>—Kip</font></div><div><font><br></font></div></div>_______________________________________________<br>
CLDR-Users mailing list<br>
<a href="mailto:CLDR-Users@corp.unicode.org" target="_blank">CLDR-Users@corp.unicode.org</a><br>
<a href="https://corp.unicode.org/mailman/listinfo/cldr-users" rel="noreferrer" target="_blank">https://corp.unicode.org/mailman/listinfo/cldr-users</a><br>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>