<div dir="ltr"><div>Kip Cole writes: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I have been assuming that the general algorithm for evaluating segmentation rules is this:</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">... </blockquote><div><br></div><div>I think you are quite close, with just a couple of comments...</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">3. If the rule matches, break or don’t break depending on the operator of the rule (one of “×÷”) <b>and then move the string pointer forward</b></blockquote><div>... </div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">6. Repeat until the end of the string</blockquote><div><br></div></div><div>The algorithm tests any single arbitrary position in the string for being or not being a boundary. If you want to apply it to every position of a string, in sequence, that's fine, but it's not required.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">5. If no rule matches, apply the default rule of "Any ÷ Any" which will always match and break and then advance the string pointer</blockquote><div><br></div><div>For some types of boundaries, the default is "Any × Any"; for others it is "Any ÷ Any". In any event, the default is always included explicitly in the rules, so the algorithm itself doesn't need to mention it. If some set of rules failed to include a default, that would be a bug in the rules.<br></div><div><br></div><div>For the specific question on the sentence break of “<b>One. Two.</b>” </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The string pointer is here: “. Two.”</blockquote><div> </div><div>Between the first "." and the space there is no boundary, with the rules applied as you described.</div><div><br></div><div>Between the space and the "<b>T</b>", rule SB11 </div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><b>SATerm Close* Sp* ParaSep? ÷</b></div></blockquote><div>applies, causing a boundary. The space character binds to the preceding sentence, not the following one.</div><div><br></div><div>It's often easier to look at the rules in Unicode <a href="https://unicode.org/reports/tr29/#Sentence_Boundary_Rules">UAX 29</a> than in CLDR. The UAX rules usually match the root CLDR rules.</div><div><br></div><div> -- Andy</div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 10, 2020 at 12:31 AM Kip Cole via CLDR-Users <<a href="mailto:cldr-users@unicode.org">cldr-users@unicode.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">I have been assuming that the general algorithm for evaluating segmentation rules is this:<div><br></div><div>1. At the current pointer in the subject string</div><div>2. Evaluate rules in order until a rule “passes” (ie matches)</div><div>3. If the rule matches, break or don’t break depending on the operator of the rule (one of “×÷”) and then move the string pointer forward</div><div>4. If the rule does not match, try the next rule</div><div>5. If no rule matches, apply the default rule of "Any ÷ Any" which will always match and break and then advance the string pointer</div><div>6. Repeat until the end of the string</div><div><br></div><div>However when applying this approach to the sentence break rules in the root locale for the string “One. Two.” the following is resolved:</div><div><br></div><div>The string pointer is here: “. Two.” Apply the following sentence break rules (partial)</div><div><br></div><div><div><!-- Break after sentence terminators, but include closing punctuation, trailing spaces, and any paragraph separator. [See note below.] Include closing punctuation, trailing spaces, and (optionally) a paragraph separator. --></div><div><rule id="9"> $SATerm $Close* × ( $Close | $Sp | $ParaSep ) </rule></div><div><!-- Note the fix to $Sp*, $Sep? --></div><div><rule id="10"> $SATerm $Close* $Sp* × ( $Sp | $ParaSep ) </rule></div><div><rule id="11"> $SATerm $Close* $Sp* $ParaSep? ÷ </rule></div></div><div><br></div><div>Rule 9 will match:</div><div> "<font color="#000000"><span>$SATerm $Close*" matches the “.”</span></font></div><div><font color="#000000"><span> </span></font><span style="color:rgb(0,0,0)"> "</span><font color="#000000"><span>( $Close | $Sp | $ParaSep )" matches the “ Two.”</span></font></div><div><font color="#000000"><span><br></span></font></div><div><font color="#000000">Since it matches, and is a `no break` match then rule processing finishes and the string pointer is advanced. Therefore there is never a sentence break. Removing rule 9 results in rule processing to get to Rule 11 which matches and then breaks as expected.</font></div><div><font color="#000000"><br></font></div><div><font color="#000000">Am I incorrectly understanding the flow of rule evaluation?</font></div><div><font color="#000000"><br></font></div><div><font color="#000000">Thanks for the help as always,</font></div><div><font color="#000000"><br></font></div><div><font color="#000000">—Kip</font></div><div><font color="#000000"><br></font></div></div>_______________________________________________<br>
CLDR-Users mailing list<br>
<a href="mailto:CLDR-Users@corp.unicode.org" target="_blank">CLDR-Users@corp.unicode.org</a><br>
<a href="https://corp.unicode.org/mailman/listinfo/cldr-users" rel="noreferrer" target="_blank">https://corp.unicode.org/mailman/listinfo/cldr-users</a><br>
</blockquote></div>