Characters recommended in both UTS#39 and MSR but excluded from the Root Zone or Reference LGR

This document has been submitted as a UTC document. For convenience in documenting the character list it is presented using an LGR template format. A few minor details of the boilerplate in that template may not be applicable in this context and should be disregarded.

The collection comprises 274 characters from the [MSR] that are recommended in UTS#39 but are not part of the Reference LGR [RefLGR], as well as the uppercase equivalents for 88 of them (Latin), plus 70 decimal digits excluded from the RefLGR, for a total of 432 characters.

Recommendation

These 432 characters should be considered Uncommon_Use, based on the fact that the expert teams charged with reviewing them for the ICANN Root Zone LGR and Reference LGR for the Second Level could not come up with evidence that they are used in common everyday writing, even for minority languages in reasonably widespread use. Consequently, they declined to include them in the respective LGRs.

Background

There are about a thousand non-Han characters with Identifier_Type Recommended that should be reclassified because they appear to fail reasonable criteria for being needed in identifiers. They come in two sets. For one set, an independent analysis [MSR] has found indications that they should have been considered Uncommon_Use, Obsolete or Technical based on information available at the time of encoding. That set is discussed in another document. The second set contains characters that were tentatively retained as Recommended in the [MSR] but upon further review by local expert teams from the [RZ-LGR] project were found to not be needed for any language or minority language in reasonably widespread use. That determination started with the [EGIDS] classification as a proxy but made further adjustments in expert review.

This analysis was carried out for the purposes of defining the repertoire for IDN Top Level Domain names for the DNS Root Zone. There are some restrictions that are specific to the Root Zone, such as a prohibition on digits, so a follow-on effort determined how to relax these restrictions in a manner appropriate for the needs of Second-Level Domains. This resulted in the Second-Level Reference Label Generation Rules [RefLGR]. The characters listed in this document are those that were not added to the [RefLGR], for lack of evidence of their use in everyday common writing for any language or minority language in vigorous and reasonably widespread use. Also listed are their uppercase equivalents as well as any native digit sets that were not added to the [RefLGR].

The implication here is that any character not included in the Reference LGR for lack of documented or identifiable usage should be considered Uncommon_Use for Unicode's default identifiers—until such time as independent evidence to the contrary is produced. Until then, in lack of a demonstrated use case, it seems not helpful to continue to suggest that these characters should be supported as recommended. This also applies to some of the sets of native digits, where local experts considered them obsolete for the purpose of identifiers.

Arriving at a precise cutoff for Uncommon_Use is difficult because there is no single source or perfect information on the use of writing systems, and the details of such use are changing over time. Accordingly, this document suggests that the UTC should consider the published results of the cited research as one of the better sources of information available and only deviate from it on the basis of even better information.

All decisions for the classification of characters in [MSR], or inclusion in [RZ-LGR] and [RefLGR] are documented and sourced on the character level; the same is not true for Unicode's classification, so it is not easily possible to verify any of the decisions that underlie the classification published in UTS39. By first making the alignment proposed here, and then carefully documenting deviations, a positive side effect might be that the classification overall becomes more transparent and reviewable.

Special Considerations

The [RZ-LGR] and [RefLGR] exclude the Bopomofo script, considering the entire script special use as it tends to be used almost exclusively in education. This could be addressed by either changing the status of the script in UAX31 to Limited_Use or by marking the entire set of Bopomofo characters as Technical. (This is not reflected in the list of characters in this document.)

No definite recommendations can be made for the Tibetan script. It is considered by ICANN as eligible for the Root Zone in principle, but work on defining the label generation rules has faced some difficulties and has not commenced. It might be reasonable to reflect that uncertainty by also not giving these characters Identifier_Type Recommended until some body, project, or group has created a definite analysis of this script for identifier purposes. (Tibetan characters have been excluded from the list of characters in this document).

Arabic combining marks are categorically excluded from domain names, see also RFC5564. In consequence, they should not be Recommended by Unicode, but if it is felt that Uncommon_Use is not the best classification, then perhaps Inclusion or Technical might be more appropriate.

Root Zone and Reference LGRs

For further background on the DNS Root Zone and Second-Level Reference LGR see the cited references and links therein.

Additional Notes

• U+0931 ऱ DEVANAGARI LETTER RRA is part of the Root Zone and Reference LGR via sequence (does not occur standalone, but should be retained in Recommended)
• U+09BC  ়  BENGALI SIGN NUKTA is part of the Root Zone and Reference LGR via sequence (does not occur standalone, but should be retained in Recommended)
• U+0DA6 ඦ SINHALA LETTER SANYAKA JAYANNA is part of the Root Zone and Reference LGR via sequence (does not occur standalone, but should be retained as Recommended)
• U+0E45 ๅ THAI CHARACTER LAKKHANGYAO is part of the Root Zone and Reference LGR via sequence (does not occur standalone, but should be retained as Recommended)
• U+1063  ၣ  MYANMAR TONE MARK SGAW KAREN HATHI is part of the Root Zone and Reference LGR via sequence (does not occur standalone, but should be retained in Recommended)

Note: All characters have tags matching their Identifier_Type values, except Uppercase equivalents are tagged as Uppercase. A comment indicates the nature of the exclusion from the [RefLGR], in this case "Not documented to be in common use". The definition of IDNs excludes uppercase characters, however, for case pairs the analysis for the lowercase letter is treated as applicable. Native digit sets excluded from the [RefLGR] based on information that their use is not preferred in that context are listed with their Identifier_Type and a comment indicating their exclusion from the [RefLGR].

Discussion and Review

Domain names are an important, and deliberately conservative set of identifiers. That said, there may be other classes of identifiers that don't require the same level of restrictions, so this proposal should not be understood to suggest that default Identifiers must be restricted to only those characters that are being recommended for IDNs. Rather, the purpose is to bring the facts discovered during the development of the IDN repertoire for the DNS Root Zone and the [RefLGR] to the attention of the Unicode Technical Committee, so that characters that were classified Recommended can be given additional scrutiny before confirming their status.

As review progresses, a number of characters have been identified that may well have documented use:

• U+0671 ٱ ARABIC LETTER ALEF WASLA - this letter is considered to be "an important Quranic character" (which would make it Technical, but not Uncommon_Use). It is also claimed to be used with a newly invented orthography Luri language in Iran.

Other issues

Combining marks: where combining marks are excluded, but needed for decompositions (such as U+0654  ٔ ), it was proposed to focus on the NFC format for Identifier_Type, documenting that combining characters may be marked as Uncommon_Use even when they are in the NFD version of a modern language's exemplar characters.

Combining marks and Arabic Script: the Internet Architecture Board [IAB] has issued a statement referencing this issue. Please also see the “Proposal for Arabic Script Root Zone LGR”, [Proposal-Arabic].

Contributors

This excerpt was prepared by Asmus Freytag, based on published data found in [RefLGR] and reference information from [MSR]. For details on the process and contributors to those projects, see [RefLGR-Overview], in particular, Section 1, “Overview” and Section 6, “Contributors”. Michel Suignard and Roozbeh Pournader have contributed feedback.