NNBSP

[Marcel Schneider via Unicode]

On 19/01/2019 01:55, Asmus Freytag via Unicode wrote:
> On 1/18/2019 2:05 PM, Marcel Schneider via Unicode wrote:
>> On 18/01/2019 20:09, Asmus Freytag via Unicode wrote:
>>>
>>> Marcel,
>>>
>>> about your many detailed *technical* questions about the history of character properties, I am afraid I have no specific recollection.
>>>
>> Other List Members are welcome to join in, many of whom are aware of how things happened. My questions are meant to be rather simple. Summing up the premium ones:
>>
>>  1. Why does UTC ignore the need of a non-breakable thin space?
>>  2. Why did UTC not declare PUNCTUATION SPACE non-breakable?
>>
>> A less important information would be how extensively typewriters with proportional advance width were used to write books ready for print.
>>
>> Another question you do answer below:
>>
>>> French is not the only language that uses a space to group figures. In fact, I grew up with thousands separators being spaces, but in much of the existing publications or documents there was certainly a full (ordinary) space being used. Not surprisingly, because in those years documents were typewritten and even many books were simply reproduced from typescript.
>>>
>>> When it comes to figures, there are two different types of spaces.
>>>
>>> One is a space that has the same width a digit and is used in the layout of lists. For example, if you have a leading currency symbol, you may want to have that lined up on the left and leave the digits representing the amounts "ragged". You would fill the intervening spaces with this "lining" space character and everything lines up.
>>>
>> That is exactly how I understood hot-metal typesetting of tables. What surprises me is why computerized layout does work the same way instead of using tabulations and appropriate tab stops (left, right, centered, decimal [with all decimal separators lining up vertically).
>
> ==> At the time Unicode was first created (and definitely before that, during the time of non-universal character sets) many applications existed that used a "typewriter model" and worked by space fill rather than decimal-point tabulation.
>
If you are talking about applications, as opposed to typesetting tables for book printing, then I’d suggest that the fixed-width display of tables could be done much like still today’s source code layout, where normal space is used for that purpose. In this use case, line wrap is typically turned off. That could make non-breakable spaces sort of pointless (but I’m aware of your point below), except if people are expected to re-use the data in other environments. In that case, best practice is to use NNBSP as thousands separator while displaying it like other monospace characters. That’s at least how today’s monospace fonts work (provided they’re used in environments actually supporting Unicode, which may not happen with applications running in terminal).
>
> From today's perspective that older model is inflexible and not the best approach, but it is impossible to say how long this legacy approach hung on in some places and how much data might exist that relied on certain long-standing behaviors of these space characters.
>
My position since some time is that legacy apps should use legacy libraries. But I’ll come back on this when responding to Shawn Steele.
>
> For a good solution, you always need to understand
>
> (1) the requirement of your "index" case (French, in this case)
>
That’s okay.
>
> (2) how it relates to similar requirements in (all!) other languages / scripts
>
That’s rather up to CLDR as I suggested, given it has the means to submit a point to all vetters. See again below (in the part that you’ve cut off without consideration).
>
> (3) how it relates to actual legacy practice
>
That’s Shawn Steele’s point (see next reply).
>
> (3a) what will suddenly no longer work if you change the properties on some character
>
> (3b) what older data will no longer work if the effective behavior of newer applications changes
>
I’ll already note that this needs to be aware of actual use cases and/or to delve into the OSes, that is far beyond what I can currently do, both wrt time and wrt resources. The vetter’s role is to inform CLDR with correct data from their locale. CLDR is then welcome to sort things out and to get in touch with the industry, which CLDR TC is actually doing. But that has no impact on the data submitted at survey time. Changing votes to tell “OK let the group separator be NBSP as long as…” would be a lie.
>
>>> In lists like that, you can get away with not using a narrow thousands separator, because the overall context of the list indicates which digits belong together and form a number. Having a narrow space may still look nicer, but complicates the space fill between the symbol and the digits.
>>>
>> It does not, provided that all numbers have thousands separators, even if filling with spaces. It looks nicer because it’s more legible.
>>>
>>> Now for numbers in running text using an ordinary space has multiple drawbacks. It's definitely less readable and, in digital representation, if you use 0020 you don't communicate that this is part of a single number that's best not broken across lines.
>>>
>> Right.
>>>
>>> The problem Unicode had is that it did not properly understand which of the two types of "numeric" spaces was represented by "figure space". (I remember that we had discussions on that during the early years, but that they were not really resolved and that we moved on to other issues, of which many were demanding attention).
>>>
>> You were discussing whether the thousands separator should have the width of a digit or the width of a period? Consistently with many other choices, the solution would have been to encode them both as non-breakable, the more as both were at hand, leaving the choice to the end-user.
>
> ==> Right, but remember, we started off encoding a set of spaces that existed before Unicode (in some other character sets) and implicitly made the assumption that those were the correct set (just like we took punctuation from ASCII and similar sources and only added to it later, when we understood that they were missing things --- generally always added, generally did not redefine behavior or shape of existing code points).
>
Now I understand that what UAX #14 calls “the preferred space for use in numbers” is actually preferred in the table layout you are referring to, because it is easier to code when only the empty decimal separator position uses PUNCTUATION SPACE, while grouping is performed with FIGURE SPACE.

That raises two questions, one of which has been often asked in this thread:

 1. How is FIGURE SPACE supposed to be supported in legacy environments? (UAX #14 mentions both its line breaking behavior and its width, but makes no concessions for legacy apps…)
 2. Why did PUNCTUATION SPACE not be declared non-breakable? (If it had, it could have been re-purposed to space off French punctuation since the beginning of Unicode, and never French users had have a reason to be upset by lack of a narrow non-breaking space.)

>>
>> Current practice in electronic publishing was to use a non-breakable thin space, Philippe Verdy reports. Did that information come in somehow?
>
> ==> probably not in the early days. Y
>
Perhaps it was ignored from the beginning on, like Philippe Verdy reports that UTC ignored later demands, getting users upset.
That leaves us with the question why it did so, downstream your statement that it was not what I ended up suspecting.

Does "Y" stand for the peace symbol?
>
>>
>> ISO 31-0 was published in 1992, perhaps too late for Unicode. It is normally understood that the thousands separator should not have the width of a digit. The allaged reason is security. Though on a typewriter, as you state, there is scarcely any other option. By that time, all computerized text was fixed width, Philippe Verdy reports. On-screen, I figure out, not in book print
>
> ==> much book printing was also done by photomechanically reproducing typescript at that time. Not everybody wanted to pay typesetters and digital typesetting wasn't as advanced. I actually did use a digital phototypesetter of the period a few years before I joined Unicode, so I know. It was more powerful than a typewriter, but not as powerful as TeX or later the Adobe products.
>
> For one, you didn't typeset a page, only a column of text, and it required manual paste-up etc.
>
Did you also see typewriters with proportional advance width (and interchangeable type wheels)? That was the high end on the typewriter market. (Already mentioned these typewriters in a previous e‑mail.) Books typeset this way could use bold and (less easy) italic spans.
>
>>> If you want to do the right thing you need:
>>>
>>> (1) have a solution that works as intended for ALL language using some form of blank as a thousands separator - solving only the French issue is not enough. We should not do this a language at a time.
>>>
>> That is how CLDR works. 
>
> CLDR data is by definition per-language. Except for inheritance, languages are independent.
>
> There are no "French" characters. When you encode characters, at best, some code points may be script-specific. For punctuation and spaces not even that may be the case. Therefore, as long as you try to solve this as if it *only* was a French problem, you are not doing proper character encoding.
>
Again, I did not do that (and BTW CLDR is not doing “character encoding”). Actually, to be able to post that blame you needed to cut off all the URLs I provided you with. These links are documenting that i did not “try to solve this as if it only was a French problem[.]”

Here they are again, this time with copy-pasted snippets below.
I wrote: “But as soon as that was set up, I started lobbying for support of all relevant locales at once:”

https://unicode.org/cldr/trac/ticket/11423
https://unicode.org/pipermail/cldr-users/2018-September/000842.html

  * “To be cost-effective, locales using space as numbers group separator should migrate at once from the wrong U+00A0 to the correct U+202F. I didn’t aim at making French stand out, but at correcting an error in CLDR. Having even the Canadian French sublocale stick with the wrong value makes no sense and is mainly due to opaque inheritance relationships and to severe constraints on vetters applying for fr-FR and subsequently reduced to look on helpless from the sidelines when sublocales are not getting fixed.”

  * “After having painstakingly catched up support of some narrow fixed-width no-break space (U+202F). the industry is now ready to migrate from U+00A0 to U+202F. Doing it in a single rush is way more cost-effective than migrating one locale this time, another locale next time, a handful locales the time after, possibly splitting them up in sublocales with different migration schedules. I really believed that now Unicode proves ready to adopt the real group separator in French, all relevant locales would be consistently pushed for correcting that value in release 34. The v34 alpha overview makes clear they are not. ​
    http://cldr.unicode.org/index/downloads/cldr-34#TOC-Migration

    I aimed at correcting an error in CLDR, not at making French stand out. Having many locales and sublocales stick with the wrong value makes no sense any more.
    ​https://www.unicode.org/cldr/charts/34/by_type/numbers.symbols.html#a1ef41eaeb6982d

    The only effect is implementers skipping migration for fr-FR while waiting for the others to catch up, then doing it for all at once.

    There seems to be a misunderstanding: The*locale setting *is whether to use period, comma, space, apostrophe, U+066C ARABIC THOUSANDS SEPARATOR, or another graphic. Whether "space" is NO-BREAK SPACE or NARROW NO-BREAK SPACE is *not a locale setting,* but it’s all about Unicode *design* and Unicode *implementation.* I really thought that that was clear and that there’s no need to heavily insist on the ST "French" forum. When referring to the "French thousands separator" I only meant that unlike comma- or period-using locales, the French locale uses space and that the group separator space should be the correct one. That did *not* mean that French should use *another* space than the other locales using space.”

https://unicode.org/pipermail/cldr-users/2018-September/000843.html
and
https://unicode.org/cldr/trac/ticket/11423#comment:2

  * “I've to confess that I did focus on French and only applied for fr-FR, but there was a lot of work, see ​
    http://cldr.unicode.org/index/downloads/cldr-34#TOC-Growth
    waiting for very few vetters. Nevertheless I also cared for English (see various tickets), and also posted on CLDR-users in a belated P.S. that fr-CA hadn’t caught up the group separator correction yet:
    ​https://unicode.org/pipermail/cldr-users/2018-August/000825.html

    Also I’m sorry for failing to provide appropriate feedback after beta release and to post upstream messages urging to make sure all locales using space for group separator be kept in synchrony.

    I think the point about not splitting up all the data into locales is a very good one.

    There should be a common pool so that all locales using Arabic script have automatically group separator set to ARABIC THOUSANDS SEPARATOR (provided it actually fits all), and those locales using space should only need to specify "space" to automatically get the correct one, ie NARROW NO-BREAK SPACE as soon as Unicode is ready to give it currency in that role.”


Do these recommendations meet your requirements and sound okay to you?
>>>
>>> Do you have colleagues in Germany and other countries that can confirm whether their practice matches the French usage in all details, or whether there are differences? (Including differently acceptability of fallback renderings...).
>>>
>> No I don’t but people may wish to read German Wikipedia:
>>
>> https://de.wikipedia.org/wiki/Zifferngruppierung#Mit_dem_Tausendertrennzeichen
>>
>> Shared in ticket #11423:
>> https://unicode.org/cldr/trac/ticket/11423#comment:15
>
>
> ==> for your proposal to be effective, you need to reach out.
>
Basically we vetters are just reporting the locale date. Beyond that, I’ve already conceded a huge effort in reporting bugs in English data and in communicating on lists and fora, including German (since the current survey that has a very limited scope). I have limited time and resources.

Normally reaching out to all relevant locales is what CLDR can do best, by posting guidelines. by e-mailing (on behalf of CLDR administrator and/or on the public CLDR-users Mail List), and by prioritizing the items on the vetters’ dashboards.

If I can do something else, I’m ready but people should not abuse since I’ve many other tasks I won’t be going to deprioritize any longer. At some point I’ll just start reporting to end-users that we’ve strived to get locale data in synch, but that CLDR ended up rolling back our efforts, alleging other priorities. If that is what you wish, I’d say that there’s no problem for me except that I strongly dislike documenting an ugly mess.
>
>>
>>> (2) have a solution that works for lining figures as well as separators.
>>>
>>> (3) have a solution that understands ALL uses of spaces that are narrower than normal space. Once a character exists in Unicode, people will use it on the basis of "closest fit" to make it do (approximately) what they want. Your proposal needs to address any issues that would be caused by reinterpreting a character more narrowly that it has been used. Only by comprehensively identifying ALL uses of comparable spaces in various languages and scripts, you can hope to develop a solution that doesn't simply break all non-French text in favor of supporting French typography.
>>>
>> There is no such problem except that NNBSP has never worked properly in Mongolian. It was an encoding error, and that is the reason why to date, all font developers unanimously request the Mongolian Suffix Connector. That leaves the NNBSP for what it is consistently used outside Mongolian: a non-breakable thin space, kind of a belated avatar 
>> of what PUNCTUATION SPACE should have been since the beginning.
>
> ==> I mentioned before that if something is universally "broken" it can sometimes be resurrected, because even if you change its behavior retroactively, it will not change something that ever worked correctly. (But you need to be sure that nobody repurposed the NNBSP for something useful that is different from what you intend to use it for, otherwise you can't change anything about it).
>
You may wish to look up Unicode’s own PRI#308 background page, where they already hinted they’ve made sure it isn’t.
http://www.unicode.org/review/pri308/pri308-background.html
https://www.unicode.org/review/pri308/
https://www.unicode.org/review/pri308/feedback.html

> If, however, you are merely adding a use for some existing character that does not affect its properties, that is usually not as much of a problem - as long as we can have some confidence that both usages will continue to be possible.
>
Actually, again, there is a problem with NNBSP in Mongolian.

Richard Wordingham reported at thread launch that Unicode have started tweaking that space in a way that makes it unfit for French.

Now since you are aware that this operating mode is wrong, I’d suggest that you reach back to them providing feedback about inappropriateness of last changes. Other people (including me) may do that as well, but I see better chances for your recommendations to get implemented. I say that because lastly I strongly recommended in several pieces of feedback that the math symbols should not be bidi-mirrored on a tilde–reversed-tilde basis, because mirroring these compromises legibility of the tilde symbol in low-end environments relying on glyph-exchange-bidi-mirroring for best-fit display, but UTC took no action, and off-list I was taught that UTC is not interested. Nothing else than that, in private mail. UTC are just not interested, without providing any technical reasons. Perhaps you better understand now why I posted what I suspected to be the reason why UTC is not interested, or was not interested, in supporting a narrow non-breaking space unless Mongolian was encoded and 
needed the same for the purpose of appending suffixes (as opposed to separating vowels, which is performed by a similar space with another shaping behavior, and proper to Mongolian). A hypothesis that you firmly dissipated in the wake, but without answering my question about */why UTC was ignoring the demand for a narrow non-breaking space, delaying support for French and heavily impacting French implementations still today/* due to less font support than if that space were in Unicode from version 1.1 on.
>
>>> Perhaps you see why this issue has languished for so long: getting it right is not a simple matter.
>>>
>> Still it is as simple as not skipping PUNCTUATION SPACE when FIGURE SPACE was made non-breakable. Now we ended up with a mutated Mongolian Space that does not work properly for Mongolian, but does for French and other Latin script using languages. It would even more if TUS was blunter, urging all foundries to update their whole catalogue soon.
>
> ==> You realize that I'm giving you general advice here, not something utterly specific to NNBSP - I don't have the inputs and background to know whether your approach is feasible or perhaps the best possible?
>
It is not “my approach”.

Other List Members may wish to help you answer my questions.
>
> As for PUNCTUATION SPACE - some of the spaces have acquired usage in math (as part of the added math support in Unicode 3.2). We need to be sure that the assumptions about these that may have been made in math typesetting  are not invalidated.
>
That adds to the reasons why I’m asking why PUNCTUATION SPACE was not made non-breakable when FIGURE SPACE was. The math usage has probably originated in repurposing that space on the basis of it’s line breaking behavior. I don’t suggest to make it non-breakable now. That deal was broken and will remain broken. Now we must live with NNBSP and get more font support, while trying to stop Unicode from making a mess of it that neither helps Mongolian nor French nor all (other) locales grouping digits with a narrow space.
>
> Not sure offhand whether UTR#25 captures all of that, but if you ever feel like proposing a property change you MUST research that first (with the current maintainers of that UTR or other experts).
>
I have NOT proposed any property change, and PUNCTUATION SPACE or "2008" are NOT found in UTR #25 (Unicode Support for Mathematics).
>
> This is the way Unicode is different from CLDR.
>
Marcel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://unicode.org/pipermail/unicode/attachments/20190119/7beac313/attachment.html>