Feedback on the proposal to change U+FFFD generation when decoding ill-formed UTF-8

Shawn Steele via Unicode unicode at unicode.org
Tue May 30 12:11:40 CDT 2017


> Which is to completely reverse the current recommendation in Unicode 9.0. While I agree that this might help you fending off a bug report, it would create chances for bug reports for Ruby, Python3, many if not all Web browsers,...

& Windows & .Net

Changing the behavior of the Windows / .Net SDK is a non-starter.

> Essentially, "overlong" is a word like "dragon" or "ghost": Everybody knows what it means, but everybody knows they don't exist.

Yes, this is trying to improve the language for a scenario that CANNOT HAPPEN.  We're trying to optimize a case for data that implementations should never encounter.  It is sort of exactly like optimizing for the case where your data input is actually a dragon and not UTF-8 text.  

Since it is illegal, then the "at least 1 FFFD but as many as you want to emit (or just fail)" is fine.

-Shawn



More information about the Unicode mailing list