Feedback on the proposal to change U+FFFD generation when decoding ill-formed UTF-8

Alastair Houghton via Unicode unicode at
Thu Jun 1 03:13:33 CDT 2017

On 31 May 2017, at 20:42, Shawn Steele via Unicode <unicode at> wrote:
>> And *that* is what the specification says.  The whole problem here is that someone elevated
>> one choice to the status of “best practice”, and it’s a choice that some of us don’t think *should*
>> be considered best practice.
>> Perhaps “best practice” should simply be altered to say that you *clearly document* your behavior
>> in the case of invalid UTF-8 sequences, and that code should not rely on the number of U+FFFDs 
>> generated, rather than suggesting a behaviour?
> That's what I've been suggesting.
> I think we could maybe go a little further though:
> * Best practice is clearly not to depend on the # of U+FFFDs generated by another component/app.  Clearly that can't be relied upon, so I think everyone can agree with that.
> * I think encouraging documentation of behavior is cool, though there are probably low priority bugs and people don't like to read the docs in that detail, so I wouldn't expect very much from that.
> * As far as I can tell, there are two (maybe three) sane approaches to this problem:
> 	* Either a "maximal" emission of one U+FFFD for every byte that exists outside of a good sequence 
> 	* Or a "minimal" version that presumes the lead byte was counting trail bytes correctly even if the resulting sequence was invalid.  In that case just use one U+FFFD.
> 	* And (maybe, I haven't heard folks arguing for this one) emit one U+FFFD at the first garbage byte and then ignore the input until valid data starts showing up again.  (So you could have 1 U+FFFD for a string of a hundred garbage bytes as long as there weren't any valid sequences within that group).
> * I'd be happy if the best practice encouraged one of those two (or maybe three) approaches.  I think an approach that called rand() to see how many U+FFFDs to emit when it encountered bad data is fair to discourage.


Kind regards,



More information about the Unicode mailing list