Feedback on the proposal to change U+FFFD generation when decoding ill-formed UTF-8

[Asmus Freytag via Unicode]

On 6/1/2017 2:32 AM, Henri Sivonen via Unicode wrote:
> O
> On Wed, May 31, 2017 at 10:38 PM, Doug Ewell via Unicode
> <unicode at unicode.org> wrote:
>> Henri Sivonen wrote:
>>
>>> If anything, I hope this thread results in the establishment of a
>>> requirement for proposals to come with proper research about what
>>> multiple prominent implementations to about the subject matter of a
>>> proposal concerning changes to text about implementation behavior.
>> Considering that several folks have objected that the U+FFFD
>> recommendation is perceived as having the weight of a requirement, I
>> think adding Henri's good advice above as a "requirement" seems
>> heavy-handed. Who will judge how much research qualifies as "proper"?

I agree with Henri on these general points:

1) Requiring extensive research on implementation practice is crucial in 
dealing with any changes to long standing definitions, algorithms, 
properties and recommendations.
2) Not having a perfect definition of what "extensive" means is not an 
excuse to do nothing.
3) Evaluating only the proposer's implementation (or only ICU) is not 
sufficient.
4) Changing a recommendation that many implementers (or worse, an 
implementers' collective) have chosen to adopt is a breaking change.
5) Breaking changes to fundamental algorithms require extraordinarily 
strong justification including, but not limited to "proof" that the 
existing definition/recommendation is not workable or presents grave 
security risks that cannot be mitigated any other way.

I continue to see a disturbing lack of appreciation of these issues in 
some of the replies to this discussion (and some past decisions by the UTC).

A./