[SG16] Draft proposal: Clarify guidance for use of a BOM as a UTF-8 encoding signature

[Tom Honermann]

On 10/12/20 4:54 PM, Shawn Steele wrote:
>
> I’m having trouble with the attempt to be this prescriptive.
>
> These make sense:  “Use Unicode!”
>
>   * If possible, mandate use of UTF-8 without a BOM; diagnose the
>     presence of a BOM in consumed text as an error, and produce text
>     without a BOM.
>   * Alternatively, swallow the BOM if present.
>
> After that the situation is clearly hopeless.  Applications should Use 
> Unicode, eg: UTF-8, and clearly there are cases happening where that 
> isn’t happening.  Trying to prescribe that negotiation should 
> therefore happen, or that BOMs should be interpreted or whatever is 
> fairly meaningless at that point.  Given that the higher-order 
> guidance of “Use Unicode” has already been ignored, at this point it’s 
> garbage-in, garbage-out.  Clearly the app/whatever is ignoring the 
> “use unicode” guidance for some legacy reason.  If they could adapt, 
> it should be to use UTF-8.   It **might** be helpful to say something 
> about a BOM likely indicating UTF-8 text in otherwise unspecified 
> data, but prescriptive stuff is pointless, it’s legacy stuff that 
> behaves in a legacy fashion for a reason and saying they should have 
> done it differently 20 years ago isn’t going to help 😊
>
There are applications that, for legacy reasons, are unable to change 
their default encoding to UTF-8, but that also need to handle UTF-8 
text.  It is not clear to me that such situations are hopeless or that 
they cannot be improved.

The prescription offered follows what you suggest.  The first three 
cases are are all of the "use Unicode!" variety.  The distinction 
between the third and the fourth is to relegate use of a BOM as an 
encoding signature to the last resort option.  The intent is to make it 
clear, with stronger motivation than is currently present in the Unicode 
standard, that use of a BOM in UTF-8 is not a best practice today.

Tom.

> -Shawn
>
> *From:* Unicode <unicode-bounces at unicode.org> *On Behalf Of *Tom 
> Honermann via Unicode
> *Sent:* Monday, October 12, 2020 7:03 AM
> *To:* Alisdair Meredith <alisdairm at me.com>
> *Cc:* sg16 at lists.isocpp.org; Unicode List <unicode at unicode.org>
> *Subject:* Re: [SG16] Draft proposal: Clarify guidance for use of a 
> BOM as a UTF-8 encoding signature
>
> Great, here is the change I'm making to address this:
>
>     Protocol designers:
>
>       * If possible, mandate use of UTF-8 without a BOM; diagnose the
>         presence of a BOM in consumed text as an error, and produce
>         text without a BOM.
>       * Otherwise, if possible, mandate use of UTF-8 with or without a
>         BOM; accept and discard a BOM in consumed text, and produce
>         text without a BOM.
>       * Otherwise, if possible, use UTF-8 as the default encoding with
>         use of other encodings negotiated using information other than
>         a BOM; accept and discard a BOM in consumed text, and produce
>         text without a BOM.
>       * Otherwise, require the presence of a BOM to differentiate
>         UTF-8 encoded text in both consumed and produced text*unless
>         the absence of a BOM would result in the text being
>         interpreted as an ASCII-based encoding and the UTF-8 text
>         contains no non-ASCII characters (the exception is intended to
>         avoid the addition of a BOM to ASCII text thus rendering such
>         text as non-ASCII)*. This approach should be reserved for
>         scenarios in which UTF-8 cannot be adopted as a default due to
>         backward compatibility concerns.
>
> Tom.
>
> On 10/12/20 8:40 AM, Alisdair Meredith wrote:
>
>     That addresses my main concern.  Essentially, best practice (for
>     UTF-8) would be no BOM unless the document contains code points
>     that require multiple code units to express.
>
>     AlisdairM
>
>
>
>         On Oct 11, 2020, at 23:22, Tom Honermann <tom at honermann.net
>         <mailto:tom at honermann.net>> wrote:
>
>         On 10/10/20 7:58 PM, Alisdair Meredith via SG16 wrote:
>
>             One concern I have, that might lead into rationale for the
>             current discouragement,
>
>             is that I would hate to see a best practice that pushes a
>             BOM into ASCII files.
>
>             One of the nice properties of UTF-8 is that a valid ASCII
>             file (still very common) is
>
>             also a valid UTF-8 file.  Changing best practice would
>             encourage updating those
>
>             files to be no longer ASCII.
>
>         Thanks, Alisdair.  I think that concern is implicitly
>         addressed by the suggested resolutions, but perhaps that can
>         be made more clear.  One possibility would be to modify the
>         "protocol designer" guidelines to address the case where a
>         protocol's default encoding is ASCII based and to specify that
>         a BOM is only required for UTF-8 text that contains non-ASCII
>         characters.  Would that be helpful?
>
>         Tom.
>
>             AlisdairM
>
>
>
>                 On Oct 10, 2020, at 14:54, Tom Honermann via SG16
>                 <sg16 at lists.isocpp.org <mailto:sg16 at lists.isocpp.org>>
>                 wrote:
>
>                 Attached is a draft proposal for the Unicode standard
>                 that intends to clarify the current recommendation
>                 regarding use of a BOM in UTF-8 text.  This is follow
>                 up to discussion on the Unicode mailing list
>                 <https://corp.unicode.org/pipermail/unicode/2020-June/008713.html>
>                 back in June.
>
>                 Feedback is welcome.  I plan to submit
>                 <https://www.unicode.org/pending/docsubmit.html> this
>                 to the UTC in a week or so pending review feedback.
>
>                 Tom.
>
>                 <Unicode-BOM-guidance.pdf>--
>                 SG16 mailing list
>                 SG16 at lists.isocpp.org <mailto:SG16 at lists.isocpp.org>
>                 https://lists.isocpp.org/mailman/listinfo.cgi/sg16
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://corp.unicode.org/pipermail/unicode/attachments/20201013/a878666b/attachment.htm>