Re: State of ECMA-48 in a Unicode age (was Re: “plain text styling”…)

[Kent Karlsson]

I will likely not have a response to all the comments below. But just two things:

1) In the proposal I wrote, all the so-called modes are deprecated. As far as I know, none were ever implemented anywhere, and all were a bad idea.

2) ”most terminal emulators (which accept bare LF as a full newline in their default modes”
I don’t think that is true. Firstly we have the ”cooked” vs. ”raw” modes (when ”echo mode” is on) in Unix/Linux ttys; that is a setting for the tty, not the terminal emulator. Then there are such things as shell command line tools (like bash) which does their own ”echoing” (tty echo off), and does so in quite  complicated ways, allowing for editing in the command line, as well as command history. But all that is out of scope for ECMA-48 except that ECMA-48 control sequences (not the styling ones, but other control sequences for ”screen editing”) are used to implement their behaviour.

Kind regards
/Kent K

> 8 jan. 2023 kl. 22:45 skrev Harriet Riddle via Unicode <unicode at corp.unicode.org>:
> 
> Kent Karlsson via Unicode wrote:
>> Well, yes... But the problem is that, IIUC, the ECMA-48 committee is currently the empty set of people…
>> 
>> /K
> 
> ---
> 
> Tangentially to this, I do much believe that a new edition of ECMA-48 which clarifies and addresses relationship both to Unicode and to established convention would be of practical benefit, with the following points standing out to me:
> 
> ① The penultimate edition of ECMA-48 (fourth edition, December 1986, still archived at the bottom of ECMA's page for ECMA-48) deprecates a number of mode flags and control functions in Appendix E (pages 84–87).  Notable amongst these deprecations is LF/NLM (E.1.3, bottom of page 84 / top of page 85), i.e. the mode flag that toggles whether linefeeds imply a carriage return.  The note (E.1) attached to that section explains the deprecation, stating essentially that full line breaks should use either NEL or CR+LF moving forward; the IND control for explicit bare linefeed is also in the deprecated features appendix as E.2.3.  In the fifth and current edition (June 1991), per the 1998 reprint available as PDF from ECMA's page, both LF/NLM and IND were removed altogether, announced in annex F.5.2 (page numbered 88 / 102nd PDF page) and annex F.8.2 (page numbered 89 / 103rd PDF page) respectively.  The definition of LF (section 8.3.74, page numbered 49 / 63rd PDF page) unambiguously specifies a move to the "corresponding character position" (as opposed to the start) of the following line.  Therefore, most terminal emulators (which accept bare LF as a full newline in their default modes) are actually in violation of the current edition of ECMA-48 (and exhibit deprecated but permitted behaviour per the edition before), as are virtually all modern text editors, for example.  Honestly, the elimination of the LF/NLM mode comes across as wishful thinking on the part of the committee, but hindsight is 20:20.  A mode such as LF/NLM probably ought to be restored so as to align the standard with the by-now-set-in-stone reality.
> 
> ② Speaking of ECMA-48 and the CR vs LF vs CRLF vs NEL vs LSEP issue, better coördination between UAX 14 and ECMA-48 might be in order.  This doesn't cause as much of an issue in practice, since the contexts where ECMA-48 is actually implemented (monospaced terminal emulators) are largely disjoint with these where UAX 14 is implemented, but it should be clearer how an implementation can be concordant with both (for example, whether an ECMA-48 conformant implementation of CR or VT is sufficient to count as a line break for UAX 14 purposes).  This is particularly relevant should one wish to use ECMA-48 in a non-terminal context, as seems to be part of the present discussion.
> 
> ③ Section 5 needs reworking to address how it interacts with Unicode Transformation Formats (other than the erstwhile abortive UTF-1, which it works fine with, for all this has any effect on anything).  The representation of the C0 and C1 codes is given in 7-bit and 8-bit column/line bit combinations.  I believe ISO/IEC 10646 briefly addresses how these translate to UTF-16 or UTF-32 (padding to code unit width), but this would be ideal to have addressed in ECMA-48 itself in this day and age; furthermore, even with that provision, "bit combinations from 08/00 to 09/15" in the context of UTF-8 arguably prescribes fragmentary or invalid UTF-8 sequences rather than the UTF-8 representations of the C1 code points.
> 
> ④ Also in section 5: command strings (DCS, OSC, PM and APC) are limited to 0x08–0D (the ASCII FEx format effectors) and 0x20–7E (the ASCII printing characters including space).  This is contrasted with character strings (SOS) which have no such restriction, with only SOS itself being forbidden (and ST not includable due to being the terminator).  In practice, not only ASCII printing characters but arbitrary Unicode characters—other than Cc control codes outside of the aforementioned 0x08–0D range—are permitted in OSC sequences recognised by terminal emulators, which often contain text.  For example, "\u{9D}0;flambé\u{9C}" will set a terminal window title to "flambé", even though "é" is not an ASCII character.  This is another area which probably needs updating to align it with both industry practice and a Unicode age.
> 
> ⑤ The characters listed as affected by the FEAM mode in section 7.2.5 needs looking at—for instance, it lists BPH (equivalent to ZWSP) but not its opposite NBH (equivalent to WJ).  It also lists CR and NEL but not LF, all of which are format effectors per section 8.2.4.  The interaction with Unicode general categories should also be addressed: presumably it would apply to the format category (Cf), and possibly also Zl and Zp, in addition to the specific listed Cc characters and CSI sequences, but this should be addressed in the FEAM definition, annex A.1, or both.
> 
> ⑥ Speaking of annex A, annex A.2 and the GCC sequence might deserve addressing as to their relation to Unicode.  Certainly, ECMA-43 (conformed to by ISO 8859—and yes, the graphical resemblence between "ECMA-43" and "ECMA-48" can be confusing whenever these two standards are discussed together) puts significant limitations on the ECMA-48 codes used for composition, prohibiting any such composition that creates a new character rather than merely a ligature of existing characters (see annex C of ECMA-43, contrast with annex A.2 of ECMA-48).  This both bans backspace composition, and constrains the use of GCC to discretionary ligatures (which is not explicitly constrained by ECMA-48 itself—indeed, annex A seems to prescribe GCC as a migration path from backspace composition—although the note on the definition of GCC itself in section 8.3.54 mentions CJK square ligatures as the simple case, not diacritic composition or APL composition).  Backspace composition is similarly not really compatible with the Unicode model of base characters, combining characters and pre-composed diacritic-bearing characters (and composed-symbol APL operators without decompositions), although discretionary ligatures are manifestly compatible with the Unicode character model (see e.g. the OpenType dlig feature and the CSS font-variant-ligatures property).
> 
> --Har.
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://corp.unicode.org/pipermail/unicode/attachments/20230108/5981b809/attachment.htm>