Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

[Egmont Koblinger via Unicode]

Hi Philippe,

> I have never said anything about your work because I don't know where you spoke about it or where you made some proposals. I must have missed one of your messages (did it reach this list?).

This entire conversation started by me announcing here my work, aiming
to bring usable BiDi to terminal emulators.

> Terminals are not displaying plain text, they create their own upper layer protocol which requires and enforces the 2D layout [...] Bidi does not specify the 2D layout completely, it is purely 1D and speaks about left and right direction

That's one of the reasons why it's not as simple as "let's just run
the UBA inside the terminal", one of the reasons why gluing the two
worlds together requires a substantial amount of design work.

> For now terminal protocols, and emulators trying to implement them; that must mix the desynchronized input and output (especially when they have to do "local echo" of the input [...]

I assume by "local echo" you're talking about the Send/Receive Mode
(SRM) of terminals, and not the "stty echo" line discipline setting of
the kernel, because as far as the terminal emulator is concerned, the
kernel is already remote, and it's utterly irrelevant for us whether
it's the kernel or the application sending back the character.

SRM is only supported by a few terminal emulators, and we're about to
drop it from VTE, too (https://gitlab.gnome.org/GNOME/vte/issues/69).

> If you look at historic "terminal" protocols,

I'm mostly interested in the present and future. In the past, only for
curiosity, and to the extent necessary to understand the present and
to plan for the future.

> Some older terminal protocols for mainframes notably were better than today's VT-like protocols: you did not transmit just what would be displayed, but you also described the screen area where user input is allowed and the position of fields and navigation between them:

This is not seen in today's graphical terminal emulators.

> Today these links are better used with real protocols made for 2D and allowing an web application to mange the input with presentation layer (HTML) and with javascript helpers (that avoid the roundtrip time).

Sure, if you need another tool, let's say a dynamic webpage in your
browser, rather than a terminal emulator to perform your taks
effectively, so be it. I'm not claiming terminal emulators are great
for everything, I'm not claiming terminal emulators should be used for
everything.

> But basic text terminals have never evolved and have lagged behind today's need.

I disagree with the former part. There are quite a few terminal
emulators out there, and many have added plenty of new great features
recently.

Whether they're up to today's needs, depends on what your needs are.
If you need something utterly different, go ahead and use whatever
that is, such as maybe a web browser. If you're good with terminals,
that's fine too. And there's a slim area where terminal emulators are
mostly good for you, you'd just need a tiny little bit more from them.
And maybe for some people this tiny little bit more happens to be
BiDi.

> Most of them were never tested for internationalization needs:

Terminal emulators weren't created with internationalization in mind.
I18n goals are added one by one. Nowadays combining accents and CJK
are supported by most emulators. Time to stretch it further with BiDi,
shaping, spacing combining marks for Devanagari, etc.

> [...] delimit input fields in input forms for mainframes, something that was completely forgotten and remains forgotten today with today's VT-* protocols, to indicate which side of the communcation link controls the content of specific areas

Something that was completely forgotten, probably for good reasons,
and I don't see why it should be brought back.

> As well today's VT-* protocols have no possibility to be scriptable: implemeint a way to transport fragments of javascripts would be fine.

I have absolutely no incentive to work in this direction.

> Text-only terminals are now aging but no longer needed for user-friendly interaction, they are used for technical needs where the only need is to be able to render static documents without interactiving with it, except scrolling it down, and only if they provide help in the user's language.

Text-only terminals are no longer needed??? Well, strictly speaking,
computers aren't needed either, people lived absolutely fine lives
before they were invented :)

If you get to do some work, depending on the kind of work, terminal
emulators may or may not be a necessary or a useful tool for you. For
certain tasks you don't really have anything else, or at least
terminals are way more effective than other approaches. For other
tasks (e.g. text editing) it's mostly a matter of taste whether you
use a terminal or a graphical app. For yet other tasks, terminal
emulators take you nowhere.

My work aims to bring BiDi into terminal emulators in a reasonably
well designed way, rather than the ad-hoc and pretty broken ways some
emulators have already attempted this. If this is what you were
looking for (as many people are), good for you. If you don't care
about it, because let's say you'd rather use other tools to get your
BiDi work done, so be it, that's also fine.

> Why not imagine a text -terminal with attributes deliminting a rectangular area containing an object in which rich-text (HTML or other) will be rendered and controled by the 2D/3D engine and left unmanaged internally by the terminal protocols?

Because I'm not redesigning the essentials of terminal emulators, just
bringing BiDi into whatever these terminal emulators already are.

Because probably all the terminal emulators are developed by a few
enthusiasts as a hobby in their pretty limited free time, so they go
for what is reasonable to implement, is likely to be used by
applications, and what they think makes sense.

> Old terminal protocols based on a single regular grid of equal cells are definitely not user friendly and not suitable for all international text (only a few scripts allow readable monospaced fonts).

Terminals are, in many aspects, not user friendly. The best you can
tell about them is that they're poweruser friendly and developer
friendly.

How much they are suitable for international text depends on how much
compromise someone is willing to make, e.g. whether they're ready to
accept monospace fonts for their language's script. If nothing else,
my proposal at least makes terminals usable for Hebrew. However, in
plenty of terminal emulator's bugtracker there's a request for BiDi,
they usually demonstrate it with Arabic, and show examples of other
terminal emulators that do some BiDi as reference. This means that for
most people requesting BiDi, having monospace fonts for Arabic (plus
shaping I assume) is apparently a good enough compromise.

> Today we need more flexibility.

Sure. You get it outside of terminal emulators. Or you can start
changing terminal emulators to accomodate to the new needs, put all
the work in that (or hire someone to do it), and see where it goes.

I, for one, am not to the slightest bit interested in abandoning the
character grid and allowing for proportional fonts. This would just
break a gazillion of things. Nor am I interested in reviving local
echo, introducing rectangle areas where data can be typed into; nor
making emulators scriptable by an app sending e.g. JavaScript code to
them.

What I am interested in, for whatever reason, is bringing BiDi into
the existing world of terminal emulators.

> But it will be difficult to readapt these old protocols to support the necessary extensions and remain compatible with applications still using them.

You see it as difficult, I see it as a challenge that requires buy-in
from so many parties and modification to so many software that I don't
see it viable.

To summarize: Terminal emulators currently have a strict character
grid model, and tons of other pecularities, limitations and legacy to
live with (and features that have practically died over the decades).
What I do is bring BiDi into this world (in its current state) with as
little modifications to the basics of terminal emulation as absolutely
necessary. If you have ideas for other small, incremental changes, I'm
curious to hear them! If you'd like to see way more substantial
changes to the very core of terminal emulation, like proportional
fonts, JS code downloaded to the terminal etc., I'm not the right guy
to talk to; thanks for your understanding!


cheers,
egmont