Bidi paragraph direction in terminal emulators (was: Proposal for BiDi in terminal emulators)

[Egmont Koblinger via Unicode]

Hi Eli,

> Not sure why.  There are terminal emulators out there which support
> proportional fonts.

Well, of course, a terminal emulator can load any font, even
proportional, but as it places them in the grid, it will look ugly as
hell (like this one: https://askubuntu.com/q/781327/398785 ). Sure you
could apply some tricks to make it look a bit less terrible (e.g. by
centering each glyph in its cell rather than aligning to the left),
but it still won't look great.

In the world of terminal emulation, many applications expect things to
align properly according to the wcwidth() of the string they emit. You
abandon this (start placing the glyphs one after the other in a row,
no matter how wide they are), and plenty of applications suddenly fall
apart big time (let alone questions like how you define the terminal's
width in characters).

> Emacs is perhaps the only one whose terminal
> emulator currently supports bidi more or less in full

Let's not get started from here, please.

In Emacs-25.2's terminal emulator I executed "cat TUTORIAL.he". For
the entire contents, LTR paragraph direction was used and was aligned
to the left. Maybe something has changed for 26.x, I don't know.

In my work I carefully evaluated 4 other "BiDi-aware" terminal
emulators, as well an ancient specification for BiDi which I had to
read about twenty times to get to pretty much understand what it's
talking about. Identified substantial issues with both the standard as
well as all the independent implementations (which didn't care about
this standard at all). I show that existing terminal emulators are
incompatible to the extent that an app cannot reliably print any RTL
text by any means at all. At this point I firmly believe it should be
clear that BiDi in terminals is not a topic where one can just go
ahead and do something, without having a specification first. I lay
down principles which a proper BiDi-supporting platform I believe
needs to meet, argue why multiple modes (explicit and implicit) are
inevitable, examine what to do with paragraph direction, cursor
location and tons of other issues, and come up with concrete
suggestion how (partially based on that ancient specifications) these
all should be exactly addressed.

Then, after putting literally months of work in it, I come here to
announce my work and ask for feedback. So far, from a thread of 100+
mails, I take away two pieces of worthful feedback: one is that
shaping should be done differently, and the other one is that – for
some use cases – a bigger scope of data should be used for
autodetecting the "paragraph direction" (as per UBA's terminology).

And now you suddenly tell that Emacs's terminal supports BiDi more or
less in full???

Sorry, I just don't buy it. If you retain this claim, I'd pretty
please like to see a specification of its behavior, one which
addresses at least all the major the issues I address in my work, one
which I could replace my work with, one which I'd be happy to
implement in gnome-terminal in the solid belief that it's about as
good as my proposal, and would wholeheartedly recommend for other
terminal emulators to adopt.

Or maybe, by any chance, when you said Emacs's terminal supported BiDi
more or less in full, did you perhaps went with your own idea what a
BiDi-aware terminal emulator needs to support; ignoring all those
things I detail in my work, such as the inevitable need for explicit
mode, the need for deciding the scope of implicit vs. explicit mode,
and much more?


thanks a lot,
egmont