Proposal for BiDi in terminal emulators

[Egmont Koblinger via Unicode]

Hi Richard,

On Sun, Feb 3, 2019 at 2:32 AM Richard Wordingham via Unicode
<unicode at unicode.org> wrote:

> That first reference doesn't even use the word 'visual'.

The Unicode BiDi algorithm does speak about "visual positions for
display", "reordering for display" etc.

> All I am saying is that your proposal should define what it means by
> visual order.

Are you nitpicking on me not giving a precise definition on the
otherwise IMO freaking obvious "visual order", or am I missing
something fundamental?

> Shaping for RTL scripts happens on strings stored in logical order.

That's what I recommend in my current v0.1, which was vetoed by you
guys, claming that the terminal emulator should do it even in cases
when it's only aware of the visual order.

> Passing text in the form of characters in left-to-right order is an
> annoying distraction, presumably forced on you by the attempt to
> maximise compatibility with existing systems.

Nope; passing text in visual order(*) is a technical necessity for
Emacs (as Eli confirmed it) and all other fullscreen apps (text
editors and such), as I provide a detailed proof for that in my
proposal. It's literally impossible to perform visual cropping on a
string (required by practically all fullscreen text editors), visual
concatenation of strings (e.g. a line of tmux which has two panes next
to each other), and in the mean time preserve the logical order that's
passed on. You just can't define a logical order after visual
operations.

(*) To be pedantic, they could pass the text in whatever order they
want to, with random cursor movements in between. The point is that
the terminal emulator won't reshuffle the cells, that is, they should
write into column 1 whichever they want to appear at the leftmost
position, into column 2 whichever they want to appear in column 2, and
so on. And unless the cursor is moved explicitly, the cursor keeps
moving forward to higher numbered columns, that is, the terminal
expects to receive visual order.

> Casting text into grids of 'characters' requires consideration of all
> types of writing elements.  The division into panes is an awkward
> complication; panes in the application not shared with the terminal is
> even worse for shaping.

I'm really not sure what you're trying to say here.

The feeling I get, and I'm happy if you can prove me wrong, is that
while you're truly knowledgeable about shaping, you haven't yet
understood the very fundamentals why terminals are vastly different
from let's say web browsers, which results in the technical necessity
of often relying on visual order. There's even a separate section
dedicated to explaining this in my spec. If terminals weren't vastly
different, BiDi there would've been solved along with the birth of the
Unicode BiDi algorithm, I wouldn't have spent months working on this
proposal, and we wouldn't be having this discussion right now :)

Remember, this whole story is about finding a compromise between what
a terminal emulator is, and what BiDi scripts require (incl. shaping).
If you want to do BiDi and shaping without compromises, you should get
away from terminal emulators (as Kent has also suggested). Having a
strict grid of characters is such a compromise. The terminal emulator
not being aware of the entire logical string, only the currently
onscreen bits (that is, a cropped version of the string), which
results in the need for the explicit mode (visual order) is another
such compromise.


cheers,
egmont