hans aberg writes
There appears to be two variations, one based on the original TeX, and one
with TeX having some kind of extensions.
As for the second approach, it seems me that the internal representation
should be 32-bit Unicode. As TeX does not seem well equipped handling the
encoding issues, one should then hook up a preprocessor providing the
suitable translations. Thus
whatever encoding -> preprocessor -> UTeX
This easy-to-write preprocessor can combine combining characters to single
Unicode characters, if possible, or otherwise write them on a form that
UTeX easily can handle, say by switching from postfix to prefix notation,
or whatever. With further tweaking of the TeX engine it could even combine
TeX combinations such as "--", "---" into single Unicode characters.
while this would obviously work for text in natural languages,
unicode will never contain all the possible "embellished" letters
and symbols used in math. (and this may include instances with two
or even more diacritics on a single letter or symbol.) this set,
while not infinite, is much too large to want to address even using
the unicode private area. but for latex (or any successor) to be
useful for the particular content for which tex was first developed,
this has to be taken into account.