LATEX-L Archives

Mailing list for the LaTeX3 project


Options: Use Forum View

Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Lars Hellström <[log in to unmask]>
Reply To:
Mailing list for the LaTeX3 project <[log in to unmask]>
Tue, 15 Nov 2005 18:36:41 +0100
text/plain (53 lines)
At 07.25 +0100 2005-11-13, Will Robertson wrote:
>But is there a reason that the stream
>of tokens (or at least the last couple) couldn't, in some
>hypothetical TeX extension, be "remembered" for possible analysis?
>Say there are tokens t1, t2, .... Obviously the node that t2 builds
>can't be queried by t1. My proposal is that after t1 was processed,
>it would be held in \previoustoken (say) while t2 was being
>processed, and so on.
>The token stream won't change retrospectively, so in theory it should
>be able to be examined. Or at least, that's what my thinking was. It
>would seem to be useful for when xTeX is performing calculations, but
>not actually typesetting anything yet, since in this case you don't
>have (useful) previous nodes to compare to anything. I admit I can't
>think of an example off the top of my head.

One problem with this, if one wants to do some linguistic processing in
reasonable amount of work, is that want begins as a simple token sometimes
ends up as a rather complicated construction when typeset (consider for
example the non-composite definitions of \c and \k). The aim should
therefore be to capture the token(s) at roughly the LICR stage, not when
they are being consumed by the stomach.

One way to do this (as we right now allow ourselves to invent new
primitives) could be to have a command, say,

  \semanticunit{<nominal material>}{<actual material>}

which would behave as <actual material> (typesetting it and such), but
append the <nominal material> to some list of "previous tokens" that can be
examined afterwards. An example:

  \semanticunit{\"{a}}{\accent127 a} % Don't recall if it's 127 in
                                     % OT1 right now, so don't worry.

Some notes:

Ordinary tokens would behave as a \semanticunit with themselves in both
arguments, so that these get inserted properly.

\semanticunit's should be nestable, so that whatever is appended inside the
<actual material> should get stripped off again when we're done executing

If \semanticunit is unexpandable, but handled in the "inner loop", then
this could provide a neat way to handle various protection issues (TeX
expanding ahead too long): Just protect anything fragile by putting inside
one of these things. (e-TeX probably provides an alternative mechanism for
this, but it's anyway an application.)

Lars Hellström