I'm glad to have provoked such a level of response; and obviously
this discussion has happened several times in the past.
On 10/11/2005, at 10pm, Bernd Raichle wrote:
> a) TeX is reading _characters_ from files/stdin which get converted
> to _tokens_.
> b) When these tokens are dealt with by TeX's main control loop where
> the typesetting is done, the _tokens_ are triggering the building
> of _nodes_.
> - If you want to look backward, you access _nodes_.
Thanks for this explanation, it certainly helped me think about the
details properly. What isn't so clear to me is why, in principle,
previously processed tokens that have already triggered nodes cannot
Obviously, in eTeX they can't. But is there a reason that the stream
of tokens (or at least the last couple) couldn't, in some
hypothetical TeX extension, be "remembered" for possible analysis?
Say there are tokens t1, t2, .... Obviously the node that t2 builds
can't be queried by t1. My proposal is that after t1 was processed,
it would be held in \previoustoken (say) while t2 was being
processed, and so on.
The token stream won't change retrospectively, so in theory it should
be able to be examined. Or at least, that's what my thinking was. It
would seem to be useful for when xTeX is performing calculations, but
not actually typesetting anything yet, since in this case you don't
have (useful) previous nodes to compare to anything. I admit I can't
think of an example off the top of my head.