Hans Aberg <[log in to unmask]> writes:
> I am not a TeX guru, but I get the impression that the TeX looks like this:
> <string of TeX tokens> <not yet gulped up ASCII (or 8-bit)>
> The string of TeX tokens buffer is normally empty, but sometimes a macro
> may insert a string of tokens (perhaps a macro expansion can be viewed as
> though the body is first inserted in this buffer, before being evaluated).
Yes, that is quite true, but Knuth calls that buffer an input stream,
and there may be multiple nested input streams open at any given moment.
This is what one sees in the multiple levels of error context (visible
when errorcontextlines > 1).
> The <not yet gulped up ASCII (or 8-bit) buffer is read converted into
> tokens at need.
TeX reads into the buffer one line at a time. In particular, the
character at the end of the line will be whatever was the value of
\endlinechar at the point when the read was triggered. But the catcode
of the endlinechar can be changed at any point until TeX takes it from
the line buffer and turns it into a token.
> TeX does not back-track.
\futurelet, \uppercase, \lowercase, \expandafter and perhaps one or two
others do backtracking in the token stream, but yes there is no
backtracking in the line buffer.