At 11:47 -0500 2001/02/13, Michael John Downes wrote:
>> I am not a TeX guru, but I get the impression that the TeX looks like this:
>> <string of TeX tokens> <not yet gulped up ASCII (or 8-bit)>
>> The string of TeX tokens buffer is normally empty, but sometimes a macro
>> may insert a string of tokens (perhaps a macro expansion can be viewed as
>> though the body is first inserted in this buffer, before being evaluated).
>Yes, that is quite true, but Knuth calls that buffer an input stream,
>and there may be multiple nested input streams open at any given moment.
Perhaps I simplified it: It should probably look like
<string of TeX tokens> <current input stream>
where the input streams are stacked (including the stream buffers then). --
I generally skip over this stacking things, because one can easily treat a
set of stacked input streams as a single input stream.
>> The <not yet gulped up ASCII (or 8-bit) buffer is read converted into
>> tokens at need.
>TeX reads into the buffer one line at a time.
How can this be true? What happens if a command in the middle of a line
changes the catcodes, or contains a macro that expands to a \input
>In particular, the
>character at the end of the line will be whatever was the value of
>\endlinechar at the point when the read was triggered. But the catcode
>of the endlinechar can be changed at any point until TeX takes it from
>the line buffer and turns it into a token.
I get the impression that you refer to a technique that TeX is using merely
to buffer the input. -- But in reality this is only a buffering technique,
which does not affect the first one character-by-one and then one token-one
>> TeX does not back-track.
>\futurelet, \uppercase, \lowercase, \expandafter and perhaps one or two
>others do backtracking in the token stream, but yes there is no
>backtracking in the line buffer.
This is not backtracking, really: Backtracking means that one reads ahead,
and later decides that the sequence was wrong based on semantic
evaluations, and then re-parses the sequences. Those commands that you
mention have a deterministic course of action, namely they save away some
tokens, and later put them back in the front of the token stream, just as
any macro can do when expanding.