Print

Print


At 11:47 -0500 2001/02/13, Michael John Downes wrote:
>> I am not a TeX guru, but I get the impression that the TeX looks like this:
>>   <string of TeX tokens> <not yet gulped up ASCII (or 8-bit)>
>> The string of TeX tokens buffer is normally empty, but sometimes a macro
>> may insert a string of tokens (perhaps a macro expansion can be viewed as
>> though the  body is first inserted in this buffer, before being evaluated).
>
>Yes, that is quite true, but Knuth calls that buffer an input stream,
>and there may be multiple nested input streams open at any given moment.

Perhaps I simplified it: It should probably look like
   <string of TeX tokens> <current input stream>
where the input streams are stacked (including the stream buffers then). --
I generally skip over this stacking things, because one can easily treat a
set of stacked input streams as a single input stream.

>> The <not yet gulped up ASCII (or 8-bit) buffer is read converted into
>> tokens at need.
>
>TeX reads into the buffer one line at a time.

How can this be true? What happens if a command in the middle of a line
changes the catcodes, or contains a macro that expands to a \input
<filename>?

>In particular, the
>character at the end of the line will be whatever was the value of
>\endlinechar at the point when the read was triggered. But the catcode
>of the endlinechar can be changed at any point until TeX takes it from
>the line buffer and turns it into a token.

I get the impression that you refer to a technique that TeX is using merely
to buffer the input. -- But in reality this is only a buffering technique,
which does not affect the first one character-by-one and then one token-one
lookahead technique.

>> TeX does not back-track.
>
>\futurelet, \uppercase, \lowercase, \expandafter and perhaps one or two
>others do backtracking in the token stream, but yes there is no
>backtracking in the line buffer.

This is not backtracking, really: Backtracking means that one reads ahead,
and later decides that the sequence was wrong based on semantic
evaluations, and then re-parses the sequences. Those commands that you
mention have a deterministic course of action, namely they save away some
tokens, and later put them back in the front of the token stream, just as
any macro can do when expanding.

  Hans Aberg