Hans Aberg <[log in to unmask]> writes:
> >> The <not yet gulped up ASCII (or 8-bit) buffer is read converted into
> >> tokens at need.
> >TeX reads into the buffer one line at a time.
> How can this be true? What happens if a command in the middle of a line
> changes the catcodes, or contains a macro that expands to a \input
Sorry, I didn't use the terminology very well. TeX input first goes into
a string buffer, one line at a time. This string buffer is the only
place where TeX deals with ASCII chars as input; all other "input
streams" are streams of tokens. Tokenization occurs by scanning
substrings from this string buffer and adding the corresponding token to
the current input stream (which if we call it a "buffer", is a different
buffer, not the one that contains simple 8-bit characters as first read
from a file).
If you get an error "TeX capacity exceeded: buffer size" it means
that a line of the input file was too long to be read into the string