Joseph Wright writes:
> Andreas Matthias wrote:
> > I disagree. I don't like it at all. Characters and tokens are
> > distinct things in terms of TeX.
> > LaTeX3 is already far away from TeX. Maybe too far. Don't push
> > it even farther away by reinventing terms and definitions.
> > "Token list pointer" might not be the most appropriate name but,
> > please, don't call it characters.
> I think the point here was that a toks is a list of tokens, and so is a
> tlp, but they are not the same. I think we are all happy with _toks for
> what TeX calls a toks. So the question is how to name/describe a macro
> used to store tokens (which is currently a tlp). I'd have called it a
> string, but I can see that this causes the same problem as calling the
> contents characters. How about a "token macro" (= _tmac)? (I see that
> l3toks describes a toks as a "token register".) Perhaps this points up
> the underlying structure better?
I'm coming somewhat late to this discussion, but ...
TeX (and as such expl3) deals with tokens and the replacement text of a macro
without arguments is a list of tokens and not a list of characters or a string
the contents of a toks register too is a list of tokens
they are not the same in the way they can be accessed or manipulated but their
content is the same. So I strongly disagree with Will's initial statement
> The problem that I'm trying to solve is what to call the contents of
> these two distinct beasts.
(but i guess in a later mail he disagreed with himself too :-)
instead we have:
- tlists = lists of tokens
and a bunch of functions to manipulate them directly in the input stream
(they got added fearly recently and I think they need to be looked at once
- tlp = a storage bin that holds tlists
and a bunch of functions that operate on bin and or its content
- toks = a register that also holds tlists
but allows a number of special processing (provided by the underlying TeX
leaving out the name question for a minute ...
a) is the concept of the tlp bins useful (separating them from
b) do we need both tlp and toks?
I think the answer to both questions is yes. We do need the additional
functions provided by toks registers only but we also need the power of simple
tlp bins all over the place and calling them a bin for tlists rather than
using macros / functions without argument for this purpose (even if internally
this is what they end up being) is useful.
Coming back to the names:
- I personally don't like the idea of token macro (i would rather get away
from using "macro" in this context). On the other hand it isn't really
- token list pointer may not be the best but it is "fairly accurate" and it
is short :-) and I started to like them
- of course one could go _tlreg and _tlmac or _treg and _tmag instead of
_toks and _tlp but I don't think this would gain anything. In fact I think
it is not going to help.
- _toks are really only there to provide the functionality that TeX offers
for toks registers so keeping a close link to that isn't A Bad Thing
- with tlp's which are conceptually a concept only within expl3 one could
choose a different name, but I don't really see the benefit
so in short I personally think the names are nice enough, what would be need
to work out better in documentation is that both _toks and _tlp hold tlists as
their content but offer different manipulation possibilities and since you
can't easily provide the "toks" ones within the tlp family the extra type is
kept. Perhaps it should be pointed out that one should restrict the use of
_toks to internal situations where this processing is needed