On Tue, 30 Dec 2008 21:59:04 +0100, Frank Mittelbach wrote:

>  c) the other important difference is how you get something out of the  
> bin:
>    with tlps you can simply use the pointer and that is really the normal
>    usage (the \tlp_use:N is really syntactic sugar) while with toks you
>    absolutely need the accessor in every case.

Generally, I like to have accessor functions.

> So my proposal for tlps is that we document they support balanced token  
> lists
> not containing #. we can add a footnote saying that technically that is  
> only
> an approimation of the truth and that the reality is ... but that we  
> recommend
> that this feature is not to be used or only in places where the author  
> is in
> full control of the processing and the input.

This would be fine by me.

> A followup question then is
>
>  d) what exactly do we want to support in the input stream, ie on the  
> level of
>     the functions currently called \tlist...
>
> Morten said he wrote those with the idea in mind that they should accept
> arbitrary balanced token lists similar to toks. If we can do this  
> reliably
> (and want to) then, yes, we should probably call them \toks_...

It's been a while since tlist was introduced but I do recall a little  
about the motivation behind them and the naming consequence of clist.

It all started with a desire to do "proper" mapping functions. For clist  
one of the problems of \@for is that it expands its comma list once which  
is fine for some things but less so for others. Therefore, the clist  
module got both a N variant for those stored in bins and an n variant for  
those straight from the input stream.

A similar need arose for tlps. However, it didn't really make sense to me  
to have a token list pointer function have an n type argument - then it  
wouldn't be a pointer. Hence, tlist was conceived which was the best I  
could come up with at the time. These quickly proved to have a life of  
their own, testing input streams for being empty, blank, turning them into  
harmless character tokens[1], providing new names for the \uppercase and  
\lowercase primitives, getting head and tail of a list, etc.

> Personally I'm not sure we can do that reliably in the end (assuming more
> functions pop up where we want to build an inline version operating on  
> the
> input stream) and I don't really think it is necessary.

As long as there is no attempt to actually store a part of such a tlist  
argument in a macro, then there is no problem. Is there? Much like my  
argument for making the kernel functions almost exclusively \long, knowing  
what gets into the low-level functions is hard to predict. That's part of  
the reason why seq and prop were transformed into toks registers some time  
ago.


[1] Another type which we have not really touched much: the "str" defined  
as tokens with catcode 12 (and space catcode 10) as output by several  
primitives.

-- 
Morten