Sent this yesterday, but it doesn't seem to have gotten through to 
listserv.uni-heidelberg.de, so I'm resending it (with copies to Will 
and Joseph, in case it's an overzealous spam lockout in Heidelberg; I 
seem to recall this happening before).

Will Robertson skrev:
> Hi Lars,
> 
> On 06/08/2009, at 8:55 AM, Lars Hellström wrote:
> 
>> I should perhaps point out that I've also explored some variations on 
>> the xparse theme (Jan 2008, in a package I called xdoc2l3 -- see 
>> http://abel.math.umu.se/~lars/xdoc/), but back then I didn't get much 
>> response to the questions that required *decisions* (and then I got 
>> distracted by other projects). I will mention the issues I recall at 
>> suitable locations below.
> 
> I'm afraid that I agree with Joseph that regardless of the technical 
> merits of xdoc2l3, the approach it takes sacrifices some measure of 
> simplicity for flexibility. And even with the flexibility we run into 
> the problems of TeX itself -- sure, we can change catcodes, but only if 
> it's not nested within an argument. And even if we use \scantokens, % 
> still kills things.

Actually, all instances I have of \catcode changes with xdoc2l3 are to
more closely mimic base LaTeX2e; I tend not to use them for commands
I've introduced myself. But if you're looking for an example of a
beneficial and non-confusing \catcode change, then consider \index.

The xdoc2l3 preprocessing is more about doing things to an argument
after you've grabbed it, than about changing the conditions under which
it is grabbed.

> The one advantage I can see to allowing { o m } is that there is then a 
> standard technique for testing with \IfNoValueTF, rather than having 
> package authors write things like
>    ... { o{\q_nil} m }
>        { ... \quark_if_nil:NTF #1 {}{} ... }
> which mean the same thing but look different.

You mean, if a default isn't mandatory, then people will be less
inclined to do the Wrong Thing here. True, but a matter that can be
solved through good documentation.


>>> Thinking about catcode-sensitive input, most cases can be handled by the
>>> e-TeX tools we have available. For true verbatim input, that obviously
>>> does not work, but I feel that truing to "crowbar" verbatim input xparse
>>> is a mistake.  Almost all user functions are not verbatim, and the
>>> particular issues for those that are, I think, better handled by hand.
>>> (The way that xparse works means that it absorbs arguments before
>>> passing them to the code you program.)  I also suspect that xparse
>>> handling of verbatim will never be reliable enough for general use.
>>
>> Is that a yes or no to my "harmless character sequences"? These can 
>> handle data like URLs which might prompt people to go verbatim, but 
>> they aren't implemented using \catcode changes.
> 
> Not sure I understand -- even URLs need to sanitise % if they're used 
> within an argument.

Harmless character sequences are like C string literals in this
respect; most characters can be typed raw, but a few need to be
escaped. % would be typed as \PrintChar{37} (or \PrintChar{`\%} if you
don't remember the ASCII table). \, {, and } are the three others that
might need escaping. In the case of URLs, one might want to introduce a
further shorthand for %xx-escapes (and there is a mechanism for doing
that).

>> Other possible xparse features I'd like to see discussed (and recall 
>> right now) are:
>>
>> * The ability to preprocess (e.g. evaluate as a calc expression)
>>   an argument before handing it to the command body.
> 
> Since expl3 functions allow you to evaluate calc expressions anyway, 
> what advantage would there be to this?

Calc is possibly not the best example anymore, but I assure you it is
not the only kind of preprocessing one might want to make. The basic
idea is that rather than going

   \DeclareDocumentCommand ... {
      \cleanup_as_A:Nn\l_tmpa_tl{#1}
      \cleanup_as_B:Nn\l_tmpb_tl{#2}
      \cleanup_as_C:Nn\l_tmpc_tl{#3}
      ...
   }

and refer to the scratch variables \l_tmpa_tl, \l_tmpb_tl, etc. where
the values are needed, we can have #1, #2, etc. be these cleaned-up
values throughout the body. The danger of generic scratch variables is
of course that we don't know whether some other command (that our
command calls) is using them too, whereas command-specific scratch
variables are generally overkill.

Another way of looking at this is that it is a matter of call-by-value
versus call-by-name (cf.
http://en.wikipedia.org/wiki/Evaluation_strategy). Evaluating arguments
before passing them as parameters to the body would amount to adding
support for call-by-value, which is generally considered to be more sane.

>> * The ability to have arguments assigned to token lists rather
>>   than handed as macro parameters (lets one get around the 9
>>   parameter limit).
> 
> 
> If we attempt to keep things simple, then I think this is stretching the 
> limits of what we'd like \DeclareDocumentCommand to achieve. Besides, 
> I've never seen an argument for when >9 arguments are actually necessary 
> that couldn't be implemented in some other way (e.g., keyval or csv list).

Not when creating new commands, no, but funny things happen as commands
grow older (from xdoc and derivatives, I have accumulated experience of 
this). With specifiers like o and s available, it is very tempting to 
every once in a while add another feature to a command, by adding 
another argument or two -- since they're optional, we stay backwards 
compatible throughout the process -- and in a couple of years the 9 
argument limit may be reached. Star-type arguments in particular eat up 
a lot more parameters than it feels like they should from the user 
point of view.

Assign-to-token-list arguments offers a work-around when the limit is
reached.

> I think a goal for xparse should be simplifying the majority of 
> approaches taken by package writers, and leave the esoteric stuff to 
> continue to be implemented by hand.

Fair enough. Just bear in mind that what seems esoteric today may be 
standard practice for the programming technique you'll learn tomorrow.

Lars Hellström