Have begun review, but only gotten about a third of the way. Some 
remarks so far:

1. Is it possible to use a space as delimiter of an argument (perhaps 
most interesting for u arguments)? I believe I spotted some places in 
the d->D conversion that would gobble a space as <token>.

2. I believe \xparse_prepare_next:w should be listed as a variable (or 
maybe "variable function"), since it is getting redefined rather 
frequently and seems to be keeping track of the state of the argspec 
parser automaton.

3. Though provided in .dtx format, I find the implementation section 
somewhat illiterate (i.e., not up to literate programming standards). 
Some concrete examples:

  * \l_xparse_processor_int and \l_xparse_processor_use_int are 
documented as "For keeping a count of post-processors and then using 
them." Well, I could guess about as much from the names alone. What 
would be more interesting to see spelt out is /how/ these keeps a track 
of post-processors; what does actual values of these variables mean? At 
what stage in the process are they used?

  * The overall structure of the code starts with lots of little things 
(macros concerned with parsing specific argspecs) and ends (I'm 
guessing, from xparse-alt) with putting them together. The more 
literate approach would be to start with the big picture -- either from 
the "what goes on at run-time" or "what goes on at define-time" point 
of view -- so that one knows what the little things will fit into when 
one gets to them.


Joseph Wright skrev:
> Lars Hellström wrote:
>> Haven't had time to review the code yet, I've only just downloaded it.
>> Will get back to you when I have had a look at it. (I'm especially
>> curious to how on earth you managed to do optional arguments at
>> expand-time.)
> 
> That is done basically following etextools. For each argument, grab #1

Aha, the optimistic approach of hoping everything can be processed as 
undelimited macro arguments. That I can believe to be doable.

>> Have latex'ed xparse.dtx, though, and it throws a few errors in the
>> index. I think those are due to \xparse_add_type_>:, as the ">" is
>> special to makeindex, forcing an extra index level where none was
>> intended. Harmless character sequences avoids that trap. ;-)
> 
> Well, now that we have a mechanism available l3doc can be altered to use
> it :-)
> 
> I'd missed this before as I'd not run makeindex on xparse. I've
> corrected my batch file so this happens!

A similar issue: Today I pdflatex'ed source3.tex (rev. 1464), and it 
goes into an infinite loop at \end{document} (\seq_if_in:NnT being one 
thingie involved).

>> In the meantime, here are some technical issues to consider.
>>
>> 1. What happens if I do this:
>>
>>   \DeclareDocumentCommand{\foo}{ +O{\par} m +O{\par} }{ ... }
>>
>> ? One technical problem I saw in xdoc2l3 was that of simultaneously
>> having \par tokens in defaults and non-long arguments.
> 
> I hope this should work.  The method I've adopted is to use two
> functions to grab each argument. The first is \long, and gets hold of
> the stuff remaining to be processed. It then sets up the real grabber,
> so m args are grabbed something like:
> 
> \cs_new:Npn \xparse_grab_m:w #1 \l_xparse_args_toks {
>   \cs_set_nopar:Nn \xparse_grab_m_aux:n {
>     \toks_put_right:Nn \l_xparse_args_toks { {##1} }
>     #1 \l_xparse_args_toks
>   }
> }

So you store it away in a macro... Yes, one probably has to do 
something like that to handle this situation. Any particular reason 
\xparse_grab_m_aux:n includes an "m" in the name, though? If you're 
constantly redefining it, then why not use the same scratch control 
sequence for all grabbers that require something of this sort...

>> 2. What happens (for the various argument types) when TeX fails to grab
>> an argument? (Failures happen for long arguments when encountering a
>> mismatched right brace, and for short argument also when encountering a
>> \par token.) To keep in line with TeX, everything up to the ungrabbable
>> token should disappear, but an uncareful implementation could leave
>> other tokens in place as well, which would subject the user to
>> mysterious additional errors.
> 
> The basic idea of xparse is not mine: ask the more senior team members!

This is not about the ideas, but whether the implementation of these 
ideas behaves well under stress.

> Seriously, my tests suggest things are okay. As illustrated above, what
> will happen is the aux function will complain, perhaps put something odd
> onto the toks

Leaving something odd in a toks register shouldn't be a problem. The 
problem is leaving odd tokens in the input.

> then get back to business with grabbing any more arguments.

This I find implausible. Getting back to business after \futurelet'ing 
and not finding what was expected, yes, that's no problem, but getting 
back to business after failing to expand a macro is hard; everything 
from the macro to the token where TeX gave up will be gone (but that is 
usually the sane thing to do).

To be clear: The only way you can keep some control over what happens 
after asking TeX to expand a macro \foo and seeing that fail for you, 
is if you \expandafter'ed to it, i.e., did something like

   \expandafter \bar \foo

as that will still leave a \bar to be expanded. As long as you don't 
have such a \foo grab some argument then we're fine: TeX will clean 
everything up for us. (But if you do use such a construction somewhere, 
then you also have to design \bar so that it cleans itself up when it 
is not followed by an expansion of \foo.)

Lars Hellström