Will Robertson wrote:
 > After all this I now agree with Frank (that the every-toks require a
> wrapper to be useful, and expl3 isn't the place to do that), so let's
> agree to add \endlinechar directly (except we want \l_char_end_line_int
> instead of _endline_ I think) and...

By the way, I assume the lack of \escapechar is deliberate?

> ... add a version of scantokens that looks approximately like this:
> (just copying Joseph's code)
> \cs_set_eq:NN \l_char_end_line_int \tex_endlinechar:D
> \cs_new:Npn \tlist_rescan:n #1 {
>  \group_begin:
>    \tex_everyeof:D {\exp_not:N}
>    \int_set:Nn \l_char_end_line_int {\c_minus_one}
>    \toks_set:Nx \l_tmpa_toks { \etex_scantokens {#1} }
>    \exp_after:NN
>  \group_end:
>  \toks_use:N \l_tmpa_toks
> }
> (Sorry for my ignorance but what's the endlinechar part for? Maybe we
> don't need that?)

Sometimes you need it, sometime you don't :-).  I got this from
biblatex, where Philip has things like:

  \do\ \do\0\do\1\do\2\do\3\do\4\do\5\do\6\do\7\do\8\do\9%

As long as the suggested function will work under a range of possible
circumstances where \scantokens can be used, I'm happy.
Joseph Wright