Will Robertson wrote:
 > After all this I now agree with Frank (that the every-toks require a
> wrapper to be useful, and expl3 isn't the place to do that), so let's
> agree to add \endlinechar directly (except we want \l_char_end_line_int
> instead of _endline_ I think) and...

By the way, I assume the lack of \escapechar is deliberate?

> ... add a version of scantokens that looks approximately like this:
> (just copying Joseph's code)
> 
> \cs_set_eq:NN \l_char_end_line_int \tex_endlinechar:D
> \cs_new:Npn \tlist_rescan:n #1 {
>  \group_begin:
>    \tex_everyeof:D {\exp_not:N}
>    \int_set:Nn \l_char_end_line_int {\c_minus_one}
>    \toks_set:Nx \l_tmpa_toks { \etex_scantokens {#1} }
>    \exp_after:NN
>  \group_end:
>  \toks_use:N \l_tmpa_toks
> }
> 
> (Sorry for my ignorance but what's the endlinechar part for? Maybe we
> don't need that?)

Sometimes you need it, sometime you don't :-).  I got this from
biblatex, where Philip has things like:

\long\def\blx@ifnum#1{%
  \blx@hook@ifnum
  \def\do##1{\uccode`##1=`\%}%
  \do\ \do\0\do\1\do\2\do\3\do\4\do\5\do\6\do\7\do\8\do\9%
  \do\i\do\v\do\x\do\l\do\c\do\d\do\m
  \do\I\do\V\do\X\do\L\do\C\do\D\do\M
  \blx@donumchars
  \let\RN\@gobble
  \let\Rn\@gobble
  \makeatletter
  \catcode`\%=9
  \endlinechar\m@ne
  \everyeof{\noexpand}%
  \let\protect\@unexpandable@protect
  \uppercase{\edef\blx@tempa{\scantokens{#1}}}%
  \ifx\blx@tempa\@empty
    \aftergroup\@firstoftwo
  \else
    \aftergroup\@secondoftwo
  \fi
  \endgroup}

As long as the suggested function will work under a range of possible
circumstances where \scantokens can be used, I'm happy.
-- 
Joseph Wright