On Wed, Mar 24, 2010 at 01:29:22PM +0200, Khaled Hosny wrote:

> On Thu, Mar 11, 2010 at 11:04:30PM +0100, Heiko Oberdiek wrote:
> > On Thu, Mar 11, 2010 at 11:25:44PM +0200, Khaled Hosny wrote:
> > 
> > > On Thu, Mar 11, 2010 at 09:24:06PM +0100, Heiko Oberdiek wrote:
> > > > On Thu, Mar 11, 2010 at 10:12:25PM +0200, Khaled Hosny wrote:
> > > > 
> > > > > What about the approach we used in etex.src; redefining \patterns and
> > > > > \hyphenation to use corresponding lua functions, so we don't need to
> > > > > duplicate hyphenation files and, AFAIU, solves the catcode issue.
> > > > 
> > > > \patterns is not read by Lua but by TeX, thus the catcode
> > > > problem remains.
> > > 
> > > I don't claim that I understand anything, but do we have some example
> > > for the likes of me to see the problem?
> > 
> > * verbatim mode
> > * babel shorthands
> > * other active characters
> > * ...
> 
> A more concrete example showing such an issue would greatly make testing
> much either, I, apparently, don't know enough catcodes black magic to
> come with such a test file myself.

% Setup for iniTeX
% ----------------
\catcode`\{=1 %
\catcode`\}=2 %
\catcode`\#=6 %

% Helper macros
% -------------
\def\ForLoopCount#1{%
  \countdef\LoopCount=255 %
  \LoopCount=#1\relax
  \DoForLoopCount
}
\def\DoForLoopCount#1#2{%
  \ifnum#1<\LoopCount
    \expandafter\Gobble
  \else 
    \expandafter\FirstOfOne
  \fi
  {%
    #2%
    \advance\LoopCount by 1 %
    \DoForLoopCount{#1}{#2}%
  }%
}
\long\def\Gobble#1{}
\long\def\FirstOfOne#1{#1}
\def\SetCatcodeRange#1#2#3{%
  \ForLoopCount{#1}{#2}{\catcode\LoopCount=#3\relax}%
}
\def\SaveCatcodes{%
  \begingroup
    \toks0=\expandafter{%
      \expandafter\endlinechar\number\the\endlinechar
      \expandafter\relax
      \expandafter\escapechar\number\the\escapechar
      \expandafter\relax  
      \expandafter\newlinechar\the\newlinechar\relax
    }%
    \ForLoopCount{0}{255}{%  
      \toks0=\expandafter{%  
        \the\toks0%
        \expandafter\catcode\number\the\LoopCount
        \expandafter=\the\catcode\LoopCount
        \relax
      }%
    }%
    \edef\x{%
      \endgroup
      \def\noexpand\RestoreCatcodes{\the\toks0}%
    }%
  \x
}
\def\CatcodeBlackMagic#1{%
  \SaveCatcodes
  \SetCatcodeRange{0}{255}{15}% invalid
  \SetCatcodeRange{65}{90}{11}% A-Z
  \SetCatcodeRange{97}{122}{11}% a-z
  \SetCatcodeRange{48}{57}{12}% 0-9
  #1%
  \RestoreCatcodes    
}  

% Testing
% -------

\def\Test{%
  % as example read file `s.tex'
  \catcode`\%=14 % comment
  \catcode`\ =10 % space  
  \catcode`\\=0 % escape  
  \catcode`\{=1 % brace left
  \catcode`\}=2 % brace right
  \catcode`\@=12 % other
  \endlinechar=13 % ^^M 
  \catcode\endlinechar=5 % end of line
  \input s %
}

\CatcodeBlackMagic{\Test}

% End of job
% ----------
\csname @@end\endcsname\end


If you are in the middle of nowhere (inside \CatcodeBlackMagic),
then you cann't simply use "\input s", but you have to ensure
the conditions (settings at the beginning of \Test) that the
file can be read properly.

Yours sincerely
  Heiko