Dear friends,

First of all, apologies for this messy reply (changing from Digest to 
Regular Mail). Thanks to Joseph for pointing me at the current 
discussion. :)

 > I think the big question is what is your goal here. My main goal
 > initially (for 2e and later for l3 code) was to have a robust test
 > suite that would enable us to identity issues upfront when making
 > changes or additions. And that on the whole has been very successful
 > in the past (provided as Joseph said, people wrote test files :-)

I can relate to that. :) Writing tests is one of the major concerns of 
the software industry. There's a lot of levels on how to test code, from 
unit to integration, and approaches, from the bowels to the interface - 
the Software Engineering wolves love these concepts. :P Of course, each 
project has its own needs and requiments. For LaTeX3, IMHO, it's 
critical to have every bit of code - from a lovely high level interface 
to an obscure undocumented feature - exhaustively tested.

Fair point, people don't write test suites as they should. /me is 
included :P

 > The important non-functional requirement is that it works
 >
 >     - automatically
 >     - and reasonably fast

Agreed. Though a bunch of tests might require a reasonable amount of 
time, each test should be as simple as possible, thus making its 
checking relatively fast.

 > Obviously you have to run TeX on the test files, but doing the
 > comparison using TeX is not very time efficient.

I trust you guys. :) I'm still in the middle of the TeXbook, fighting 
with some chapters there. :P

 > Now is it important that it is OS independent? Only if your goal is to
 > have *many* people use the mechanism and so far that wasn't really
 > part of the spec.

I believe that OS independent test suites might help us catch things 
that are OS-specific, or even detect possible issues with a certain 
vendor implementation. Again, I speak as an outsider; I have no idea how 
the code actually works.

 > Perl is easily available both for unix and windows so effectively for
 > developers the current system is not to difficult to  install or use.
 > The idea of using Lua is interesting, but as Joseph said, it is not
 > that this would then work out of the box for most people either (not
 > now anyway).

I share the same opinion. :) One idea I see is to pack the test system 
as a batteries-included file (.sh for Unix, .exe for Windows) packing 
the interpreter together. I've seen this before with some languages - 
Ruby, Python, Perl, Java - and it's doable. Of course, there's a 
drawback of having the payload of the virtual machine / interpreter 
bundled with the script, but at least it would be very easy to deploy. 
Or we can write the system with another language - say C - and generate 
a binary file.

I like the idea of using Lua. Enrico and I wrote a Lua script and the 
deployment in TeX Live was very, very easy. It runs under texlua, which 
is already available in the modern TeX distros. If the guy doesn't have 
a TeX distro, why in the Earth he wants to run a LaTeX3 test suite? :P

 > I would even claim that it works better than "pretty well" as it
 > allows to write the right kind of tests for typesetting results as
 > well as for testing functionality of code and interface specs and all
 > that in an automated way, and in fact it caught quite a number of
 > errors in the past.

I'm really impressed with the current test infrastructure. It looks 
awesome. :)

 > not sure they are alternatives, but there could be approaches that are
 > worth incorporating into the current setup.

One possibility is to have a test spec, so we can have a "generic" test 
infrastructure which reads this spec and "knows" how to perform a 
certain analysis.

I volunteer myself to help. :) You guys know that TeX is still a monster 
to me, but at least I think I can help in other battle fronts. :)

Paulo

Em 10/08/2012 09:00, LATEX-L automatic digest system escreveu:
> There is 1 message totaling 69 lines in this issue.
>
> Topics collected thus far:
>
>    1. Examples of l3doc & unit testing?
>
> ----------------------------------------------------------------------
>
> Date:    Fri, 10 Aug 2012 12:22:56 +0200
> From:    Frank Mittelbach <[log in to unmask]>
> Subject: Re: Examples of l3doc & unit testing?
>
> Am 09.08.2012 23:17, schrieb Joseph Wright:
>> On 09/08/2012 18:30, Bruno Le Floch wrote:
>>> We run our test suite using a Makefile (or make.bat on Windows), which
>>> - calls the appropriate TeX engine the appropriate number of times, then
>>> - calls a Perl script to remove paths and other parts of the log file
>>> specific to a given installation,
>>> - calls diff to compare the result with a saved result.
>>> The drawbacks are that it is os-dependent, it uses perl, which may not
>>> be installed everywhere.
>
> I think the big question is what is your goal here. My main goal
> initially (for 2e and later for l3 code) was to have a robust test suite
> that would enable us to identity issues upfront when making changes or
> additions. And that on the whole has been very successful in the past
> (provided as Joseph said, people wrote test files :-)
>
> The important non-functional requirement is that it works
>
>     - automatically
>     - and reasonably fast
>
> Obviously you have to run TeX on the test files, but doing the
> comparison using TeX is not very time efficient.
>
> Now is it important that it is OS independent? Only if your goal is to
> have *many* people use the mechanism and so far that wasn't really part
> of the spec.
>
> Perl is easily available both for unix and windows so effectively for
> developers the current system is not to difficult to  install or use.
> The idea of using Lua is interesting, but as Joseph said, it is not that
> this would then work out of the box for most people either (not now anyway).
>
> Midterm, or if we think that this should be a package outside expl3 or
> 2e core development, it would perhaps be a good option though, but for
> now my feeling is it would mean putting resources onto something that
> doesn't actually bring any new value.
>
>
>> The current LaTeX3 test system works pretty well, provided the tests are
>> written to actually test behaviour correctly :-) Checking log file info
>> seems to work well both for 'programmatic' information (writing the
>> result to the log), and for 'typesetting' (by using \box_show:N to again
>> write to the log). So it does not seem like a bad model. It might be
>> worth looking at the TeX part of the process (the test .tex file) to see
>> if it needs any tidying up to be more generally usable.
>
> I would even claim that it works better than "pretty well" as it allows
> to write the right kind of tests for typesetting results as well as for
> testing functionality of code and interface specs and all that in an
> automated way, and in fact it caught quite a number of errors in the past.
>
>> Of course, if you do want to look at this then it would also be worth
>> looking at the alternatives (e.g. qstest).
>
> not sure they are alternatives, but there could be approaches that are
> worth incorporating into the current setup.
>
> frank
>
> ------------------------------
>
> End of LATEX-L Digest - 9 Aug 2012 to 10 Aug 2012 - Unfinished
> **************************************************************
>