Hi all,

>> The important non-functional requirement is that it works
>>
>>     - automatically
>>     - and reasonably fast
>>
>> Obviously you have to run TeX on the test files, but doing the
>> comparison using TeX is not very time efficient.
>>
>> Now is it important that it is OS independent? Only if your goal is to
>> have *many* people use the mechanism and so far that wasn't really part
>> of the spec.

I would add to your list

    - easy to install / understand

=> more people run the testsuite => it is easier for people outside
the kernel team to add tests.  In particular, many of the l3packages
are drastically lacking tests.

>> Perl is easily available both for unix and windows so effectively for
>> developers the current system is not to difficult to  install or use.
>> The idea of using Lua is interesting, but as Joseph said, it is not that
>> this would then work out of the box for most people either (not now
>> anyway).
>>
>> Midterm, or if we think that this should be a package outside expl3 or
>> 2e core development, it would perhaps be a good option though, but for
>> now my feeling is it would mean putting resources onto something that
>> doesn't actually bring any new value.

Agreed.

>  > I think the big question is what is your goal here. My main goal
>  > initially (for 2e and later for l3 code) was to have a robust test
>  > suite that would enable us to identity issues upfront when making
>  > changes or additions. And that on the whole has been very successful
>  > in the past (provided as Joseph said, people wrote test files :-)
>
> I can relate to that. :) Writing tests is one of the major concerns of
> the software industry. There's a lot of levels on how to test code, from
> unit to integration, and approaches, from the bowels to the interface -
> the Software Engineering wolves love these concepts. :P Of course, each
> project has its own needs and requiments. For LaTeX3, IMHO, it's
> critical to have every bit of code - from a lovely high level interface
> to an obscure undocumented feature - exhaustively tested.

We're working on it, and I think most of l3kernel is tested.
Obviously not enough, as I keep finding obscure bugs.  Latest one: the
msg system sometimes x-expands several times in a row the same
arguments.  It'll get fixed soon.

One thing that we don't have is detailed information about which
functions are tested.  I've added recently a "tested = <testfile>" key
to the macro environment in l3doc, but I've only used it in a couple
of modules, and the key is entirely ignored.  It may be good to extend
this to other modules.

>  > The important non-functional requirement is that it works
>  >
>  >     - automatically
>  >     - and reasonably fast
>
> Agreed. Though a bunch of tests might require a reasonable amount of
> time, each test should be as simple as possible, thus making its
> checking relatively fast.

My priority is to go for thorough tests which test all corner cases,
rather than fast ones.  I may be wrong doing that, though, since the
tests are starting to take a rather long time to build.

>  > Now is it important that it is OS independent? Only if your goal is to
>  > have *many* people use the mechanism and so far that wasn't really
>  > part of the spec.
>
> I believe that OS independent test suites might help us catch things
> that are OS-specific, or even detect possible issues with a certain
> vendor implementation. Again, I speak as an outsider; I have no idea how
> the code actually works.

One of my main concerns, after looking at the plethora of Makefiles
and make.bat is that there is a lot of redundancy there, which means
that (1) changes don't propagate well (2) we occasionally forget to
add a module to the Makefile of the parent directory, and it never
gets tested, installed, unless someone is working on that module.

So I guess that in the LaTeX3 case the discussion should be extended
to the build system as a whole: the test system works well, as Frank
says.  It would be nice if we could get rid of Makefiles and make.bat,
and replace them by light-weight configuration files.  After all, the
Makefiles are all very similar, so a top-level Lua script (or other)
should be able to do most of the work.

>  > Perl is easily available both for unix and windows so effectively for
>  > developers the current system is not to difficult to  install or use.
>  > The idea of using Lua is interesting, but as Joseph said, it is not
>  > that this would then work out of the box for most people either (not
>  > now anyway).
>
> I share the same opinion. :) One idea I see is to pack the test system
> as a batteries-included file (.sh for Unix, .exe for Windows) packing
> the interpreter together. I've seen this before with some languages -
> Ruby, Python, Perl, Java - and it's doable. Of course, there's a
> drawback of having the payload of the virtual machine / interpreter
> bundled with the script, but at least it would be very easy to deploy.
> Or we can write the system with another language - say C - and generate
> a binary file.

There is also the question of licenses if you bundle an external
dependency with the code repo.  I really don't know the details,
perhaps Frank has some ideas?

> I like the idea of using Lua. Enrico and I wrote a Lua script and the
> deployment in TeX Live was very, very easy. It runs under texlua, which
> is already available in the modern TeX distros. If the guy doesn't have
> a TeX distro, why in the Earth he wants to run a LaTeX3 test suite? :P

The guy might have an old TeX distro :).  Probably you're right that
all potential LaTeX3 testers probably have a reasonably up-to-date
distro.  However, how "up-to-date" does it have to be?  Anyone knows
when texlua was introduced?

>  > not sure they are alternatives, but there could be approaches that are
>  > worth incorporating into the current setup.
>
> One possibility is to have a test spec, so we can have a "generic" test
> infrastructure which reads this spec and "knows" how to perform a
> certain analysis.

Can you elaborate?  Currently we simply run some functions, and check
the output.

> I volunteer myself to help. :) You guys know that TeX is still a monster
> to me, but at least I think I can help in other battle fronts. :)

Yeay!

One problem with building a build system in Lua is that Lua does not
provide any function to list files in a directory.  The only way in
pure Lua is to do the equivalent of \immediate\write18{os-specific
command}. There are external libraries which do that in an
os-dependent way.  Paulo, how hard would it be to ship the
LuaFileSystem library with a hypothetical Lua-based test file?

Regards,
Bruno