Hi friends! :)

Sorry for the delay. The Mac setup was driving me crazy. :P

 > => more people run the testsuite => it is easier for people outside
 > the kernel team to add tests.  In particular, many of the l3packages
 > are drastically lacking tests.

Indeed. More tests, more stability IMHO.

 > My priority is to go for thorough tests which test all corner cases,
 > rather than fast ones.  I may be wrong doing that, though, since the
 > tests are starting to take a rather long time to build.

Ah I see. :) It's a good approach I think. And running test suites are 
usually slow, but it's a good excuse for us to go for a cup of coffee 
while the tests are running. :)

 >> One possibility is to have a test spec, so we can have a "generic"
 >> test infrastructure which reads this spec and "knows" how to perform
 >> a certain analysis.
 >
 > Can you elaborate?  Currently we simply run some functions, and check
 > the output.

I was just thinking out loud. :) I was thinking of a input format, say 
written in XML (/me looks at David) or another markup language, with a 
list of assertions, e.g, (sorry I don't know the innards of the tests 
neither L3 sutff)

<asserttrue input="\foo" expected="bar" />

or something similar. To test a certain code snippet, a single test spec 
would suffice.

 > Paulo, how hard would it be to ship the LuaFileSystem library with
 > a hypothetical Lua-based test file?

Bruno, I have very good news! texlua is shipped with lfs out of the box! 
IIRC we only need to install it for a "normal" Lua interpreter.

We already have the OS commands ready for us to use via lfs. Yay! :)

Best wishes,

Paulo

Em 10/08/2012 15:29, Bruno Le Floch escreveu:
> Hi all,
>
>>> The important non-functional requirement is that it works
>>>
>>>      - automatically
>>>      - and reasonably fast
>>>
>>> Obviously you have to run TeX on the test files, but doing the
>>> comparison using TeX is not very time efficient.
>>>
>>> Now is it important that it is OS independent? Only if your goal is to
>>> have *many* people use the mechanism and so far that wasn't really part
>>> of the spec.
>
> I would add to your list
>
>      - easy to install / understand
>
> => more people run the testsuite => it is easier for people outside
> the kernel team to add tests.  In particular, many of the l3packages
> are drastically lacking tests.
>
>>> Perl is easily available both for unix and windows so effectively for
>>> developers the current system is not to difficult to  install or use.
>>> The idea of using Lua is interesting, but as Joseph said, it is not that
>>> this would then work out of the box for most people either (not now
>>> anyway).
>>>
>>> Midterm, or if we think that this should be a package outside expl3 or
>>> 2e core development, it would perhaps be a good option though, but for
>>> now my feeling is it would mean putting resources onto something that
>>> doesn't actually bring any new value.
>
> Agreed.
>
>>   > I think the big question is what is your goal here. My main goal
>>   > initially (for 2e and later for l3 code) was to have a robust test
>>   > suite that would enable us to identity issues upfront when making
>>   > changes or additions. And that on the whole has been very successful
>>   > in the past (provided as Joseph said, people wrote test files :-)
>>
>> I can relate to that. :) Writing tests is one of the major concerns of
>> the software industry. There's a lot of levels on how to test code, from
>> unit to integration, and approaches, from the bowels to the interface -
>> the Software Engineering wolves love these concepts. :P Of course, each
>> project has its own needs and requiments. For LaTeX3, IMHO, it's
>> critical to have every bit of code - from a lovely high level interface
>> to an obscure undocumented feature - exhaustively tested.
>
> We're working on it, and I think most of l3kernel is tested.
> Obviously not enough, as I keep finding obscure bugs.  Latest one: the
> msg system sometimes x-expands several times in a row the same
> arguments.  It'll get fixed soon.
>
> One thing that we don't have is detailed information about which
> functions are tested.  I've added recently a "tested = <testfile>" key
> to the macro environment in l3doc, but I've only used it in a couple
> of modules, and the key is entirely ignored.  It may be good to extend
> this to other modules.
>
>>   > The important non-functional requirement is that it works
>>   >
>>   >     - automatically
>>   >     - and reasonably fast
>>
>> Agreed. Though a bunch of tests might require a reasonable amount of
>> time, each test should be as simple as possible, thus making its
>> checking relatively fast.
>
> My priority is to go for thorough tests which test all corner cases,
> rather than fast ones.  I may be wrong doing that, though, since the
> tests are starting to take a rather long time to build.
>
>>   > Now is it important that it is OS independent? Only if your goal is to
>>   > have *many* people use the mechanism and so far that wasn't really
>>   > part of the spec.
>>
>> I believe that OS independent test suites might help us catch things
>> that are OS-specific, or even detect possible issues with a certain
>> vendor implementation. Again, I speak as an outsider; I have no idea how
>> the code actually works.
>
> One of my main concerns, after looking at the plethora of Makefiles
> and make.bat is that there is a lot of redundancy there, which means
> that (1) changes don't propagate well (2) we occasionally forget to
> add a module to the Makefile of the parent directory, and it never
> gets tested, installed, unless someone is working on that module.
>
> So I guess that in the LaTeX3 case the discussion should be extended
> to the build system as a whole: the test system works well, as Frank
> says.  It would be nice if we could get rid of Makefiles and make.bat,
> and replace them by light-weight configuration files.  After all, the
> Makefiles are all very similar, so a top-level Lua script (or other)
> should be able to do most of the work.
>
>>   > Perl is easily available both for unix and windows so effectively for
>>   > developers the current system is not to difficult to  install or use.
>>   > The idea of using Lua is interesting, but as Joseph said, it is not
>>   > that this would then work out of the box for most people either (not
>>   > now anyway).
>>
>> I share the same opinion. :) One idea I see is to pack the test system
>> as a batteries-included file (.sh for Unix, .exe for Windows) packing
>> the interpreter together. I've seen this before with some languages -
>> Ruby, Python, Perl, Java - and it's doable. Of course, there's a
>> drawback of having the payload of the virtual machine / interpreter
>> bundled with the script, but at least it would be very easy to deploy.
>> Or we can write the system with another language - say C - and generate
>> a binary file.
>
> There is also the question of licenses if you bundle an external
> dependency with the code repo.  I really don't know the details,
> perhaps Frank has some ideas?
>
>> I like the idea of using Lua. Enrico and I wrote a Lua script and the
>> deployment in TeX Live was very, very easy. It runs under texlua, which
>> is already available in the modern TeX distros. If the guy doesn't have
>> a TeX distro, why in the Earth he wants to run a LaTeX3 test suite? :P
>
> The guy might have an old TeX distro :).  Probably you're right that
> all potential LaTeX3 testers probably have a reasonably up-to-date
> distro.  However, how "up-to-date" does it have to be?  Anyone knows
> when texlua was introduced?
>
>>   > not sure they are alternatives, but there could be approaches that are
>>   > worth incorporating into the current setup.
>>
>> One possibility is to have a test spec, so we can have a "generic" test
>> infrastructure which reads this spec and "knows" how to perform a
>> certain analysis.
>
> Can you elaborate?  Currently we simply run some functions, and check
> the output.
>
>> I volunteer myself to help. :) You guys know that TeX is still a monster
>> to me, but at least I think I can help in other battle fronts. :)
>
> Yeay!
>
> One problem with building a build system in Lua is that Lua does not
> provide any function to list files in a directory.  The only way in
> pure Lua is to do the equivalent of \immediate\write18{os-specific
> command}. There are external libraries which do that in an
> os-dependent way.  Paulo, how hard would it be to ship the
> LuaFileSystem library with a hypothetical Lua-based test file?
>
> Regards,
> Bruno
>