Hi all,
>> The important non-functional requirement is that it works
>>
>> - automatically
>> - and reasonably fast
>>
>> Obviously you have to run TeX on the test files, but doing the
>> comparison using TeX is not very time efficient.
>>
>> Now is it important that it is OS independent? Only if your goal is to
>> have *many* people use the mechanism and so far that wasn't really part
>> of the spec.
I would add to your list
- easy to install / understand
=> more people run the testsuite => it is easier for people outside
the kernel team to add tests. In particular, many of the l3packages
are drastically lacking tests.
>> Perl is easily available both for unix and windows so effectively for
>> developers the current system is not to difficult to install or use.
>> The idea of using Lua is interesting, but as Joseph said, it is not that
>> this would then work out of the box for most people either (not now
>> anyway).
>>
>> Midterm, or if we think that this should be a package outside expl3 or
>> 2e core development, it would perhaps be a good option though, but for
>> now my feeling is it would mean putting resources onto something that
>> doesn't actually bring any new value.
Agreed.
> > I think the big question is what is your goal here. My main goal
> > initially (for 2e and later for l3 code) was to have a robust test
> > suite that would enable us to identity issues upfront when making
> > changes or additions. And that on the whole has been very successful
> > in the past (provided as Joseph said, people wrote test files :-)
>
> I can relate to that. :) Writing tests is one of the major concerns of
> the software industry. There's a lot of levels on how to test code, from
> unit to integration, and approaches, from the bowels to the interface -
> the Software Engineering wolves love these concepts. :P Of course, each
> project has its own needs and requiments. For LaTeX3, IMHO, it's
> critical to have every bit of code - from a lovely high level interface
> to an obscure undocumented feature - exhaustively tested.
We're working on it, and I think most of l3kernel is tested.
Obviously not enough, as I keep finding obscure bugs. Latest one: the
msg system sometimes x-expands several times in a row the same
arguments. It'll get fixed soon.
One thing that we don't have is detailed information about which
functions are tested. I've added recently a "tested = <testfile>" key
to the macro environment in l3doc, but I've only used it in a couple
of modules, and the key is entirely ignored. It may be good to extend
this to other modules.
> > The important non-functional requirement is that it works
> >
> > - automatically
> > - and reasonably fast
>
> Agreed. Though a bunch of tests might require a reasonable amount of
> time, each test should be as simple as possible, thus making its
> checking relatively fast.
My priority is to go for thorough tests which test all corner cases,
rather than fast ones. I may be wrong doing that, though, since the
tests are starting to take a rather long time to build.
> > Now is it important that it is OS independent? Only if your goal is to
> > have *many* people use the mechanism and so far that wasn't really
> > part of the spec.
>
> I believe that OS independent test suites might help us catch things
> that are OS-specific, or even detect possible issues with a certain
> vendor implementation. Again, I speak as an outsider; I have no idea how
> the code actually works.
One of my main concerns, after looking at the plethora of Makefiles
and make.bat is that there is a lot of redundancy there, which means
that (1) changes don't propagate well (2) we occasionally forget to
add a module to the Makefile of the parent directory, and it never
gets tested, installed, unless someone is working on that module.
So I guess that in the LaTeX3 case the discussion should be extended
to the build system as a whole: the test system works well, as Frank
says. It would be nice if we could get rid of Makefiles and make.bat,
and replace them by light-weight configuration files. After all, the
Makefiles are all very similar, so a top-level Lua script (or other)
should be able to do most of the work.
> > Perl is easily available both for unix and windows so effectively for
> > developers the current system is not to difficult to install or use.
> > The idea of using Lua is interesting, but as Joseph said, it is not
> > that this would then work out of the box for most people either (not
> > now anyway).
>
> I share the same opinion. :) One idea I see is to pack the test system
> as a batteries-included file (.sh for Unix, .exe for Windows) packing
> the interpreter together. I've seen this before with some languages -
> Ruby, Python, Perl, Java - and it's doable. Of course, there's a
> drawback of having the payload of the virtual machine / interpreter
> bundled with the script, but at least it would be very easy to deploy.
> Or we can write the system with another language - say C - and generate
> a binary file.
There is also the question of licenses if you bundle an external
dependency with the code repo. I really don't know the details,
perhaps Frank has some ideas?
> I like the idea of using Lua. Enrico and I wrote a Lua script and the
> deployment in TeX Live was very, very easy. It runs under texlua, which
> is already available in the modern TeX distros. If the guy doesn't have
> a TeX distro, why in the Earth he wants to run a LaTeX3 test suite? :P
The guy might have an old TeX distro :). Probably you're right that
all potential LaTeX3 testers probably have a reasonably up-to-date
distro. However, how "up-to-date" does it have to be? Anyone knows
when texlua was introduced?
> > not sure they are alternatives, but there could be approaches that are
> > worth incorporating into the current setup.
>
> One possibility is to have a test spec, so we can have a "generic" test
> infrastructure which reads this spec and "knows" how to perform a
> certain analysis.
Can you elaborate? Currently we simply run some functions, and check
the output.
> I volunteer myself to help. :) You guys know that TeX is still a monster
> to me, but at least I think I can help in other battle fronts. :)
Yeay!
One problem with building a build system in Lua is that Lua does not
provide any function to list files in a directory. The only way in
pure Lua is to do the equivalent of \immediate\write18{os-specific
command}. There are external libraries which do that in an
os-dependent way. Paulo, how hard would it be to ship the
LuaFileSystem library with a hypothetical Lua-based test file?
Regards,
Bruno
|