Hi all, >> The important non-functional requirement is that it works >> >> - automatically >> - and reasonably fast >> >> Obviously you have to run TeX on the test files, but doing the >> comparison using TeX is not very time efficient. >> >> Now is it important that it is OS independent? Only if your goal is to >> have *many* people use the mechanism and so far that wasn't really part >> of the spec. I would add to your list - easy to install / understand => more people run the testsuite => it is easier for people outside the kernel team to add tests. In particular, many of the l3packages are drastically lacking tests. >> Perl is easily available both for unix and windows so effectively for >> developers the current system is not to difficult to install or use. >> The idea of using Lua is interesting, but as Joseph said, it is not that >> this would then work out of the box for most people either (not now >> anyway). >> >> Midterm, or if we think that this should be a package outside expl3 or >> 2e core development, it would perhaps be a good option though, but for >> now my feeling is it would mean putting resources onto something that >> doesn't actually bring any new value. Agreed. > > I think the big question is what is your goal here. My main goal > > initially (for 2e and later for l3 code) was to have a robust test > > suite that would enable us to identity issues upfront when making > > changes or additions. And that on the whole has been very successful > > in the past (provided as Joseph said, people wrote test files :-) > > I can relate to that. :) Writing tests is one of the major concerns of > the software industry. There's a lot of levels on how to test code, from > unit to integration, and approaches, from the bowels to the interface - > the Software Engineering wolves love these concepts. :P Of course, each > project has its own needs and requiments. For LaTeX3, IMHO, it's > critical to have every bit of code - from a lovely high level interface > to an obscure undocumented feature - exhaustively tested. We're working on it, and I think most of l3kernel is tested. Obviously not enough, as I keep finding obscure bugs. Latest one: the msg system sometimes x-expands several times in a row the same arguments. It'll get fixed soon. One thing that we don't have is detailed information about which functions are tested. I've added recently a "tested = <testfile>" key to the macro environment in l3doc, but I've only used it in a couple of modules, and the key is entirely ignored. It may be good to extend this to other modules. > > The important non-functional requirement is that it works > > > > - automatically > > - and reasonably fast > > Agreed. Though a bunch of tests might require a reasonable amount of > time, each test should be as simple as possible, thus making its > checking relatively fast. My priority is to go for thorough tests which test all corner cases, rather than fast ones. I may be wrong doing that, though, since the tests are starting to take a rather long time to build. > > Now is it important that it is OS independent? Only if your goal is to > > have *many* people use the mechanism and so far that wasn't really > > part of the spec. > > I believe that OS independent test suites might help us catch things > that are OS-specific, or even detect possible issues with a certain > vendor implementation. Again, I speak as an outsider; I have no idea how > the code actually works. One of my main concerns, after looking at the plethora of Makefiles and make.bat is that there is a lot of redundancy there, which means that (1) changes don't propagate well (2) we occasionally forget to add a module to the Makefile of the parent directory, and it never gets tested, installed, unless someone is working on that module. So I guess that in the LaTeX3 case the discussion should be extended to the build system as a whole: the test system works well, as Frank says. It would be nice if we could get rid of Makefiles and make.bat, and replace them by light-weight configuration files. After all, the Makefiles are all very similar, so a top-level Lua script (or other) should be able to do most of the work. > > Perl is easily available both for unix and windows so effectively for > > developers the current system is not to difficult to install or use. > > The idea of using Lua is interesting, but as Joseph said, it is not > > that this would then work out of the box for most people either (not > > now anyway). > > I share the same opinion. :) One idea I see is to pack the test system > as a batteries-included file (.sh for Unix, .exe for Windows) packing > the interpreter together. I've seen this before with some languages - > Ruby, Python, Perl, Java - and it's doable. Of course, there's a > drawback of having the payload of the virtual machine / interpreter > bundled with the script, but at least it would be very easy to deploy. > Or we can write the system with another language - say C - and generate > a binary file. There is also the question of licenses if you bundle an external dependency with the code repo. I really don't know the details, perhaps Frank has some ideas? > I like the idea of using Lua. Enrico and I wrote a Lua script and the > deployment in TeX Live was very, very easy. It runs under texlua, which > is already available in the modern TeX distros. If the guy doesn't have > a TeX distro, why in the Earth he wants to run a LaTeX3 test suite? :P The guy might have an old TeX distro :). Probably you're right that all potential LaTeX3 testers probably have a reasonably up-to-date distro. However, how "up-to-date" does it have to be? Anyone knows when texlua was introduced? > > not sure they are alternatives, but there could be approaches that are > > worth incorporating into the current setup. > > One possibility is to have a test spec, so we can have a "generic" test > infrastructure which reads this spec and "knows" how to perform a > certain analysis. Can you elaborate? Currently we simply run some functions, and check the output. > I volunteer myself to help. :) You guys know that TeX is still a monster > to me, but at least I think I can help in other battle fronts. :) Yeay! One problem with building a build system in Lua is that Lua does not provide any function to list files in a directory. The only way in pure Lua is to do the equivalent of \immediate\write18{os-specific command}. There are external libraries which do that in an os-dependent way. Paulo, how hard would it be to ship the LuaFileSystem library with a hypothetical Lua-based test file? Regards, Bruno