Hi friends! :) Sorry for the delay. The Mac setup was driving me crazy. :P > => more people run the testsuite => it is easier for people outside > the kernel team to add tests. In particular, many of the l3packages > are drastically lacking tests. Indeed. More tests, more stability IMHO. > My priority is to go for thorough tests which test all corner cases, > rather than fast ones. I may be wrong doing that, though, since the > tests are starting to take a rather long time to build. Ah I see. :) It's a good approach I think. And running test suites are usually slow, but it's a good excuse for us to go for a cup of coffee while the tests are running. :) >> One possibility is to have a test spec, so we can have a "generic" >> test infrastructure which reads this spec and "knows" how to perform >> a certain analysis. > > Can you elaborate? Currently we simply run some functions, and check > the output. I was just thinking out loud. :) I was thinking of a input format, say written in XML (/me looks at David) or another markup language, with a list of assertions, e.g, (sorry I don't know the innards of the tests neither L3 sutff) <asserttrue input="\foo" expected="bar" /> or something similar. To test a certain code snippet, a single test spec would suffice. > Paulo, how hard would it be to ship the LuaFileSystem library with > a hypothetical Lua-based test file? Bruno, I have very good news! texlua is shipped with lfs out of the box! IIRC we only need to install it for a "normal" Lua interpreter. We already have the OS commands ready for us to use via lfs. Yay! :) Best wishes, Paulo Em 10/08/2012 15:29, Bruno Le Floch escreveu: > Hi all, > >>> The important non-functional requirement is that it works >>> >>> - automatically >>> - and reasonably fast >>> >>> Obviously you have to run TeX on the test files, but doing the >>> comparison using TeX is not very time efficient. >>> >>> Now is it important that it is OS independent? Only if your goal is to >>> have *many* people use the mechanism and so far that wasn't really part >>> of the spec. > > I would add to your list > > - easy to install / understand > > => more people run the testsuite => it is easier for people outside > the kernel team to add tests. In particular, many of the l3packages > are drastically lacking tests. > >>> Perl is easily available both for unix and windows so effectively for >>> developers the current system is not to difficult to install or use. >>> The idea of using Lua is interesting, but as Joseph said, it is not that >>> this would then work out of the box for most people either (not now >>> anyway). >>> >>> Midterm, or if we think that this should be a package outside expl3 or >>> 2e core development, it would perhaps be a good option though, but for >>> now my feeling is it would mean putting resources onto something that >>> doesn't actually bring any new value. > > Agreed. > >> > I think the big question is what is your goal here. My main goal >> > initially (for 2e and later for l3 code) was to have a robust test >> > suite that would enable us to identity issues upfront when making >> > changes or additions. And that on the whole has been very successful >> > in the past (provided as Joseph said, people wrote test files :-) >> >> I can relate to that. :) Writing tests is one of the major concerns of >> the software industry. There's a lot of levels on how to test code, from >> unit to integration, and approaches, from the bowels to the interface - >> the Software Engineering wolves love these concepts. :P Of course, each >> project has its own needs and requiments. For LaTeX3, IMHO, it's >> critical to have every bit of code - from a lovely high level interface >> to an obscure undocumented feature - exhaustively tested. > > We're working on it, and I think most of l3kernel is tested. > Obviously not enough, as I keep finding obscure bugs. Latest one: the > msg system sometimes x-expands several times in a row the same > arguments. It'll get fixed soon. > > One thing that we don't have is detailed information about which > functions are tested. I've added recently a "tested = <testfile>" key > to the macro environment in l3doc, but I've only used it in a couple > of modules, and the key is entirely ignored. It may be good to extend > this to other modules. > >> > The important non-functional requirement is that it works >> > >> > - automatically >> > - and reasonably fast >> >> Agreed. Though a bunch of tests might require a reasonable amount of >> time, each test should be as simple as possible, thus making its >> checking relatively fast. > > My priority is to go for thorough tests which test all corner cases, > rather than fast ones. I may be wrong doing that, though, since the > tests are starting to take a rather long time to build. > >> > Now is it important that it is OS independent? Only if your goal is to >> > have *many* people use the mechanism and so far that wasn't really >> > part of the spec. >> >> I believe that OS independent test suites might help us catch things >> that are OS-specific, or even detect possible issues with a certain >> vendor implementation. Again, I speak as an outsider; I have no idea how >> the code actually works. > > One of my main concerns, after looking at the plethora of Makefiles > and make.bat is that there is a lot of redundancy there, which means > that (1) changes don't propagate well (2) we occasionally forget to > add a module to the Makefile of the parent directory, and it never > gets tested, installed, unless someone is working on that module. > > So I guess that in the LaTeX3 case the discussion should be extended > to the build system as a whole: the test system works well, as Frank > says. It would be nice if we could get rid of Makefiles and make.bat, > and replace them by light-weight configuration files. After all, the > Makefiles are all very similar, so a top-level Lua script (or other) > should be able to do most of the work. > >> > Perl is easily available both for unix and windows so effectively for >> > developers the current system is not to difficult to install or use. >> > The idea of using Lua is interesting, but as Joseph said, it is not >> > that this would then work out of the box for most people either (not >> > now anyway). >> >> I share the same opinion. :) One idea I see is to pack the test system >> as a batteries-included file (.sh for Unix, .exe for Windows) packing >> the interpreter together. I've seen this before with some languages - >> Ruby, Python, Perl, Java - and it's doable. Of course, there's a >> drawback of having the payload of the virtual machine / interpreter >> bundled with the script, but at least it would be very easy to deploy. >> Or we can write the system with another language - say C - and generate >> a binary file. > > There is also the question of licenses if you bundle an external > dependency with the code repo. I really don't know the details, > perhaps Frank has some ideas? > >> I like the idea of using Lua. Enrico and I wrote a Lua script and the >> deployment in TeX Live was very, very easy. It runs under texlua, which >> is already available in the modern TeX distros. If the guy doesn't have >> a TeX distro, why in the Earth he wants to run a LaTeX3 test suite? :P > > The guy might have an old TeX distro :). Probably you're right that > all potential LaTeX3 testers probably have a reasonably up-to-date > distro. However, how "up-to-date" does it have to be? Anyone knows > when texlua was introduced? > >> > not sure they are alternatives, but there could be approaches that are >> > worth incorporating into the current setup. >> >> One possibility is to have a test spec, so we can have a "generic" test >> infrastructure which reads this spec and "knows" how to perform a >> certain analysis. > > Can you elaborate? Currently we simply run some functions, and check > the output. > >> I volunteer myself to help. :) You guys know that TeX is still a monster >> to me, but at least I think I can help in other battle fronts. :) > > Yeay! > > One problem with building a build system in Lua is that Lua does not > provide any function to list files in a directory. The only way in > pure Lua is to do the equivalent of \immediate\write18{os-specific > command}. There are external libraries which do that in an > os-dependent way. Paulo, how hard would it be to ship the > LuaFileSystem library with a hypothetical Lua-based test file? > > Regards, > Bruno >