Warp wrote:

>   I myself am quite fond of putting assert()s everywhere where it makes
> sense. That has helped me catch tons of bugs which would have otherwise
> been quite laborious to find later.

This is probably something I should do more of...

Haskell makes it possible to prevent many things at compile-time (e.g., 
no null pointers, no empty lists, etc.), but there will always be things 
you can only check at runtime.

>   Another thing which I cannot stress enough is the importance of developing,
> if possible (ie. if the program/library type is suitable for it), an
> automatic regression testing program.

The Haskell guys have a tool they call QuickCheck, which (with a little 
help from you) will randomly generate X tests and check the results. 
Obviously this isn't the same as having a set of hand-written tests. If 
you're trying to write a parser of some kind, you probably want 
hand-written test strings to check out the edge-cases of the parser. 
But, for example, a while back I was extending Haskell's binary parser 
library to support bit-level alignment. Having QuickCheck autogenerate 
half a trillion random bitstrings, write then and then read them back 
and check they haven't changed is a pretty rapid way to find out that 
your program is broken.

> What makes it
> regression testing is that every time a new feature is added to the
> program/library, a test or tests are added to the testing program and,
> most importantly, every time a bug is found in the program/library, a
> test which triggers the bug is added to the testing program.
> 
>   Thus every time a change is made to the program/library, the automatic
> testing program can be run against it, and a huge amount of tests will
> be performed automatically and easily, catching most bugs right there.

The guys working on the GHC compiler do this. The compiler's source 
repository contains an entire directory full of code snippets. Some of 
them are supposed to be rejected with a compile-time error, some of them 
are meant to crash at runtime, and some of them are supposed to run 
perfectly and produce a particular output. GHC includes a compiler and 
an interpretter, and the interpretter can run source code and object 
code, so there's at least 3 seperate ways to run a given program. Using 
the test framework, you can check that the program produces the same 
result when run all three ways, etc.

Every night, a set of "build bots" sync their repos to the central tree 
and try to recompile the entire GHC system from scratch. (By that I mean 
build the compiler, the interpretter, the standard libraries, the 
runtime system...) This is a two-stage process; first, the source code 
for the compiler is compiled using an old, old version of GHC, with half 
the features turned off. Then the source code is compiled again with the 
new compiler just created (and all features turned on). Assuming all 
that works (i.e., it doesn't crash somewhere with Haskell syntax errors 
or undefined linker symbols or C preprocessor glitches), the build bot 
runs the "validate" script. This runs all XXX-thousand test programs and 
checks that invalid programs get rejected, valid ones get accepted, each 
program produces the output it should, running it in all possible ways, 
etc. There's build bots for a dozen different flavours of Linux, 
Windows, Mac OS... you name it.

In short, if somebody commits a patch that breaks something, you'll know 
about it in 24 hours when all the build bots start reporting errors.

Also, like Warp says, every time somebody submits a bug to the bug 
tracker [which turns out to be a valid bug, not something already known 
about, etc.], they try to add a test case for it to the test suite.

On top of all that, there's a series of performance benchmarks - for the 
compiler itself and for the programs it compiles...

Man, that's one hell of a complicated system! Much bigger than anything 
I'll ever work on. Still, you can see why so much test infrastructure is 
necessary. A compiler is a complex thing. I imagine similar projects do 
something like this too.

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message