|
![](/i/fill.gif) |
Warp wrote:
> I myself am quite fond of putting assert()s everywhere where it makes
> sense. That has helped me catch tons of bugs which would have otherwise
> been quite laborious to find later.
This is probably something I should do more of...
Haskell makes it possible to prevent many things at compile-time (e.g.,
no null pointers, no empty lists, etc.), but there will always be things
you can only check at runtime.
> Another thing which I cannot stress enough is the importance of developing,
> if possible (ie. if the program/library type is suitable for it), an
> automatic regression testing program.
The Haskell guys have a tool they call QuickCheck, which (with a little
help from you) will randomly generate X tests and check the results.
Obviously this isn't the same as having a set of hand-written tests. If
you're trying to write a parser of some kind, you probably want
hand-written test strings to check out the edge-cases of the parser.
But, for example, a while back I was extending Haskell's binary parser
library to support bit-level alignment. Having QuickCheck autogenerate
half a trillion random bitstrings, write then and then read them back
and check they haven't changed is a pretty rapid way to find out that
your program is broken.
> What makes it
> regression testing is that every time a new feature is added to the
> program/library, a test or tests are added to the testing program and,
> most importantly, every time a bug is found in the program/library, a
> test which triggers the bug is added to the testing program.
>
> Thus every time a change is made to the program/library, the automatic
> testing program can be run against it, and a huge amount of tests will
> be performed automatically and easily, catching most bugs right there.
The guys working on the GHC compiler do this. The compiler's source
repository contains an entire directory full of code snippets. Some of
them are supposed to be rejected with a compile-time error, some of them
are meant to crash at runtime, and some of them are supposed to run
perfectly and produce a particular output. GHC includes a compiler and
an interpretter, and the interpretter can run source code and object
code, so there's at least 3 seperate ways to run a given program. Using
the test framework, you can check that the program produces the same
result when run all three ways, etc.
Every night, a set of "build bots" sync their repos to the central tree
and try to recompile the entire GHC system from scratch. (By that I mean
build the compiler, the interpretter, the standard libraries, the
runtime system...) This is a two-stage process; first, the source code
for the compiler is compiled using an old, old version of GHC, with half
the features turned off. Then the source code is compiled again with the
new compiler just created (and all features turned on). Assuming all
that works (i.e., it doesn't crash somewhere with Haskell syntax errors
or undefined linker symbols or C preprocessor glitches), the build bot
runs the "validate" script. This runs all XXX-thousand test programs and
checks that invalid programs get rejected, valid ones get accepted, each
program produces the output it should, running it in all possible ways,
etc. There's build bots for a dozen different flavours of Linux,
Windows, Mac OS... you name it.
In short, if somebody commits a patch that breaks something, you'll know
about it in 24 hours when all the build bots start reporting errors.
Also, like Warp says, every time somebody submits a bug to the bug
tracker [which turns out to be a valid bug, not something already known
about, etc.], they try to add a test case for it to the test suite.
On top of all that, there's a series of performance benchmarks - for the
compiler itself and for the programs it compiles...
Man, that's one hell of a complicated system! Much bigger than anything
I'll ever work on. Still, you can see why so much test infrastructure is
necessary. A compiler is a complex thing. I imagine similar projects do
something like this too.
--
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*
Post a reply to this message
|
![](/i/fill.gif) |