http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx

-- 
   Darren New, San Diego CA, USA (PST)
   I ordered stamps from Zazzle that read "Place Stamp Here".

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx

  I myself am quite fond of putting assert()s everywhere where it makes
sense. That has helped me catch tons of bugs which would have otherwise
been quite laborious to find later.

  Another thing which I cannot stress enough is the importance of developing,
if possible (ie. if the program/library type is suitable for it), an
automatic regression testing program.

  This would simply be a program which automatically tests the validity of
the program or library being developed. In other words, it runs it against
a set of test data and checks that the results are correct (or within
acceptable margin if the result is expected to be fuzzy). What makes it
regression testing is that every time a new feature is added to the
program/library, a test or tests are added to the testing program and,
most importantly, every time a bug is found in the program/library, a
test which triggers the bug is added to the testing program.

  Thus every time a change is made to the program/library, the automatic
testing program can be run against it, and a huge amount of tests will
be performed automatically and easily, catching most bugs right there.

  One project where I have done this is with my function parser library.
I have been developing it for something like 10 years, and during those
years I have also developed an automatic regression testing program
alongside it. It has grown to be quite large and extensive during the
years, and it works phenomenally. I'd say at least 90% of all bugs are
caught immediately by just running the testing program. That's a HUGE
timesaver.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:

>   I myself am quite fond of putting assert()s everywhere where it makes
> sense. That has helped me catch tons of bugs which would have otherwise
> been quite laborious to find later.

This is probably something I should do more of...

Haskell makes it possible to prevent many things at compile-time (e.g., 
no null pointers, no empty lists, etc.), but there will always be things 
you can only check at runtime.

>   Another thing which I cannot stress enough is the importance of developing,
> if possible (ie. if the program/library type is suitable for it), an
> automatic regression testing program.

The Haskell guys have a tool they call QuickCheck, which (with a little 
help from you) will randomly generate X tests and check the results. 
Obviously this isn't the same as having a set of hand-written tests. If 
you're trying to write a parser of some kind, you probably want 
hand-written test strings to check out the edge-cases of the parser. 
But, for example, a while back I was extending Haskell's binary parser 
library to support bit-level alignment. Having QuickCheck autogenerate 
half a trillion random bitstrings, write then and then read them back 
and check they haven't changed is a pretty rapid way to find out that 
your program is broken.

> What makes it
> regression testing is that every time a new feature is added to the
> program/library, a test or tests are added to the testing program and,
> most importantly, every time a bug is found in the program/library, a
> test which triggers the bug is added to the testing program.
> 
>   Thus every time a change is made to the program/library, the automatic
> testing program can be run against it, and a huge amount of tests will
> be performed automatically and easily, catching most bugs right there.

The guys working on the GHC compiler do this. The compiler's source 
repository contains an entire directory full of code snippets. Some of 
them are supposed to be rejected with a compile-time error, some of them 
are meant to crash at runtime, and some of them are supposed to run 
perfectly and produce a particular output. GHC includes a compiler and 
an interpretter, and the interpretter can run source code and object 
code, so there's at least 3 seperate ways to run a given program. Using 
the test framework, you can check that the program produces the same 
result when run all three ways, etc.

Every night, a set of "build bots" sync their repos to the central tree 
and try to recompile the entire GHC system from scratch. (By that I mean 
build the compiler, the interpretter, the standard libraries, the 
runtime system...) This is a two-stage process; first, the source code 
for the compiler is compiled using an old, old version of GHC, with half 
the features turned off. Then the source code is compiled again with the 
new compiler just created (and all features turned on). Assuming all 
that works (i.e., it doesn't crash somewhere with Haskell syntax errors 
or undefined linker symbols or C preprocessor glitches), the build bot 
runs the "validate" script. This runs all XXX-thousand test programs and 
checks that invalid programs get rejected, valid ones get accepted, each 
program produces the output it should, running it in all possible ways, 
etc. There's build bots for a dozen different flavours of Linux, 
Windows, Mac OS... you name it.

In short, if somebody commits a patch that breaks something, you'll know 
about it in 24 hours when all the build bots start reporting errors.

Also, like Warp says, every time somebody submits a bug to the bug 
tracker [which turns out to be a valid bug, not something already known 
about, etc.], they try to add a test case for it to the test suite.

On top of all that, there's a series of performance benchmarks - for the 
compiler itself and for the programs it compiles...

Man, that's one hell of a complicated system! Much bigger than anything 
I'll ever work on. Still, you can see why so much test infrastructure is 
necessary. A compiler is a complex thing. I imagine similar projects do 
something like this too.

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> Man, that's one hell of a complicated system!

How long does it take to run both compilation stages followed by all the 
unit tests?

...Chambers

Post a reply to this message

Orchid XP v8 <voi### [at] devnull> wrote:
> The guys working on the GHC compiler do this. The compiler's source 
> repository contains an entire directory full of code snippets.

  AFAIK the gcc project also has a gigantic set of automated tests, so
that every new release is first run against it. Every time a legit bug
is reported, a test triggering that bug is added to the set.

-- 
                                                          - Warp

Post a reply to this message

>> The guys working on the GHC compiler do this. The compiler's source 
>> repository contains an entire directory full of code snippets.
> 
>   AFAIK the gcc project also has a gigantic set of automated tests, so
> that every new release is first run against it. Every time a legit bug
> is reported, a test triggering that bug is added to the set.

It would seem the only sensible way to work on a project as huge as GHC 
or GCC, yes.

Given how much more popular GCC is than GHC, I imagine the test set 
is... kinda huge by now.

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

>> Man, that's one hell of a complicated system!
> 
> How long does it take to run both compilation stages followed by all the 
> unit tests?

How fast is your PC?

It's kind of like asking how long it takes to render SKYVASE.POV; it 
depends. ;-)

On my ancient laptop, with it's single-core 32-bit processor clocked at 
less than 2 GHz, it took about 2 hours:

30 minutes to compile stage1 using stage0 + stage0-libs.
30 minutes to compile stage1-libs using stage1.
30 minutes to compile stage2 using state1 + stage1-libs.
15 minutes building the documentation files. (They're embedded in the 
source code as comments.)
45 minutes to run all tests.

You can check the output of the actual buildbots yourself if you like:

http://darcs.haskell.org/buildbot/

It's not very intuitive, but... looks like the test suit contains about 
2,000 files, when produce 9,000 tests (taking into account that some 
tests can be run multiple ways - through the compiler, through the 
interpretter, with debugging on, with debugging off, with profiling on, 
etc.)

Looks like the whole process takes several hours on the buildbots too.

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Warp wrote:
>   I myself am quite fond of putting assert()s everywhere where it makes
> sense. That has helped me catch tons of bugs which would have otherwise
> been quite laborious to find later.

Me too. I think what I understood the articles to be saying is that this is 
very helpful, but only if you are experienced enough to use the right assert()s.

>   Another thing which I cannot stress enough is the importance of developing,
> if possible (ie. if the program/library type is suitable for it), an
> automatic regression testing program.

Sadly, the vast majority of my work these last few years is completely 
antithetic to this sort of testing.

I did this even for programs running off floppies in 64K, where I'd save 
inputs and outputs, and if the input no longer generated the same output, 
I'd manually eyeball it to see if the change I just made should have created 
that output, and if so, save the new output.  That was helpful.

>   One project where I have done this is with my function parser library.

Some programs lend themselves to that, some don't.

Did you do this with Hopido?  What parts could you test?

-- 
   Darren New, San Diego CA, USA (PST)
   I ordered stamps from Zazzle that read "Place Stamp Here".

Post a reply to this message

Orchid XP v8 wrote:
> Man, that's one hell of a complicated system! Much bigger than anything 
> I'll ever work on. 

I saw a better one. When someone made a change, it runs thru the smoke 
tests, and then gets pushed up to about 5% to 10% of the web servers. If 
after 15 minutes the web servers haven't reported any statistically unusual 
number of script failures, it gets pushed out to more and more servers. So 
they do a deploy on their production system every 30 to 60 minutes during 
the workday.

-- 
   Darren New, San Diego CA, USA (PST)
   I ordered stamps from Zazzle that read "Place Stamp Here".

Post a reply to this message

Warp wrote:
> Orchid XP v8 <voi### [at] devnull> wrote:
>> The guys working on the GHC compiler do this. The compiler's source 
>> repository contains an entire directory full of code snippets.
> 
>   AFAIK the gcc project also has a gigantic set of automated tests, so
> that every new release is first run against it. Every time a legit bug
> is reported, a test triggering that bug is added to the set.

It's not hard to organize for big batch process type things with 
well-defined inputs and outputs. To the extent that you can turn difficult 
problems into collections of functional input->output streams, you can apply 
this to messier problems.

-- 
   Darren New, San Diego CA, USA (PST)
   I ordered stamps from Zazzle that read "Place Stamp Here".

Post a reply to this message